How to build only meaningful tools
Learn your manager's incentives in order to build meaningful tools
I recently watched a YouTube video of a technical manager from Criteo, Justin Coffey, demoing the in-house business intelligence tool his team had built. It’s a great presentation. Justin actually lists reasons why his team decided to build its own tool; all are good reasons, except none answer why they would have reinvented something when a better tool already exists. (see footnotes)
Since I work across vendors and indirectly benefit from the complexity of new data tools—open source included—I also frequently hear reasonable arguments against buying a new tool. The specifics vary among the non-techies, but among the technologists, most arguments boil down to just a few:
- vendor lock-in
- poor feature prioritization
- product reliability
We are building in-house because….
Vendor lock-in is an issue. Migrating away to another solution is costly. But let’s consider the alternative: building on top of an open source stack.
Migration away from Apache to Nginx has been one of the widespread open source phenomena over the past few years. Both are very popular. Search for “apache better than nginx,” however, and you would be hard-pressed to find an argument in favor of Apache. So why do many companies still rely on it?
Looking at a W3Techs web stats, one thing is clear: more frequently visited sites are more likely to make the switch to Nginx. This trend makes sense, as the more important website performance becomes for your core business, the more likely you are to make the migration.
“Top ranked website heavily favor Nginx over Apache”
There are two sides to this coin, however. The majority of less frequently visited sites remain on Apache (45% vs. 36.2%). A quick search (or this) proves that Nginx is supported by the old-school managed web-hosting services providers as well. Old or new, web server is simply not core to these businesses.
As a more specific example, it took Facebook more than six years to navigate new code development away from php (Emil Protalinski). And to support that initial development decision, they had to invent a whole host of software to accommodate the inefficiencies of php. Consider the cost of those engineering resources as a cost of an open-source lock-in.
If your open-source-powered solution is not the core of your business, consider yourself locked in.
So whether it is your company’s reliance on a language, framework, or software, migrating open-source solutions is not less costly.
Feature prioritization is a hard one to argue in favor of vendors. As an owner, you can certainly prioritize features to your needs better. If you need a custom visualization and you don’t care about the new SQL dialect, you just focus on the former. But just as with the above, there is a significant hidden cost.
For every new feature you introduce, there is a cost in maintenance. To quote one statistic dating back to 1999, the maintenance of software used to comprise 77% of resources in the product lifecycle. Whatever that statistic is today, people consistently underestimate the overall resources required to maintain anything in tech. Consider Josh Pigford’s Build vs. Buy calculator. It highlights the cost of maintenance assuming X number of days per month for a team of just three people earning $150k as software engineers.
Already at 1.5 days per month, it takes five years to break even against a typical business intelligence vendor buy-in.
Product reliability can be more important than any of the above, since unattended bugs can cost your company more than the cost of maintenance. So it’s great to have direct access to that very engineer who can push the fix when stuff breaks. But here’s more truth: who is to say that one or two years after building that product in-house, the manager or owner of the new internal tool will still have access to the needed engineer?
Engineers do not stay on the same project forever. They frequently change projects and teams. It’s not enough for one manager to have a maintenance plan. An entire management team needs to plan resource allocation for years to come on every new product built in-house.
Forcing engineering to respond to bug issues is simply a lot easier as a paying customer than as a manager with questionable internal political clout. Most SaaS vendors do not make money on the initial annual contract. To quote Tom Tunguz’s sales efficiency survey, most SaaS companies average around 1.25 years for a payback on customer acquisition.
“Payback = 1/sales efficiency (average of 1⁄0.8 = 1.25)”
The entire industry survives because of renewals. That gives a buyer a lot of leverage with vendors. Contrast that with a manager (A) who needs to make a case for a manager (B) on resource allocation. Bottom line, vendor-supplied tools are more reliable than in-house solutions that are not core to the primary business, because vendors have to respond to feedback or else they go out of business.
Real incentives behind every managerial argument
With the above out of the way, I want to now focus on how technical managers are actually evaluated at most companies. The above three arguments should be induced by one of these incentives:
- Is the manager meeting product requirements on schedule and on budget?
- Is the manager maintaining a talented team of engineers?
Often there are other official and unofficial KPIs, such as the ones measuring the success of creative ideas from engineers themselves, but most of that still boils down to (1) and (2).
A typical CEO is more likely to hear the opinion of our hypothetical engineering manager on vendor lock-in, feature prioritization, and product reliability insofar as (1) is concerned. This is just easier for business people to understand: can the team consistently deliver results on schedule without exponential growth in resources required?
However as the above counter-arguments have shown, when it comes to solutions not core to the business, > meeting product requirements on schedule and on budget has nothing to do with choosing a vendor vs. building your solution in-house.
It is okay to be biased in favor of in-house development, but alluding to product requirements and budget is a misrepresentation of facts. There is no inherent advantage in building something internally on open source over a vendor-supplied tool.
Difficulties in maintaining engineering talent
With (1) covered, it’s time to examine the talent question. Most successful tech organizations are aggressive on this topic, and rightfully so—it’s hard to keep engineering talent happy. Engineers are rarely satisfied with mundane repeated tasks. They need room to create and innovate.
The problem is that only the roles close to the core of their employer’s competency actually have that satisfaction consistently. The same just cannot be said of a whole host of other roles (e.g., data engineering, DevOps, etc.). The lifecycle for the latter roles typically looks something like this:
- deploy new infrastructure
- handle ticket, handle ticket, handle ticket
Compare this to a typical software engineering lifecycle:
- push new code
- handle ticket (fix code)
- push new code
- …
As a company, you need these data and DevOps roles really badly, but only sometimes. The question becomes: how do you keep them all the time? Or rather, how do engineering managers negotiate for these experts to stick around?
Managers vs. Developer
As with all negotiations involving asymmetrical information, the inner-team “let’s build it in-house” argument plays to incompetence on facts around existing solutions. Unlike managers, engineers are typically not exposed to product demos or sales-driven trials. Hence only the former know when a needed tool likely already exists (hint: for any problem worth solving, there is likely a business trying to solve it).
Despite this knowledge, many managers ignore this simple fact and choose to build an inferior solution in-house. Building a solution in-house keeps engineering staff engaged even for those engineers outside of the core product development. This works under one condition: engineers must not be aware of what other tools exist. Just think about it—no one ever approaches you in life with a proposition like this:
“Can you help me make this cake? It won’t be as good as the one you can buy at a store, and it will cost us more in ingredients. Oh, and you will have to clear the shit after people eat it.”
You might love making cakes, but the truth would not be very enticing. And so is the absence of truth in solution purchasing decisions. Technical managers really don’t give a damn about technical pros and cons when deciding whether to buy or build. They care about talent.
Comparative advantage (closing notes)
As an engineer, it’s your job to stay informed on the latest technologies. So do stay informed on the vendors as well.
The rules for product development for the world at large apply just as well to the inner workings of tech companies. Don’t build something unless you have a comparative advantage in it.
Technical managers will continue to be biased in favor of building in order to maintain their talent. Stay informed, and you will spend your 80% on the 5% that actually matter.