Pop quiz, hotshot. How much does it cost to build a self-hosted Kubernetes cluster? Quick, no conferring.

If you thought the answer was “nothing”, go to the back of the class. According to distributed systems expert Cindy Sridharan, quoted in Cloud Native DevOps with Kubernetes, the answer is “one million dollars”:

It takes well over a million dollars just in engineer salary to get Kubernetes up and running from scratch. And you still might not get there.

How much is an hour of time worth?

Whether or not that figure is exact, here's the point: engineering ain't free. The sticker price for a piece of software is just the start: you need to skill up your engineering team to operate it, or hire engineers who already can. You need to provision infrastructure to run that software; you'll need to handle updates and security upgrades, you'll need to monitor it and alert on problems, and you'll need to audit what you've done regularly to make sure it's secure and reliable.

How much is your time worth? If you're a consultant or freelancer, you already know. If you're an employee, divide your annual take-home salary by 262 (the average number of working days in a year) and then by another factor of 8, to get your hourly rate. If you work for yourself, divide your annual revenue by your working hours in the same way.

If we say that a typical salary for a US software engineer is $110,000, just to pick a figure, that gives us a value for their time of about fifty dollars an hour. Let's see how to use that to reason about technology decisions.

What does it really cost to self-host a service?

Suppose you are considering running the commercial software Product X, and you're not sure whether to host it yourself or pay for a managed service. Let's look at the self-hosted option first.

How many engineer-hours does it take to get up to speed on Product X, develop a proof of concept, provision the necessary infrastructure, set up testing and deployment, change control, train staff, integrate with other tools, and all the other tasks that need to be done before you're actually ready to use Product X? Suppose we estimate that a team of five great engineers could do it in a week (this sounds optimistic, but let's run with it anyway).

That's about 200 engineer-hours, or $10,000. Now, how much time per month will you need to maintain and manage your Product X installation, deal with updates, support, outages, backups, and so forth? Let's guess 8 team-hours a month, or about $24,000 a year. So we have a cost schedule that looks like this:

Setup: $10,000 Year 1: $24,000 Year 2: $24,000 Year 3: $24,000

You may object that those engineers are on salary, so they'd be paid anyway, whether or not you use Product X. True, but their time has value, and we just worked out approximately how much. Every hour they spend on Product X is an hour they can't spend on something else valuable to the business.

How does the “buy” option compare?

Now let's compare this with a typical hosted service. You might expect to pay about $125 per month, for example, for a 5-node Kubernetes cluster. Most hosted services probably cost in the region of $100-1000 per month, depending on scale. Let's assume it's $1000 and see how that works out:

Setup: $0 Year 1: $12,000 Year 2: $12,000 Year 3: $12,000

Over three years, that's less than half the total cost of the self-hosted option, and there's another important point. Suppose after year 1 you decide that Product X isn't right for you, and you need to switch to Product Y. If you spent $10,000 of engineering time in getting it up and running, that investment has gone down the tubes. You face the awkward choice of writing off that money, and spending another $10,000 spinning up Product Y, or being stuck with a product that isn't right for you.

Why is self-hosting so expensive?

From a purely financial perspective, it already looks like maybe it isn't such a great idea to self-host. (Remember, we're not talking about building Product X yourself from scratch, which would obviously be a much more expensive endeavour, but only about hosting an existing product yourself.)

Why is it so much cheaper for a SaaS company to host your software? They have many customers, and can leverage economies of scale. The marginal cost of adding one customer to an existing SaaS infrastructure is pretty small.

The illusion of control

Even if it costs about twice as much to run software yourself, isn't that partly outweighed by the fact that you have direct control over it, so you'll be able to make Product X more secure, reliable, and performant?

No. You won't. Big cloud vendors are very good at running production services at scale, and however good you are, you're probably not that good. This isn't your core business, and you haven't had years to develop the requisite experience with Product X to really make it sing.

Why are these results surprising?

Really, this shouldn't come as a surprise. You probably didn't build your own house, for example, and if you did, I'm sure it's lovely, but it may not be quite as well-built as if you'd hired someone who builds houses full-time.

So why is it often our default assumption that we'll build a homebrew monitoring system, or our own deployment pipeline, or our own fault-tolerant database clusters?

Engineers gonna engineer

Convinced now that your engineers should be spending their time on your unique product or service, instead of laboriously reinventing an inferior version of something that already exists, you may encounter some resistance to this idea from a surprising source: your engineers.

But maybe that's not so surprising. The thing about engineers is that they like engineering stuff. That's why they got into this game in the first place. And that's a good thing—no, a great thing. Engineering talent is rare and valuable. What should we have our talented engineers working on?

What is “undifferentiated heavy lifting”?

There are things it makes sense for your engineers to build themselves, and things it doesn't. A useful phrase here is undifferentiated heavy lifting. In other words, this describes work which is hard, time-consuming, and adds no value. It's not something that makes money for your business, it's something that's part of the cost of doing business. This includes things like issue tracking, project planning, monitoring and alerting, CI/CD, container orchestration, and so on.

All these things are quite generic; anybody could do them, so it's likely that someone can do them cheaper and better than you can. Your team should be spending their time on unique stuff that creates value for your business, not undifferentiated heavy lifting.

Conclusion: weighing the options

What's the takeaway here? Are we saying that you should never build anything in-house? No. The takeaway is that when you make the decision to buy or build, you now have some quantitative tools to help you.

Assign a number to the cost of an engineer-hour, make a realistic estimate of the engineer-hours required to build and run something, and weigh that against the dollar cost of the equivalent SaaS product. Don't forget to factor in the cost of changing your mind down the road.

Don't just regard your engineers as nature's bounty; their time is a very valuable resource. Spend it wisely. Look closely into what your engineers are actually spending their time on; if you don't know, this is a good time to find out. How much of that activity is genuinely creating value for the business, in the short or long term? And how much is just wheel-reinvention or undifferentiated heavy lifting, that could be done better and cheaper by someone else?