How much does a slow build cost?

At work recently, I’ve been having a series of conversations about build times and the effect thereof.  As an exercise, I ran through a quick estimate of what a minute of build time actually costs.  Frankly, the result shocked me.

This post consists of two parts: the economic analysis of the cost of a minute of build time, and the implications in terms of what corrective measures might be justified.

Cost Analysis

For the purposes of this exercise, let’s try to come up with a fairly conservative estimate.  We want something which is easily defensible so that we can talk about the actual issues, not the details of the calculation.  Personally, I suspect a good case could be made for a much larger cost estimate, but as we’ll see below, we don’t really need to.

Let’s start with a fairly standard estimate of the cost to hire one developer: $250,000 a year.  This may at first seem too high, but remember that it includes all taxes, health insurance, other benefits, and overheads.  Those amounts can easily double (or more) the stated base salary.

Now, let’s say that developer works 50 weeks during the year, 40 hours each.  In practice, a real developer would probably work a bit less given (my guess at) an industry average of around 15 days vacation and 7-8 holidays.   That’s fine though, over estimating the amount worked can only understate our final cost.

Given 2,000 hours (= 50 x 40) for that per employee cost of $250,000 per year, that gives an effective rate of $125 an hour or a bit over $2 per minute.

Now, let’s go back to our original interest, the cost of that minute of build time.  Let’s estimate that a typical developer runs a local build 20 times a day on average.  This includes all of the silly syntax fixes, tweaking, bug fixing, etc…  This is honestly probably the least reliable estimate I’ll use, but it’s at least in the right order of magnitude.

Ok, that’s interesting, but what does all that mean?  Well, here it comes.

One minute of build time has an opportunity cost of $40 per day per developer.  (=20 x $2)  That same minute of build time costs $8,000 per year per developer. (= $40 x 50 x 40)

I don’t know about you, but that was surprising high to me the first time I ran through it.

Cost Effectiveness of Corrective Action

Given that cost, what approaches to reducing build times might be justified?
Well, we can estimate that pretty easily as well.

Buy More Build Machines

First, let’s look at simply adding more hardware to make the build faster.  Using Amazon EC2 pricing as an easily accessible stand-in for the cost of a well run server fleet, we can see that one minute of developer time is worth somewhere between 4 and 67 CPU hours of EC2 On-Demand usage.  (2nd Gen Extra Large: $0.5 per hour, 1st Gen Small: $0.06 per hour)  That’s something on the order of 80-1200 CPU hours per day per developer.

Keep in mind that with proper allocation of reserved instances and spot pricing, you could probably justify even more.  Depending on the size of the development group, a dedicated server (or several!) might also be justified.

Invest in Build Engineering

The alternate approach is to invest more effort into the build system: incrementalize, reduce false dependencies, tune for compile time using hash checks or timestamps, split apart large files, and generally solve the interesting technical problem of task scheduling in the context of build execution.

With a single developer, it is profitable to invest one day a month if you can reduce build times by one minute.  (Developer cost per day: $1,250 = $250k/(50*4), opportunity cost of one minute build time in 32 days:  $1,280=$50 x 32)

With a team of 32 developers, it is profitable to pay a full time engineer to focus on reducing build times if that engineer can reduce build times by one minute per month.

Note that for small teams it is far more effective to throw machine time at the problem then engineering resources.  Even for larger teams, the appropriate cost calculation is NOT the cost of the build time, it is instead the cost of the machine resources needed to eliminate that build time.

Hopefully, you found this article interesting.  I may follow it up with a set of posts on related issues, but there’s a couple of things queued up first.  If you’re particularly interested in this topic and want to hear more about, please let me know.