Software Projects Will Be Late

I've been a product manager by trade, and I've done various kinds of engineering over time, especially software engineering. Back-end, front-end, infrastructure, architecture. That means that I've had to deal both directly and indirectly with the eternal struggle of software estimation. This is the tug-of-war between the non-engineers who call the shots the product teams about when exactly Product A or Feature B will be ready. (Some call those non-engineers "the business people", but I'll use the nicer "project sponsors".)

Everyone in software is familiar with that struggle, and understands instinctively why it happens. No one on the other side has a clue (when they do, the struggle doesn't happen). So here I'll attempt a very brief and simplified explanation of this woeful corporate trap. If you're about to punch a sponsor, consider linking this page to them instead.

I'll start with the upshot: software projects will be late, because of their very nature.

To people who have never written production software, this feels like a cop out, an excuse to be sloppy with one's estimates. This is understandable for inexperienced managers. "You told me you'd finish it in three weeks, and it's been over a month. Therefore you're either lying or bad at your job." It does sound like good, straightforward reasoning. The catch is upsteam: expecting to have neat dates to mark in a calendar in the first place.

The tendency to compare software work to other kinds of engineering makes the misunderstanding even more insidious. The argument goes like this:

Sure, construction projects are known to be always late too, but they involve huge teams, they have to deal with the weather, they need to provision materials in the right order, etc. Software usually lacks all of those complications and force majeur factors. It's just a few people (sometimes very few) typing at a keyboard. It's supposed to be the epitome of autonomy. Who else is there to blame?

Fred Brooks wrote a famous book about the pitfalls of software estimation in the 1970's called The Mythical Man Month. After almost 50 years, it's still very relevant. His most famous statement is what software people know as Brook's Law:

Adding manpower to a late software project makes it later.

His argument is that software output is not additive. More people in a team means more people everyone has to talk to and coordinate with, which actually slows everyone down. Also Brooks: "the bearing of a child takes nine months, no matter how many women are assigned."

Another point that Brooks makes is that all programmers are optimists, so they tend to underestimate the problems and dead-ends ahead of them and overestimate how quickly they can get things done. He is right that optimism is common and makes things worse, but I don't believe that programmers are particularly more optimistic than any other category of people. It's just what humans do when faced with software. The real culprit is in the peculiar nature of the job.

A key difference between software and other engineering tasks is that, whenever you encounter a new challenge that you have already solved before, you only need to copy and paste the code to solve it. Unlike houses and robots, creating a new copy of past work is effectively free and instantaneous. This usually removes 99% of the repeatable parts of the process, so that all of the things you actually have to do are the non-repeatable parts.

Paradoxically, this doesn't save you project time, although it does save you total time. Usually the engineers know what parts they can reuse instantly, and will only include in their estimates the other parts, those that are new and strange and full of unknowns. By definition, software development is composed almost entirely of solving problems you're not sure how to solve.

To put it another way, you should think of software work as equivalent not to the construction of a bridge, but to its design (I borrow this idea from an insightful Hacker News comment). The actual building part is more similar to the act of "compiling" code, which is almost entirely automated and only takes a few minutes. Or, if you really want to equate software to building the bridge, then it's like building the first bridge ever, or the first bridge with a new set of materials or never-before-tried environmental conditions or new physical laws. Every single time. This means that you can't predict with confidence what's going to be possible at every stage until you've done the previous work leading to that stage, nor can you predict all the problems that will surface once you attempt to solve the problems that you can predict.

If that metaphor didn't resonate, here's another. Software product development is like having to climb an unexplored mountain. Usually it is not impossible, and you may even know pretty accurately where you want to end up (e.g. the mountaintop), but you have to figure out the way as you go, one or a few steps at a time. If you don't even know the path you're going to take to get there, good luck guessing how long it'll take you to walk it. You have to make the map as you go—and in software the map will become unnecessary the moment you're finished, because you never climb the same mountain twice.

That is not to say that no time estimate is ever possible in software. There are methods to increase the accuracy, and good engineers use them. But for all the effort that you might put in it, the accuracy is never going to be high, and it's never even going to be medium. You have to live bravely with poor estimates.

So how do you do it then? Seen in this framing, the Agile movement that everybody knows—including the project sponsors—makes more sense, I think. The core values of the Agile Manifesto go like this:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Agile is all about improvisation and trial and error. It's about being good at orienting yourself in the wilderness rather than following a map that doesn't exist.

Exactly how to implement this philosophy is a tougher nut to crack. People have come up with many approaches, of which Scrum is the most popular. They have all been criticized, because very often they don't "work". Now, as much as I like being the balanced, unbiased guy, in this case I have to lean heavily for the programmers' side. Agile is a way of thinking about software development, and it's a very wise way to think about it. But it's never going to work if only the engineers adopt it. If the sponsors of a project keep on asking for calendarized roadmaps and Gantt charts, it will all be for nothing. It won't matter what practical declination of Agile the team uses: your project will be late.

The sponsors who embrace the call of the wild, on the other hand, gain the presence of mind to set up things so that delays won't spoil everything. That's where Agile will work. Of course, the software projects will still be late, but it won't matter very much. ●