back

by yen223·8mo ago·view on hn ↗
The unique thing about estimates in software engineering is that if you do it right, projects should be impossible to estimate!

Tasks that are easiest to estimate are tasks that are predictable, and repetitive. If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.

But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.

But if the predictable tasks take 0 time, how long a project takes will be dominated by the novel, unpredictable parts.

That's why software estimates are very hard to do.

18 comments
And I'd add that the need for them is a sign they aren't worth doing.

As you say, worthwhile software is usually novel. And to justify our expense, it needs to be valuable. So to decide whether a project is worth doing, we're looking at some sort of estimate of return on investment.

That estimate will also, at least implicitly, have a range. That range is determined by both the I and the R. If you don't have a precise estimate of return, making your estimate of investment more precise doesn't help anything. And I've never seen an estimate of return both precise and accurate; business is even less certain than software.

In my opinion, effort put into careful estimates is almost always better put into early, iterative delivery and product management that maximizes the information gained. Shipping early and often buys much clearer information on both I and R than you can ever get in a conference room.

Of course all of this only matters if running an effective business is more important than managerial soap opera and office politics. Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.

  > worthwhile software is usually novel.
Another type of worthwhile software is business CRUD. And unfortunately in my experience even for these mundane types of tasks time estimates are typically misjudged - for a multitude of reasons.

The typical reason is that developers in large teams often don't know all the places where a field may be used - and their IDE could very well miss some. So automated testing or QA finds an issue, and then the unaccounted-for, time-consuming bugtracking begins.

CRUD: Create, Read, Update, Delete. Basically just inputting and displaying data from a database.

> As you say, worthwhile software is usually novel.

This is an interesting assumption. I’d argue that the overwhelming majority of software is the most boring LoB CRUD apps you can imagine, and not novel at all. Yet, people need to estimate the tasks on these projects as well.

And starting in the late 1970s, there were tools available to simplify building LoB CRUD apps. [1] That has continued with things like Rails and Salesforce and no-code tooling.

If something is truly boring in software, it gets turned into a library or a tool for non-programmers to use. Our value is always driven by the novelty of the need.

And no, people don't need to estimate the tasks. My dad did LoB apps in the 1970s to the 1990s. E.g., order entry and shop floor management systems for office furniture factories. His approach was to get something basic working, see how it worked for the users, and then iteratively improve things until they'd created enough business advantage and/or cost savings to move on. Exploratory, iterative work like that can at best be done with broad ballpark estimates.

I grant that people want estimates. But that is usually about managerial fear of waste and/or need for control. But I think there are better ways to solve those problems.

[1] e.g., https://en.wikipedia.org/wiki/DBase

But it's doing something novel, something the same people haven't done before, otherwise there would be no point in writing it.
< “Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.”

This is so good.

Thanks. It was hard won. I spent maybe a decade naively thinking that if we just made software methods that worked in service of stated business goals and values, they'd get adopted and we'd all live happily ever after.

It took me a long time to come to grips with the POSIWID [1] version of the purpose of planning and estimates. One of the things that really blew my mind is Mary Poppendieck's story about how they built the Empire State Building on time and under budget even though they didn't have it fully designed when they started. [2] Different, more effective approaches are not only possible, they exist. But they can no longer win out, and I think it's because of the rise of managerialism, the current dominant ideology and culture of big business. [3]

[1] https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...

[2] Talk: https://www.infoq.com/presentations/tyranny-of-plan/ And transcript: https://web.archive.org/web/20140311004931/https://chrisgagn...

[3] See, e.g., https://www.amazon.com/Confronting-Managerialism-Business-Ec...

Further: In The Next Generation, when Scotty shows up, he mentions to Geordi he anyways padded his estimates because he knew Kirk would do things like that.
The solution you described is basically agile, and that definitely includes estimates and deadlines.
There are agile methods that forgo estimates and deadlines though

This is what "agile" is: https://agilemanifesto.org/

More specific methodologies that say they are agile may use concepts like estimates (story points or time or whatever), but even with Scrum I've never run into a Scrum-imposed "deadline". In Scrum the sprint ends, yes, but sprints often end without hitting all the sprint goals and that, in conjunction with whatever you were able to deliver, just informs your backlog for the next sprint.

Real "hard" deadlines are usually imposed by the business stakeholders. But with agile methods the thing they try to do most of all isn't manage deadlines, but maximize pace at which you can understand and solve a relevant business problem. That can often be just as well done by iteratively shipping and adjusting at high velocity, but without a lot of time spent on estimates or calendar management.

It’s Agile philosophically, and how it should be.

But that is rarely how it works. In the dozens of different projects across ten or twelve companies I’ve had insight into, “doing Agile” is analogous with “we have a scrum master, hold stand ups, and schedule iterations” while the simple reality is “Agilefall.”

I can say with some confidence, having been involved in the movement since before the term "Agile" was coined, that it requires neither.

I grant that both of those are common, but that's because the median "Agile" implementation quickly devolved into mini-Waterfall with more hip names.

I missed that part of the manifesto
From another thought-experiment-y perspective:

Say you have problem A to solve. Then either one of those is true:

1) it has been solved before, ergo by virtue of software having a zero cost of copying (contrary to, say, a nail, a car, or a bridge), so there is no actual problem to be solved.

2) it hasn't been solved before, ergo it is a new problem, and thus at any moment you may turn a stone and discover something that was not foreseen (whether they are rabbits, yaks, bikesheds, dragons, or what have you eldritch horrors) and thus of unknown cost.

Any task that cannot be obviously fit into one or the other can nonetheless be split into an assembly of both.

Thus any attempt at estimates is as futile as gambling to win, tasks are only ever done when they're done, and "successful estimators" are kings of retconning.

It's all make-believe.

I was with you until this part:

> Thus any attempt at estimates is as futile as gambling to win, tasks are only ever done when they're done, and "successful estimators" are kings of retconning.

> It's all make-believe.

Software estimates are not futile or make believe. They are useful even if they are not always precise. That’s why the industry continues to use them.

This argument proves too much:

"Bloodletting is not futile or make believe. It is useful even if the patient does not always survive. That's why physicians continue to use it."

"Trial by ordeal is not futile or make believe. It is useful even if sometimes Inquisitors reach mistaken conclusions. That's why Inquisitors continue to use it."

"Lottery-number-picking systems are not futile or make believe. They are useful even if some players never win. That's why players continue to use them."

It is a fully general argument which, if correct, would demonstrate that no practice that had continued for a period of time could ever be ineffective or counterproductive.

The amount of tap dancing and philosophizing some developers are willing to do to dodge estimates is hilarious.

It’s a skill… a basic part and critical part of engineering. IME the common thread between objectors is that they haven’t made a consistent effort to improve — developing, iterating, and refining their estimation process over time.

Yeah, every line of code is a unique snowflake piece of undefinable research the universe has never seen, equally unknowable and inscrutably enigmatic. But the workers at EngiCorp building EngiCorp products using EngiCorp project routines and resources first, second, and third quarter of 2025 are literal world experts at EngiCorp outcomes. They very reasonably should be able to estimate EngiCorp work in Q4, and account for EngiCorp realities, providing maps of future costs that can drive EngiCorp process improvement and investment.

If I ask for a decking estimate and get back sophistry and smug incompetence, I’m not talking with a super skilled professional deck builder. Doesn’t matter how they hammer, saw, or draw.

In my experience, they have negative usefulness in general.

I did work at one company that used an Agile/Scrum process, and though we didn't estimate actual time, we did estimate effort to complete tasks. That worked pretty well: we had an idea which tasks would be easy and which would be hard, and used that in sprint planning to more effectively use developer time.

But the key is no one was held to an estimate, since it was only that: an estimate. If something took 2x as long as predicted, oh well: we just adjusted our planning and worked around it, delaying other tasks for instance.

The problem I see is when managers demand an estimate, and then turn the estimate into a deadline and try to hold developers to that estimate no matter what.

> Software estimates are not futile or make believe. They are useful even if they are not always precise. That’s why the industry continues to use them.

The industry continues to fail when trying to use them. They have negative usefulness.

I am in a project where we have to give estimates in hours and days.

Needless to say we always underestimate. Or overestimate. Best case we use the underestimated task as buffer for the more complex ones.

And it has been years.

Giving estimations based on complexity would at least give a clear picture.

I honestly don’t know what the PO and TL gains with this absurd obscenity.

The last director I had would ask "is it a day, a week, a month, or a year" he understood that's about as granular as it's possible to be.

And he really only used them in comparison to estimates for other tasks, not to set hard deadlines for anything.

Knowing nothing else about him, I like him based on this alone.

I've been in planning sessions where someone would confidently declare something would take half a day, was surprised when I suggested that it would take longer then that since they were basically saying "this'll be finished mid-afternoon today"...and was still working on it like 3 weeks later.

This is essentially t-shirt sizing without all the baggage that comes from time. Your boss is trying to use the relative magnitude but it's inevitable that people will (at least internally) do math like "7 day tasks is the same as one week task", or worse over-rotate on the precision you get from day/week/month, or even worse immediately map to the calendar. Suggestion: don't use time.
Here's my observation: ballparking an estimate for a whole project, in my experience, tends to be more accurate than estimating each task and adding them together.

I like to think of this as 'pragmatic agile': for sure break it down into tasks in a backlog, but don't get hung up on planning it out to the Nth degree because then that becomes more waterfall and you start to lose agility.

Hours is insane. But ultimately time is money and opportunity cost. Software engineering can’t be the only engineering where you ask the engineers how much something will cost or how much time it will take and the answer is “it’s impossible to know”. Even very inaccurate estimates can be helpful for decision making if they are on the right order of magnitude
> I honestly don’t know what the PO and TL gains with this absurd obscenity

There are marketing campaigns that need to be set up, users informed, manuals written. Sales people want to sell the new feature. People thinking about road maps need to know how many new features to can fit in a quarter.

Development isn't the only thing that exists.

Another reason is that figuring out what the software to be written should actually do, and how it should work, is work that is part of the project and the time it will take needs to be estimated.

As well as the actual development work that will result, which isn't known yet at the time of estimation.

I call it the "Persistent Incompetence of Software Development", which is another perspective on estimation, focused more on expertise. A chef that cooks pizzas, cooks the same pizza over and over again and becomes amazing at it. If you are a developer that writes the same code over and over, you are terrible at software development. A good software developer should always be solving new problems, as by the nature of software, once they solve a problem, they never solve that (exact) problem again. So we are persistently incompenent.

Which is why software development can't be estimated, as well. Because it is all, as you say, novel. With infinite error bars.

At this point, I can't take anyone seriously that believes software dev can be estimated.

There is very little novel about most B2B CRUD and internal bespoke apps that most developers are doing. The novel part if any is implementing the business vertical logic
"... anybody with any brains has already left town..." -- Bob Dylan

Anybody with any business experience has already isolated themselves from the certainty of software project failure, where "failure" is a euphemism for "late." So it doesn't matter if software can't be estimated.

This can be nerve-wracking to a beginner, but one gets used to it over time.

As I’ve gotten older, I find this to be untrue.

Estimating the unknown is itself a skill.

It’s like choosing to crash random parties everyday — at first, exceptionally novel but facing the unknown becomes itself mundane. Humans adapt. They build mental modals to approach the unknown and these mental modals are predictable.

I just don’t think most people get much practice.

That's the point: how can you tell WHEN you are going to reach a place you've never been before traveling an uncertain path?

Making mistakes over and over again. And adding a lot of time buffers just to be safe

> If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.

But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated

The slow part isn't following a written "log in and run ALTER TABLE" runbook, it's the review of "does this solution make sense given what we're trying to do".

Spot on! Now add one more dimension (from machine learning projects and pretty much other research fields) -- you also never know if the result are going to even work.

Maybe the data just doesn't have the correlation you want. Maybe you just didn't try harder, never know for sure. But owners reasonably demand some estimations.

Repetitive and easy or worth automating are not the same. The fundamental problem is that a setup capable of solving any of the requests that come to it is as complex as just a whole programming language and then you’re back to square one.
As a project manager, it sounds like you're making excuses. Just give me a number, trust your gut!

We have a fundamental failure to communicate, what we're doing. The game project managers and finance believe we're all playing is a regression towards the mean, where everything is additive and rounds up to nice consistent formulaic sums. Whereas software development follows power law distributions. A good piece of software can deliver 10x, 100x, or 1000x the cost to produce it (ex: distribution, cost of delivering another copy of software is near 0 once written). You don't get that sort of upside with many other investments. Finance is happy with an NPV 8% above what they invest in. This means that when software people talk, everything they say sounds foreign, and everyone assumes it's because of jargon. It's not. The fish don't know they're swimming in water. When the fisherman comes, everyone is caught off guard.

So we get what the author talks about

> The estimates stopped being estimates. They became safety railings against being held accountable for unreasonable expectations.

We. Pad. Like. Crazy. Yes this is inefficient. Some project managers recognize this. We get theory of constraints. But rather than cull the layers of hierarchy that lead to the padding in the first place, all the blame for failure goes back to developers. Get hit on the head enough and you will stop acting in good faith and pad to save your ability to feed and cloth yourself.

> As a project manager, it sounds like you're making excuses. Just give me a number, trust your gut!

If we're just making up numbers, why don't you just make it up yourself and save the developers the trouble?

It's not obvious to me that we should avoid padding, or why it's seen as undesirable.
Very insightful. This is also my go-to argument for why software engineering is real engineering.
This is such an interesting observation, makes a lot of sense.
This is going to be immediately downvoted, but whenever I see software developers mention that estimating is uniquely challenging for software it immediately calls out a bunch of red flags. Estimates in software are challenging but no more or less challenging than other industries.

Typically the people that single out software as a unique snowflake in the world of delivery and estimation really just say out loud they have no experience in project management and no experience working outside of authoring code. The things that make estimations most challenging in software are the same factors that make estimations challenging everywhere else.

I don't think it's unique, but many software projects are inherently prone to big surprises you can't catch until you start the work. I would love to keep estimating and estimating and get an accurate number, but it costs money and, eventually, you're just working on the project.
Can you give an actual counterargument?
I don't know how typical teams work, but in my case, new projects always come on top of other obligations. It may take 1 day to add the field, but how many meetings, fires, or other disturbances will happen during that day?
If software developers want to be then seriously as a profession, they need to be able to provide and justify estimates for their work.

Everything you said could apply to a new bridge, building, pharmaceutical compound, or anything else that is the result of a process with some known and some unknown steps.

Pharmaceutical compounds frequently don't make it to market after significant investment.

No one in that industry is giving estimates based on developing brand new drugs - they're giving estimates related to manufacturing lead times, unalterable physics time lines, and typical time to navigate administrative tasks which are well known and generally predictable (but also negotiable: regulations have a human on the other end). All of this after they have a candidate drug in hand.

Same story with bridge building basically: no one puts an estimate on coming up with a brand new bridge design: they're a well understood, scalable engineering constructions which are the mostly gated by your ability to collect the data needed to use them - i.e. a field survey team etc. - and also once again, regulatory processes and accountability.

> Everything you said could apply to a new bridge, building, pharmaceutical compound

"Everything"? So

> predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.

Also applies to bridges? Bridges require a ton of manual human input at every stage of construction, regardless of how predictable and repetitive the work is. With software, we can write software to make those tasks disappear. I've yet to see the bridge that can build itself.

Drug discovery chemists do not, to my knowledge, provide estimates on how long it will take them to discover a marketable drug.