An up-front design is sometimes necessary and valuable, in both agile or waterfall projects, but it always carries a risk of including mistakes or mistaken assumptions that result in dangerous flaws.
When such a flaw is discovered, especially late in the project, it is often very expensive and difficult to fix, because it has become foundational to the solution. This is true regardless of whether an waterfall or agile process has been adopted, and there will often be significant pressure to implement a hacky fix, instead of the truly correct one.
Where agile differs from waterfall is that it does, at least, philosophically accept that nothing is set in stone, even at a late stage. Discussions regarding how to address a flaw can then proceed on a basis of cost vs. risk vs. difficulty, and a hopefully rational choice can be made.
In contrast, the rigidity of the waterfall process is frequently used as a bureaucratic weapon to prevent these discussions from even happening. Decisions made in the project's initial stages are held to be settled and unalterable, and those seeking to overturn them usually have to fight a horrendous, uphill battle to do so, even when they're in the right.
I would liken the difference between design in agile and waterfall to that of scientific theory vs religious dogma. Neither necessarily has a monopoly on truth: A theory might be mistaken while dogma, by chance, might be correct. And challenging well-established scientific theory can, in practice, be almost as difficult and painful as disputing dogma. But science at least accepts the possibility of change, however hard it may be.
So much this. I can't count how many times I had a client tell me "We want the delivery on the proposed time, on budget, I don't care" even when analysing the client's design it shows obvious flaws and won't work as intended. Most of them usually gets fired from doing this, months later, but even so they just get another same-level job on another company, and the cycle goes on.
There's no development methology that will ever work on irrational business demands, and that happens all the time. Sometimes I think HN's users are just too entrenched at product-driven companies, or B2C ones, and never get to experience the amount of BS and CYA that happens on B2B, large-business deals.
In the end no matter what you do software and hardware development is unpredictable. No methodology will ever change that.
This is especially true with estimates. New work, new problems, unexpected difficulties, etc.
Over-planning can lead to a lot of rigidity in the process if it tries to plan too far ahead.
Or they try to compensate for technical problems by doing endless replanning meetings instead of spending the time on solving the problem.
But a good long term road map is really helpful when you have to make a decision to go one way or the other. Just accept that it’s a rough guideline and not an exact plan that’s set in stone.
This a false straw man argument. One the project plan is created, it should be updated regularly (daily/weekly depending on the size of the project) as tasks are completed and new information comes to light. The plan is to give a prediction for the likely end date and cost of the project, but must be update as new information is known.
Welcome to my world :). Just a few weeks ago I had a conversation with a project manager explaining that no amount of additional planning meetings, motivational speeches or free pizza can change reality.
I'm not saying those project managers don't exist, but I wish people wouldn't compare perfect agile against pathological waterfall.
Agreed. I think some people think agile means no plan, whereas I think it means frequent scheduled review of the plan as work is completed and more is learned.