Originally posted by Bert Bates:
Such a foundation often requires a fair amount of upfront analysis and prototyping to build a development model. The development model often relies upon specific design patterns and entity relationship diagrams (ERD). Without this upfront foundation, Iterative development can create long term challenges that are significant in terms of cost and quality.
I also think this isn't truly an Agile vs non-Agile issue, but there is a kernel in here that has a bit of traction in terms of up-front versus iterative, but I think is really something else in disguise. I'll get to the something in disguise later. Long post, bear with me.
I don't personally object to the up-front model, but I object to the notion that the up-front work is invariably near-flawless. That is the fundamental difference I think between the up-front versus the iterative approaches, and it isn't an issue of textbook definition of their tasks, but of the behaviour you see in people (at least in large organizations) when something goes wrong.
When you work iteratively, it isn't a big surprise if you hit a problem and go "oops, we goofed, we have to really rethink this widget implementation before adding the next feature; time to haul out the heavy-duty refactoring plugin...". The problem may be unwanted, but in general why would people freak out over the need to change current code when changing current code is a routine situation anyways?
The up-front situation is different, and not necessarily because of the flesh-and-blood human being (i.e. another developer, so we should sympathize) wearing the architect hat. Anybody truly skilled enough to be an architect should be equally capable of coping with technical problems when it turns out the up-front work missed an issue. The true problem is often going to be with the people around the architect... even with developers around the architect. The unstated problem for up-front design is that some organizational dynamics make it near impossible to identify and take corrective action for big problems (sometimes even for small problems).
For companies *without* that kind of blame-averse/risk-averse dynamic, they could probably make solid cases for why up-front design works for them. Even if they don't talk about iterating, when the design breaks down, they'll end up with an iteration (which they may call "the next release"). Alas, for companies *with* the negative blame-averse dynamic, they'll stick with the design. Yes, the design now known not to work. The implementation code will duck and weave and contort until eventually, if everybody is really lucky, somehow something approximating the original design is achieved. It may be slow as a pig, cost 10 times as much to write the code and QA it, and require 10 times as much hardware to run it, but it will be done.
As a result of all that, Agile people can make a very legitimate case that for blame-averse organizations, the risk of up-front design is even higher than it is for iterative development, which tends to hit larger numbers of small (low anxiety) issues, not small numbers of show-stopping (high-anxiety) issues.
And now for the something in disguise, something I learned from a really smart team lead/boss I had once upon a time. Developers who spend a lot of their career working in the I/T and systems integration space think differently than those who work primarily in the application development space. Not good versus bad, just different. His view, I've come to find that he was correct, and that the issue matters for some classes of project.
If all your coders on a project come from the application space, iteration can spend a lot of time before somebody realizes that the project objectives are literally impossible because other things in the organization render the objectives impossible. This is particularly true when nobody does even a quick first pass at a data model to figure out who *outside* of your project will depend on your data, or vice versa. This isn't an Agile vs non-Agile issue, this is a "your skills reflect your experience" issue.
However, since these problems do come up, now the up-front argument seems to have legs. Now you can make the case "but if you did an up-front design and it went through all the organizational review processes, we would have obviously spotted that this project was impossible". It is a correct statement because often such reviews can spot problems, however, I don't think this indicates a fundamental limitation in the Agile/iterative approach because reviews aren't the *only* way to spot such problems. I think it just means that in such situations, for an Agile team, the team should have somebody working with them who knows about issues the other team members haven't encountered before or aren't in the habit of caring about.
If you include an integration-savvy person who has a hot list of issues they always watch out for, they'll rush to spew them out in the very first meeting of the Agile team. Those issues should end up on some Scrum list somewhere, and the nitty-gritty details dug into as appropriate. Problem solved.
[ March 19, 2006: Message edited by: Reid M. Pinchback ]