How does Architecture fit with Agile?

[Peter Dawson]

Hi,
I have been working as a non-agile software architect for a number of years. I've started a new project and we've gone for a Scrum approach. The development seems to be off to a flying start and looks good. However, we don't seem to have cracked a couple of bits of work that would normally be called 'architecture'; they are:

- The company has and enterprise strategy and this needs to be fed into the project somehow. However the strategy is not as tangible as a set of functional requirements, so how do you feed it into a scrum?

- The company want our project to utilise a new piece of infrastructure. We really need to prototype it and get it up as quick as possible but there doesn't seem to be a place in the agile process to do that?

- The solutions architect wants the project to adopt several 'ilities' e.g. flexibility. Normally, I'd influence the dev process as it runs along, but the sprint team want to work from requirements - should I write some technical requirements?

- We are working with a 3rd party who are supplying a big piece of the solution. Our integration with them needs work up front to get it sorted because the reality of the relationship with them will make it impossible to refactor the interface later - how do you feed that into an agile process?

Does anyone have any pointers on the best way to address the above and 'architecture' in general. I've read some zealous agile stuff that says that you just code and refactor, but I really don't think that will work for the points above. Any advice much appreciated.

Best
Peter

[James Shore]

Hi Peter,

One of the challenges of Scrum is that it doesn't include technical practices, and yet the Analyze-Design-Code-Test SDLC doesn't work well in a two-to-four week cycle. Something else is needed instead.

This is one of the reasons I like XP (and part of why we focused on XP in the book): it includes specific technical practices to address this problem. Now, this is a big subject so I'm only able to give an overview here. I'll refer you to the book for more.

Many people squeeze the Analyze-Design-Code-Test sequence of phases into Scrum's two-to-four week Sprints. As you're discovering, this doesn't work all that well--it leads to technical debt and often causes testing bottlenecks as well. XP provides a solution, but it's probably quite different from what you're used to.

In XP we prefer not to do any work that doesn't provide customer value. So there should be no technical infrastructure effort that doesn't support current or past stories (aka deliverables)... and stories must be customer-valued, so technical infrastructure stories aren't allowed either.

This requires that all technical infrastructure be build incrementally, alongside stories. To achieve this, XP uses simultaneous phases rather than sequential phases. Analysis, design, coding, and testing are performed simultaneously.

The driver for this effort is test-driven development (TDD). Test-driven development mixes together design, coding, and testing into a single activity. Briefly, it works in very small, repeated steps (each about 30 seconds long) as follows:

Decide on the next, very small increment of functionality

Define behavior and define interface; write test that checks that behavior and interface; run test and watch it fail (about five lines of test code)

Implement interface and behavior; run test and watch it pass (about five lines of production code)

Review design, identify improvements, and use refactoring to implement those improvements; make sure tests continue to pass after each micro-refactoring.

Repeat with next small increment

Now here's the trick. Much of the design improvement comes during the "refactor" step. I call it reflective design: it's a process of reviewing existing code, extrapolating its design, coming up with improvements, and then refactoring to implement those improvements. The design actually comes after the code, which is why it's reflective.

Designing after coding sounds ridiculous, I'm sure. From a phase-based perspective, it would be. But remember that design and coding actually happen simultaneously. It's not a case of spending days coding, then days more fixing the design. It's actually a matter of creating a few lines of code that provably solve a problem, then critiquing that code and how it fits into the overall design of the system and making immediate improvements, which are also verified by your tests.

This probably seems impossibly low-level from an architecture point of view... and it is. But I had to introduce the basic concept of reflective design before I could talk about how design and architecture works in XP.

Test-driven development alone isn't enough. Constantly reviewing your code and refactoring leads to good method- and class-level design, but it doesn't address the larger question of properly partitioning class responsibilities and relationships, or the even larger question of overarching architectural patterns.

Continuously throughout this process, the programmer should also be performing reflective design at the package and architectural level as well. Just as refactoring can eliminate duplication, clarify concepts, and improve design at the method and class level, it can solve problems at the higher level as well. And, if the design is kept simple and the design is done continuously, the cost of these changes is low.

Thinking at multiple levels simultaneously is difficult, which is one of the reasons pair programming is valuable in XP. While one person types and thinks about the tactical issues in the current class, the other person is thinking ahead to the next set of tests and considering how the changes the pair is making influences the overall design. Is there are an opportunity to improve relationships or responsibilities? Is the current architecture sufficient to solve the problems the pair is seeing?

There's one more element: risk-driven architecture. Architectural issues are often cross-cutting and difficult to change. For example, if your code doesn't support internationalization, then adding it after the fact could be difficult.

Risk-driven architecture looks ahead to potentially risky architectural problems (such as internationalization) and directs refactoring efforts to make those risks go away, without actually implementing the speculatory feature. In other words, if your customers don't want any localization yet, you wouldn't internationalize yet. However, you might improve your design so that there was only one class to internationalize when a localization story finally did come along.

I'm condensing 45 pages of material into a few paragraphs here, so I'm afraid this still may not sound feasible. I guess you'll have to buy the book if you're still not convinced.

To answer your specific questions:

- Enterprise strategy: If you have a overall architecture or strategy that you have to plug into, it's a constraint on your development process that team members should keep in mind as they work. When you discuss design issues as a team, this should be part of those discussions; when you do test-driven development, this should influence your decisions.

- Delivering new infrastructure: This is largely incompatible with the idea that all deliverables should be valued by your on-site customers (the business experts on the team). You can engage in some tortuous logic that says that, since the infrastructure is important to the company, it's customer-valued, but that usually doesn't fool anybody. Often in this situation there's a schism between what the business experts care about and what the technical organization cares about... perhaps because the technical organization has not yet grokked delivering technical infrastructure incrementally.

You can deliver technical infrastructure incrementally using reflective, continuous, incremental design as I described above. However if people don't believe it's possible, they'll want the technical infrastructure to be developed all at once. I would push back against this, personally, but that's because I know how to do it incrementally. If I wasn't confident in my ability to deliver infrastructure incrementally, I might ask for it to be a "technical story"... but expect a lot of flack (and rightfully so) from your business experts. Introducing technical stories and insisting that they come first upsets the balance of power in agile planning: estimates are supposed to come from developers, priorities are supposed to come from business experts.

We can talk about this further if you like. There's a lot of context here.

- Ilities: There are a lot of different kinds of ilities. Scalability, Stability, and performance...ility are customer facing and can be scheduled with stories. "Flexibility" and "maintainability" are more abstract. In XP, however, flexibility and maintainability are built-in to the process. You're constantly improving your design, which constantly improves flexibility and maintainability.

One thing that isn't done in XP is speculative flexibility. In other words, you don't code in hooks or plug-in points for things you don't currently need. That's because, in XP, you're constantly improving the design. Change actually gets easier over time, not more difficult, so adding code now is more expensive than adding it later. Also, speculative flexibility is sometimes wrong, and when it is, it's often difficult to fix because too many things are depending on it.

This idea--no speculative generality--can be difficult for people to swallow. One of the things that separates good designers from poor designers is their ability to see and implement abstractions. You're still allowed to see the abstractions; we just ask that you delay implementing them until there's an actual need. Sometimes waiting will lead to new information that allows you to see simpler and more powerful abstractions. (Going through this process--seeing how my abstractions were simpler and more powerful if I waited to introduce them--is what convinced me that incremental/evolutionary design was feasible.)

- Third-party solutions: I'd have to know more about your situation to comment on this. In general, most third party code can be isolated behind an interface (or interfaces) that you can refactor.

[Ilja Preuss]

Martin Fowler has an interesting definition for architecture: architecture is the set of those things that you need to decide on early because they are hard to change later.

If you use that definition, one of the goals of an Agile team is to reduce the amount of architecture it needs, by making as many decisions reversible as possible.

See http://martinfowler.com/ieeeSoftware/whoNeedsArchitect.pdf

[Peter Dawson]

Thanks Guys, that gives me a lot to think about. I probably need to study up on XP a bit more and consider the same things again afterwards. The risk-driven architecture idea and the delayed abstraction idea both stand out as really useful, so will have to get into them.

RE: Delivering new infrastructure: This is a situation that I have encountered on many projects. It usually runs as follows - the company wants to move the technological platform forward in a certain direction but the budget is largely (or totally) allocated to specific projects rather than to strategic work; thus sterategic work like building new databases or adding new servers gets nibbled off the project budgets. If you completely separated the strategic work from the tactical work then it seems impossible to create business-focussed stories for the strategic bit; merging the two seems like a corruption, but appears to happen a lot in practice, so maybe it's the least worst option. Personally, I have found that making small, achievable prototypes of new infrastructure and squeezing them into the projects works a bit but its not ideal.

RE: 3rd party : Basically, the interface definition requires discussion and agreement - another common scenario IMO. You suggest that you can refactor behind the interface, but you still seem to need the interface up front. Working to an interface that is changing is a nightmare IMO, so doesn't the interface defintion work need addressed non-agile?

[James Shore]

Hi Peter,

The issue of paying for infrastructure is an important one; often there's a way to proceed that will work and is cheaper in the short run but sacrifices maintainability. This leads to technical debt, of course.

The XP position (and mine as well) is that technical debt is so costly that we simply do not tolerate it at any time. Part of the balance of responsibilities is that, although business experts are responsible for priorities, technical experts (programmers) are responsible for costs and estimates. As programmers, we have a professional responsibility to deliver maintainable systems, except in those very rare instances where the software is truly not going to be maintained or developed for more than a month or so. (And when does that ever happen, really?)

As a result, I don't give business people a choice about technical infrastructure; I simply include it in my estimates as part of the cost of doing business. When challenged, I explain in my best non-confrontational "you hired me to make these decisions" manner: "This story seems more expensive than usual because it requires us to upgrade the database, which is necessary to achieve the performance you've asked for."

Incremental design/architecture is crucial to this work because it allows you to split the technical infrastructure work into small pieces that are spread (more or less) evenly over all of the stories. As a result the customers don't see big technical infrastructure hits and are less likely to be spooked. From their perspective, every single story is focused entirely on delivering value.

Learning how to split infrastructure so evenly and incrementally takes time, but it's a valuable skill that's well worth learning.

Regarding the third party code, there are ways to make this work and still be agile. A good resource is Eric Evans' discussion of "strategic design" (which covers multiple ways of interfacing multiple teams) in his book Domain-Driven Design. I think it's chapter 14.

[Stan James]

Robert Martin's blog post Architecture is a Second Order Effect is another good view on agile architecture.

[Peter Dawson]

I think I see what Rob Martin is saying, but I don't buy it personally. The XP theory that you can achieve everything empirically seems to be a rehash of the original theory of the empiricists:
http://en.wikipedia.org/wiki/Empiricism

Whilst, i'm a big fan of empiricism, you can't achieve everything in life that way and the debate about the limits of empiricism has been had long and hard historically. In reality, XP-ers must either:

- have an architecture in their heads which they use to make day-to-day value judgements about which direction to go; they simply don't record that concept into a format that someone else can understand

or

- don't know where they are going until they've got there

Maybe there's other options - I hope so? - I guess that was my original question??

[Stan James]

I think the "architecture in their heads" bit may be a big factor. I worry that it might require a great mental catalog of past successes and failures to go this route successfully.

On the other hand, we don't have an architecture just in our heads. We have one in code from day one. Everything has a structure, whether or not it's planned, beautiful, worth keeping, etc. I think we can grow an architecture from rude beginnings if we're careful enough to build the right abstractions and isolate decisions so they are rarely expensive to change. But maybe that's exactly where that heap of experience comes into play.

I just reminded myself of a good video of Martin Fowler and a panel of Thoughtworks architects on Modifiable Architecture. Some good success and failure stories about change late in a project.
[ November 02, 2007: Message edited by: Stan James ]

[Peter Dawson]

Thanks. I like Fowler's take on Agile.

I'm starting to think that development without a-priori architecture + design is an aspiration rather than a reality. You're right that the code is the architecture at any moment in time, but the decisions about where to take it next come from somewhere. I'm surprised that the 'somewhere' hasn't been explored more as it seems to be a significant gap in agile. Having uncertain bits of the agile process also seems to be a bit of a risk, as its hard to know whether they are going to work out during the project or not - at least waterfall is a known (albeit limited) methodology. A number of writers on this forum has suggested that the gaps in Agile make it harder to get it adopted - I think I'm in agreement with them.

Maybe the idea is that the plan isn't written down and the team simply establish a common understanding amongst themselves; keeping the vision in RAM, makes the development free, agile and light - like a group of free jazz musicians jamming. This is great when it works but frequently won't, so then what's the plan? If a group of musicians don't establish a common understanding of where they were going, you can hear it immediately - how can you tell if a group of developers have got that shared understanding and what do you do if they havn't?

[Ilja Preuss]

Originally posted by Peter Dawson:
You're right that the code is the architecture at any moment in time, but the decisions about where to take it next come from somewhere. I'm surprised that the 'somewhere' hasn't been explored more as it seems to be a significant gap in agile.

I'm not sure why you perceive it as a gap in Agile.

It seems to me that the architecture comes from the experience of the team members, informed by discussions of the needs, well known design principles and feedback from the code. What Agile puts into the mix is plenty of the latter and a strong focus on principles such as Once And Only Once.

Having uncertain bits of the agile process also seems to be a bit of a risk, as its hard to know whether they are going to work out during the project or not - at least waterfall is a known (albeit limited) methodology.

Quite to the contrary - with waterfall, you won't know until the very end whether the project will be a success or total failure. With an Agile approach you will get your first feedback after a week.

A number of writers on this forum has suggested that the gaps in Agile make it harder to get it adopted - I think I'm in agreement with them.

I'm not sure what gaps you are referring to. Could you please elaborate?

Maybe the idea is that the plan isn't written down and the team simply establish a common understanding amongst themselves; keeping the vision in RAM, makes the development free, agile and light - like a group of free jazz musicians jamming. This is great when it works but frequently won't, so then what's the plan? If a group of musicians don't establish a common understanding of where they were going, you can hear it immediately - how can you tell if a group of developers have got that shared understanding and what do you do if they havn't?

You can hear it immediately from the jazz ensemble because you continuously observe the end product - the music.

Agile strives to do exactly the same: make the end product observable continuously. Week after week, an Agile team delivers a running, fully integrated, tested, deployable system. If the developers don't have a shared understanding, that will become visible quite early.

In contrast, waterfall doesn't guarantee a shared understanding, either, but it also defers feedback for very long. Often enough you only find out that there wasn't shared understanding after months of development, when the developers for the very first time try to integrate their parts to get a working system.

[Peter Dawson]

Ilja,
I can believe you when you say that

It seems to me that the architecture comes from the experience of the team members, informed by discussions of the needs, well known design principles and feedback from the code. What Agile puts into the mix is plenty of the latter and a strong focus on principles such as Once And Only Once

. However, other people say different things and even with your definition, its very hard to measure/test. One of the big things in Agile development is for code to be testable, yet the architecture you're describing is completely un-testable and its really an act of faith that it exists at all. I was hoping that there would be an Agile technique that describes how to derive a measurable, testable, definable architecture - I haven't found that yet, hence the comment that there is a gap. Using traditional, 'hulking' techniques does at least create a measurable, testable architecture that's easy to locate e.g. it's in the CASE tool or 'just ask Jim' etc.. Having a measurable architecture makes it a lot easier to know whether a project is going to achieve a strategic goal for example. I'm not sure if Agile has nailed areas like strategy or cross-team dependencies. It might be the case that Agile buys ease and certainty for developers by making life hard for designers + architects - that might not be a bad thing, but it would be good to get that straight if that is indeed the case.

As far as the music analogy goes: just 'cos a group can jam the first few chords together, doesn't mean that they can play well for an hour or two. 99.9% of music is played to some sort of script/sequence/tab.

[Ilja Preuss]

Originally posted by Peter Dawson:
However, other people say different things and even with your definition, its very hard to measure/test. One of the big things in Agile development is for code to be testable, yet the architecture you're describing is completely un-testable and its really an act of faith that it exists at all.

There *always* *exists* an architecture, of course - else there would be no system.

The question then is whether the architecture is appropriate and well-understood, I guess?

I was hoping that there would be an Agile technique that describes how to derive a measurable, testable, definable architecture - I haven't found that yet, hence the comment that there is a gap.

I'm not sure what you mean by those terms in regard to an architecture, but let me try:

- measurable: I guess you could use some high level metrics on the code to measure the quality of the architecture. Distance from the Main Sequence for modules comes to mind: http://www.objectmentor.com/resources/articles/stability.pdf

- testable: one purpose of an architecture is to support the extension and maintenance of the system - a good test for this would be whether the team is able to maintain a stable development velocity. Another purpose might be to allow for a certain performance, portability etc. I guess the tests for those are more or less obvious.

- definable: many teams maintain high level UML diagrams or similar of their architecture - often on a white board. Some teams use metaphors to describe their architecture.

Using traditional, 'hulking' techniques does at least create a measurable, testable architecture that's easy to locate e.g. it's in the CASE tool or 'just ask Jim' etc.

I don't understand how that makes it measurable or testable.

And my impression was that most traditional team regard their architecture as "a big binder of UML diagrams" or something similar - and there is in fact not guarantee at all that the architecture depicted in those diagrams is the one that got implemented.

Having a measurable architecture makes it a lot easier to know whether a project is going to achieve a strategic goal for example.

How does it make that easier? (I hope that your answer to that question helps me understand what you are getting at.)

I'm not sure if Agile has nailed areas like strategy or cross-team dependencies.

Well, there are whole companies run in an Agile fashion, so I guess they must have nailed it. Don't have enough experience to comment, though.

It might be the case that Agile buys ease and certainty for developers by making life hard for designers + architects - that might not be a bad thing, but it would be good to get that straight if that is indeed the case.

Agile teams actually strive to more or less negate the difference between those roles. Every developer codes, designs and architects, often in parallel. Although there certainly are some that are more talented in one area than the other. The solution is *very* close collaboration.

As far as the music analogy goes: just 'cos a group can jam the first few chords together, doesn't mean that they can play well for an hour or two. 99.9% of music is played to some sort of script/sequence/tab.

The same holds true for a fully scripted play with a conductor - you won't know whether the group can play together for the whole play until the play is over.

The trick of an Agile orchestra would be to separate the composition into many small parts that can stand on their own. The orchestra would first record the most important, say, minute, and then search feedback from the audience. If the feedback is promising, they record the next minute and put the pieces together, again getting feedback from the audience. And so on. Of course all they time they incorporate the feedback of the audience in what they produce next. From time to time they might decide to redo a just recorded part.

When they recorded enough pieces to make a marketable composition, they press a CD and release it.

As soon as it becomes obvious that there is no value in continuing (for example because the audience becomes bored), the project can be stopped - and the output of the project is a releasable, piece of music - the most valuable piece this orchestra could come up with in the allotted time.

Well, in fact I find it hard to imagine how that could actually work with music. I can tell you that it *does* work with software.

[Peter Dawson]

I think the point I was driving at is that there needs to be a mechanism to have certain defined design/architecture in advance of the coding. However, I get the impression that advanced planning is 'not allowed' in pure Agile and that everything has to evolve organically. It is this issue that I don't buy. This seems dogmatic IMO and doesn't map to the projects I have worked on. Even on the Agile projects, people _do_ plan ahead.

For example, it might be necessary for a system to implement something in a particular way due to some strategic requirement. In that situation, it seems necessary to me to put that requirement (possibly expressed as a design) on the table in advance of it being coded. If the development team are unwilling or unable to follow that requirement then that is an issue, but the fact remains that if something is needed, it needs to be stated. Stating the strategic goals and working to get them achieved makes is easier to achieve them than leaving it to chance that these things will happen organically.

You suggest that it is OK for teams to have high level UML diagrams or similar - I guess that would be a way to detail things that need to be achieved so long as it is allowed for these things to be produced in advance of the coding.

I don't disagree that there are a lot of places that use Agile succesfully and I have worked on some myself. However, IMO the need for advance planning of architecture and design doesn't go away, all that happens is that the most dominant individuals maintain a mental plan, which they feed into the day-to-day working of the team(s). Sometimes this is great, sometimes not. Personally, I'd like something less alpha-male and more systematic, although I can understand that 'systematic' might cramp the freedom sought by the Agile movement.

To return to the music:
I bought Miles Davis, 'Bitches Brew' recently. On first listen, it appears to be one of the least structured things I've ever heard. On the paperwork that came with the CD there is a photograph of the structure of the title track, which illustrates beautifully and in great detail how the tune was to be constructed. Apparently, Davis handed out this 'architecture' just a few minutes before the band played it. The players were so good and Davis' vision so clear that the recording was made the same day and is a classic. This kind of 'architecture' was what I had in mind. Sorry if I don't have a better explanation - I was trying to find an agile/software version of what Davis did.

[Ilja Preuss]

Originally posted by Peter Dawson:
I think the point I was driving at is that there needs to be a mechanism to have certain defined design/architecture in advance of the coding.

And I think the point *Agile* is driving at is that that's probably less true than you think.

However, I get the impression that advanced planning is 'not allowed' in pure Agile and that everything has to evolve organically. It is this issue that I don't buy. This seems dogmatic IMO and doesn't map to the projects I have worked on. Even on the Agile projects, people _do_ plan ahead.

I don't think that Agile can allow or disallow something, let alone people to think up something in advance.

I guess what it comes down to is the Lean principle of deciding at the last responsible moment: http://www.codinghorror.com/blog/archives/000705.html

Having said that, what we do now and then is sit together with the whole team for half an hour to come up with an architectural idea - for example when for one customer we had integrate a scripting API into our product. It's still important to do this immediately before it's needed and to validate the ideas by coding them.

For example, it might be necessary for a system to implement something in a particular way due to some strategic requirement. In that situation, it seems necessary to me to put that requirement (possibly expressed as a design) on the table in advance of it being coded.

Well, if it's an important requirement with high business value, an Agile team would naturally start to work on it early, before much of the architecture had been settled.

If the development team are unwilling or unable to follow that requirement then that is an issue, but the fact remains that if something is needed, it needs to be stated. Stating the strategic goals and working to get them achieved makes is easier to achieve them than leaving it to chance that these things will happen organically.

I don't know of any Agile approach that suggests to not tell the developers things they might need to know.

You suggest that it is OK for teams to have high level UML diagrams or similar - I guess that would be a way to detail things that need to be achieved so long as it is allowed for these things to be produced in advance of the coding.

There is nothing in Agile that would hold a developer taking a short time of sketching a design idea before starting to code.

IMO the need for advance planning of architecture and design doesn't go away, all that happens is that the most dominant individuals maintain a mental plan, which they feed into the day-to-day working of the team(s). Sometimes this is great, sometimes not. Personally, I'd like something less alpha-male and more systematic, although I can understand that 'systematic' might cramp the freedom sought by the Agile movement.

I don't think that "freedom" is the right word.

What it is about is increased flexibility and reduced waste by delaying making decisions as long as possible.

To return to the music:
I bought Miles Davis, 'Bitches Brew' recently. On first listen, it appears to be one of the least structured things I've ever heard. On the paperwork that came with the CD there is a photograph of the structure of the title track, which illustrates beautifully and in great detail how the tune was to be constructed. Apparently, Davis handed out this 'architecture' just a few minutes before the band played it. The players were so good and Davis' vision so clear that the recording was made the same day and is a classic. This kind of 'architecture' was what I had in mind. Sorry if I don't have a better explanation - I was trying to find an agile/software version of what Davis did.

I guess that's where the analogy breaks down for me. In my experience, most software systems today are so complex, and depend on so many details, that it's impossible for a single mind to make all the important decisions upfront.

[Stan James]

From Peter ...

I think the point I was driving at is that there needs to be a mechanism to have certain defined design/architecture in advance of the coding.

That's a pretty key point that we can probably all agree on if we abstract it out far enough and realize we disagree only on how much of it is designed, in how much detail and how early.

Agile folks have found we can be more productive and develop better designs when we design only as much as we need right now.

I think the fear that non-agile folks have is that we will overlook something important and paint ourselves into a corner where we can't accommodate the next requirement. Then we'll have to back up, change something core and redo a lot of work. So we should eliminate this risk by doing more up front work.

How can doing just enough stuff as late as possible be more productive? For one thing, we learn continuously, so we know more about everything - the technology, the business, the people - today than we did yesterday. What happens when we make decisions at the time when we know the least? We guess! We think "This might come up" so we design for it. In a desire to think of every possibility and eliminate risk, we can make stuff up for days on end. Been there, done that. What if it never comes up? We wasted time and made the design more complex for nothing. Done that, too, made a framework (or two) with all kinds of nifty features nobody ever needed.

What if we design and build only what we need right now? We have the best possible knowledge about what's needed and we avoid building anything that is wasted. Nice.

How do we avoid the dead end and rework problems? That takes some real skill. We try to isolate big decisions that might impact a lot of code and hide them behind abstractions. We can't make an interface and factory for every darned thing from the start, but we can manage dependencies so it's easy - with powerful IDEs - to introduce them later.

How much do we have to do up front? The answer is not zero, I'm pretty sure. Agile methods often suggest an architectural spike, a small number of key people building a tiny slice of the system from end to end. I started to list out the kinds of decisions they might make this early, but every one of them could be deferred. We need just enough to bring in the full team and say "Here's where we're headed right now and here's how to participate."

I used to end little ad-hoc design sessions by asking if everybody knew enough to stay busy for two days and feel safe about the rework risk. We can always huddle again then.

Any of that ring true?

[Scott Ambler]

As much as the discussion of "pure agile" is interesting, it's mostly rhetoric. At Dr Dobb's Journal (DDJ) we run surveys to actually discover what's going on in practice, as opposed to what we're being told is going on by the methodologists. The surveys reveal time and again that there is often a very big difference. What we discovered was that the vast majority of agile teams do in fact do some up front requirements and architecture modeling (78% and 77% respectively). It's important to note that it doesn't imply that they're writing huge docs or overbuilding, it's just that they're doing enough modeling to think things through. It's not a black and white world, you can in fact gain significant benefit from modeling without taking the hit of needless documentation. The source of the survey, including the data, can be found at http://www.ambysoft.com/surveys/agileMarch2007.html . A summary of the modeling adoption rates are posted at http://www.agilemodeling.com/essays/initialRequirementsModeling.htm#AdoptionRate (and a few other spots on the web for that matter).

You might want to take a look at the concept of Agile Model Driven Development (AMDD) at http://www.agilemodeling.com/essays/amdd.htm which describes how modeling fits into the overall agile picture. AMDD is one way to scale TDD (my Feb 2008 column in DDJ will cover this in detail) as well as the XP concept of On-Site Customer and the Scrum concept of Product Owner (my January 2008 column covers this and will be online the first week of December at www.ddj.com).

I also have some detailed thoughts about agile project architecture at http://www.agilemodeling.com/essays/agileArchitecture.htm and enterprise architecture at http://www.agiledata.org/essays/enterpriseArchitecture.html

- Scott

[Peter Dawson]

Thanks for your comments folks - its been really helpful to knock a few thought around on this topic.

[Ilja Preuss]

Originally posted by Scott Ambler:
As much as the discussion of "pure agile" is interesting, it's mostly rhetoric. At Dr Dobb's Journal (DDJ) we run surveys to actually discover what's going on in practice, as opposed to what we're being told is going on by the methodologists.

On the other hand, I'm quite positive that most of those "methodologists" are actively doing what they teach.

It's also a fallacy to infer from the number of teams doing something how good that practice is.

What we discovered was that the vast majority of agile teams do in fact do some up front requirements and architecture modeling (78% and 77% respectively).

And the other 20+% are doing *zero* of it? That's really hard to believe for me.

I'd rather guess that those numbers reflect a different understanding of what it means to do or not do those things.

[Scott Ambler]

Granted, which is why we also look into the effectiveness of various practices as well. In the Agile Adoption Survey we found that whiteboard sketches where the fourth most valuable work products on agile teams (working software, source code and tests were the top three) out of 19 artifacts that we asked about.

Yet, the rhetoric around developer testing is easily an order of magnitude greater than around agile modeling techniques. I suspect it has something to do with the codeing culture that dominates most agile discussions.

I believe that we continue to hobble ourselves by not discussing what we actually do in practice and instead focus mostly on the "really cool" stuff that we prefer to talk about.

- Scott

[Ilja Preuss]

Originally posted by Scott Ambler:
Granted, which is why we also look into the effectiveness of various practices as well. In the Agile Adoption Survey we found that whiteboard sketches where the fourth most valuable work products on agile teams (working software, source code and tests were the top three) out of 19 artifacts that we asked about.

I can easily connect to that.

I'm not sure that whiteboard sketches are what Peter had in mind, though.

Yet, the rhetoric around developer testing is easily an order of magnitude greater than around agile modeling techniques. I suspect it has something to do with the codeing culture that dominates most agile discussions.

I can think of at least two additional reasons:

- testing seems to be harder/more complex - it feels like there is more to discuss, and

- techniques like TDD are easier misunderstood and/or questioned