Jeff and Tim...a question about Agile and good design

[Tom Fulton]

Please excuse my ignorance in advance. I have some exposure to agile (XP and Scrum) through co-workers, but haven't participated in an Agile process personally in production. My question is this: does agile, in your opinion, tend to focus on functional code production to the detriment of good design? In other words, does the focus on immediate functional deliverables make the average developer spend less time on quality design? I find that good design often takes more time and thought than seems possible in agile iterations. I'm happy to be proven wrong, by the way...it just seems to me that good design might suffer.

Thanks in advance for your thoughts on this.

[Ilja Preuss]

Hi Tom,

quite to the contrary, Agile is about continuous attention to design. The only way to be successful while embracing change is to keep the design simple and adapt it to new circumstances. A good Agile devoper never stops thinking about design issues.

[Sonny Gill]

That is a great question Tom.

I think I see a difference in what is intended and what happens in practice.

In practice, when a new project is started with Scrum, many teams seem to focus on quickly delivering functionality without spending sufficient time on "just enough" design. This may partly be because the team is trying to convince the customer or the client that Scrum works, and Agile is the way to go. The only thing that the customer understand is the functionality being delivered, hence this disproportionate emphasis on quickly reaching a high velocity.

I am really interested in hearing what Jeff and Tim have found in their practice too...

[Jeff Langr]

Greetings Tom,

As usual, Ilja is dead on.

Your question is extremely important. I view the impact of very short iterations as potentially devastating--they can exacerbate and highlight whatever challenges you may have.

In agile, the customer is allowed to change focus as their needs and the needs of the marketplace change. This allows them to steer software so that it remains competitive. But that kind of change can create havoc on a code base, as developers try to shoe-horn in the new features week after week, particularly in trying to meet fast and continually-coming iteration deadlines.

It's concerns like this that Tim and I try to touch on throughout the cards. We don't think you'll be successful if you don't do what Ilja says, and continually pay attention to design. We cover practices such as TDD, pair programming, continual refactoring, and simple design, all of which are about trying to ensure the design is as clean as possible for the current set of features in the system. (We also relate these practices back to the underlying concern of design--it's possible to do things like TDD and pairing and not understand how they impact design quality.)

Scrum doesn't officially talk about these technical practices. The sensible Scrum proponents will say that you absolutely have to consider good software "engineering" practices in order to sustain system quality (and the ability to keep delivering at a consistent rate) over time.

One more way to look at this, and then it's time to hit the dentist. In our view, to be successful in agile, you must consider design at all times. We do enough design up front to provide a roadmap, but not low-level detail that we know will change as we learn and as things change. During every iteration/scrum planning session, we discuss design, sometimes explicitly and sometimes implicitly. We break stories into tasks, which requires some design considerations. As we code, we first write tests, which drives decoupled and cohesive designs (mostly because such a design is easier to test). Since we have TDD-created tests, we are afforded opportunities every few minutes to examine the small amount of changes we just made to the design. We refactor the code, looking at every possible design element we are aware of (including Kent Beck's four Simple Design rules, which are covered in the deck, and including other design heuristics and principles such as SOLID). Since we have the tests that tell us immediately if we messed up, if we just created deficiencies in the design, we fix it, instead of fearing making a change.

Jeff

[Tim Ottinger]

Jeff and I are sticklers for design, and to help people maintain good internal design, we are presenting The Big Four at the Agile In A Flash blog, which is really just indexing a four-article searies at Pragmatic Bookshelf Magazine (a very good magazine, and better every month it seems) on the primary concerns of software development and how agile practices help us to maintain a good design.

The enemy, as given, is the idea of adding features by slapping them on the outside of the system, perhaps with a flag and a few conditional statements. This is a very "junior maintenance programmer" way of working a system. Agile expects constant attention to design, and enables it, and expects that practitioners will not be doing slap-dash work. \

I can also recommend the Software Craftsmen groups for teaching clean practices, and Robert Martin's book Clean Code (to which Jeff and I were contributors).

[Tom Fulton]

Thanks, everyone, for your thoughts. I think my limited and second-hand experience revolves around the differences between theory and practice (which, as they say, in theory is the same, but in practice is not).

Here is a specific example, on which I was called by a "Certified Scrum Master". I noticed in a group of classes that some code was doing type-specific functionality in an if/else kind of structure, and observed that it was an opportunity to use the Template Method pattern to allow the subclasses to "do their thing". The overall structure was defined at the superclass level, and each subclass had the ability to perform one or more steps in the algorithm according to their type. It was, to me, a classic case of using a Design Pattern to clean up an obviously bad structure in the first place, and to provide a good basis for future changes...and I think of Design Patterns as kind of "tactical" design tools, as opposed to "strategic" design decisions. I estimated that it would take about an hour to code and maybe half that to test. Anyway, the Scrum Master made it very clear that we should not do anything that was not specifically called for as part of a deliverable, and that the definition of the deliverable was to be viewed as that of functionality, not (as he put it) "nice to haves".

So where does the concept of design changes that DON'T specifically change functionality fall in the agile spectrum? More than one person has told me that the "Certified Scrum Master" was just plain wrong in his judgment on this issue, but honestly I see his position as being right in line with the overall philosophy of agile. My position was simple: the original code was just bad, by almost anyone's measure. And yet it worked, if in a really nasty and hard-coded way. Again, my exposure and experience is quite limited, so please help me understand when and where non-functional design changes fits in Scrum or XP iterations.

Thanks all!

[Tim Ottinger]

Tom Fulton wrote:
Here is a specific example, on which I was called by a "Certified Scrum Master". I noticed in a group of classes that some code was doing type-specific functionality in an if/else kind of structure, and observed that it was an opportunity to use the Template Method pattern to allow the subclasses to "do their thing". The overall structure was defined at the superclass level, and each subclass had the ability to perform one or more steps in the algorithm according to their type. It was, to me, a classic case of using a Design Pattern to clean up an obviously bad structure in the first place, and to provide a good basis for future changes...and I think of Design Patterns as kind of "tactical" design tools, as opposed to "strategic" design decisions. I estimated that it would take about an hour to code and maybe half that to test. Anyway, the Scrum Master made it very clear that we should not do anything that was not specifically called for as part of a deliverable, and that the definition of the deliverable was to be viewed as that of functionality, not (as he put it) "nice to haves".

I would have likely disagreed with your CSM given only what I see in this paragraph. In general, I think that getting code under control (wrapped in tests) and refactored is part of the red-green-refactor cycle, and that a pair partner who cares wouldn't have allowed changes to be added in a shallow way. I would expect the pairs who touched the code to have cleaned it up.

However, the value of a coach is partly that they are embedded in the culture and context of the team and can guide a team differently based on its immediate "team growth" needs. I am a little surprised that your CSM was presiding over your coding practices at that level, but I don't know where you are in your journey at this time.

Jeff and I have an article coming out this month in Pragmatic Bookshelf which will cover software volatility, which is a factor you should consider. Joshua Kerievsky talks about design that doesn't have to be great and generally he is referring to non-volatile modules having tolerable design flaws (not sloppy or untested code). If the volatility of your code is very low, you might be able to ignore the design debt until someone opens it up to modify it next time. If it is a high-volatility file, then I suggest you bring it back up. More on this is available in my agileotter blog posts on heatmap and explaining refactoring, and these might give you new ideas, at least until the next Pragmatic Bookshelf issue.

Tim Ottinger

[Tom Fulton]

Thanks for your reply, Tim. I have to ask what you mean by "design debt", however. The seems to be used in the sense of "the need to redesign/refactor", but how does the design debt become incurred? Or am I misunderstanding the term?

To me, good code is both functional and well-designed. And just as correctly functioning code requires overhead (implementation and thorough testing), design requires overhead. When you say "design debt", are you implying that all functionality incurs a debt (required overhead) that must be addressed by regular design changes or at least study? If so, I agree entirely, and am happy to hear that agile supports the principle.

Thanks again!

[Tim Ottinger]

Tom Fulton wrote:Thanks for your reply, Tim. I have to ask what you mean by "design debt", however. The seems to be used in the sense of "the need to redesign/refactor", but how does the design debt become incurred? Or am I misunderstanding the term?

Technical Debt is technical work not yet done, and design debt is the same for design. The design of the code is not all that it should be, and you know it has to be changed eventually. This is like a loan, where someone has chosen to pay later for the opportunity to release something sub-optimal today. It is expected that it might cost more later (like interest) and that eventually the debt must be repaid. For instance, if more functionality is heaped on the class you mentioned without refactoring, it will become harder to clean up the design. Eventually, the code might become riddled with flags and conditionals to the point that any change is likely to result in breakage and unwanted side-effects. It will become expensive, indeed. Here is what we say on the blog about it: http://agileinaflash.blogspot.com/2009/02/technical-debt.html

But maybe it won't become difficult. Maybe this code will not be touched for years, meaning that there is no real pressure on cleanup. One time-honored approach is to mark it, and the next person who picks it up changes it. If nobody touches it, then it's probably okay as-is. This is why I mention volatility as a factor. If this file is constantly changing, then cleaning it now makes a lot more sense and is defensible. If it's in a backwater of the app, it might not make as much sense now. I would point you to the volatility article, but it's not published yet. As we mentioned in our card (http://agileinaflash.blogspot.com/2010/01/refactoring-inhibitors.html) technical debt can be so bad as to effectively prevent the refactoring "cure."

Please don't misunderstand me. The idea is not to "first burn the toast and then scrape it" -- that is wasteful of energy and time. Code should be written well all the time, hence the 'refactor' step in Red/Green/Refactor. The question is what you do when the code has already been written with a design defect that is not a functional defect. What is the basis? I would think that an hour or two is negligible and a good risk, because it's not a big investment and it is a reasonable bit of insurance. Your CSM may feel that a lot of time has been spent in reworking old code and that it has not shown its benefits yet in fluid delivery, or may feel that this code is not worth the time right now. You have to ask the reasons.

One final note: "Technical debt" was never intended to mean shoddy workmanship. It was intended to refer to releasing software speculatively to see what is really needed by customers. It would be like taking a loan now and then paying it off when you know enough to do so. Here is a video on the debt metaphor: http://www.youtube.com/watch?v=pqeJFYwnkjE

[Jeff Langr]

Hi Tom,

This is a great topic to bring up. I don't frequently introduce quotes in order to project authority, but the 12 agile principles (hey--we have a card for that!) are pretty clear:

"Continuous attention to technical excellence and good design enhances agility."
and
"Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely."

Our experience: You can't keep a sustained, reasonable pace of development if you allow the design to progressively degrade. The interesting thing about "sprinting" is that it requires ridiculous expenditures of energy. If you run such a sprint, then immediately run another as fast as possible, and continue to do that, you'll simply collapse and die. I have many tales from personal experience (both watching the sick little race and being in it) that back this up--for example, systems that had to be discarded after 12-18 months total development because they could no longer accommodate changes easily or safely. Or a government entity that called after two months of doing TDD without refactoring, saying, "we have a code disaster on our hands."

In your case, I could also take the side of the CSM. Technically, you shouldn't be changing the system without some sort of story to drive that change. You will be spending a significant amount of time on something the customer cannot see or understand easily; going down that path will begin to eat away at the trust relationship between them and you. If your if/else logic is stable, and you have no need to add new logic into its midst, then the best bet is to leave it alone until a change in that area is required.

Don't ask for separate stories / tasks around refactoring--it just creates bad blood and also generates risk that may be deemed unacceptable. Better, try to find a way to incorporate it into an upcoming, relevant story. Address the fix incrementally if you cannot do it without using up inordinate amounts of time.

One of the reasons we like TDD is that the refactoring step provides us the ability to incrementally address design deficiencies. The lesson to take back is to never pass up the opportunity to take a look at the design impact and correct it if necessary. An if-else structure should have suggested the opportunity for introducing some form of polymorphism from the very first "else." Trivial and premature as it might have seemed at the time (to whoever was building it), it's almost never too soon to begin fixing the problem.

If you build clean-up into a simple, regular cycle like this, no one should ever notice or complain. I've tried to advance my "incrementalist thinking" over the years, and have found that taking yet smaller steps (i.e. in this case addressing impacts to design even sooner) is in most cases a better choice.

Regards,
Jeff

[Jeff Langr]

And darn it, Tim. You're right. I apologize for the length of my posts--I visited the dentist this morning, got some bad news and some hefty meds, and so I'm finding it a little tough to focus. No doubt some of you have noted the typos in my responses.

Turns out Tim and I wrote these last two messages at around the same time, so I didn't see his before replying. (And it looks like it took me 8 minutes longer to compose my message. Drat.)

Jeff

[Tim Ottinger]

I'm a frightfully fast answer composer. But latitude to Jeff L for his medicated condition.

[Tim Ottinger]

On more technical debt resource for you: http://bit.ly/hGSxrp

[Sonny Gill]

Tim, the Heatmap idea is really great.. thanks for posting that.

Incidentally, a couple of days ago Martin Fowler posted a nice entry that I think has much relevance for this topic - http://martinfowler.com/bliki/TradableQualityHypothesis.html.

[Tim Ottinger]

Sonny,

That really is a great paper. I've been keeping it open in my browser since Sunday.
Tim