One of the challenges of Scrum is that it doesn't include technical practices, and yet the Analyze-Design-Code-Test SDLC doesn't work well in a two-to-four week cycle. Something else is needed instead.
This is one of the reasons I like XP (and part of why we focused on XP in the book): it includes specific technical practices to address this problem. Now, this is a big subject so I'm only able to give an overview here. I'll refer you to the book for more.
Many people squeeze the Analyze-Design-Code-Test sequence of phases into Scrum's two-to-four week Sprints. As you're discovering, this doesn't work all that well--it leads to technical debt and often causes testing bottlenecks as well. XP provides a solution, but it's probably quite different from what you're used to.
In XP we prefer not to do any work that doesn't provide customer value. So there should be no technical infrastructure effort that doesn't support current or past stories (aka deliverables)... and stories must be customer-valued, so technical infrastructure stories aren't allowed either.
This requires that all technical infrastructure be build incrementally, alongside stories. To achieve this, XP uses simultaneous phases
rather than sequential phases. Analysis, design, coding, and testing are performed simultaneously.
The driver for this effort is test-driven development (TDD). Test-driven development mixes together design, coding, and testing into a single activity. Briefly, it works in very small, repeated steps (each about 30 seconds long) as follows: Decide on the next, very small increment of functionality Define behavior and define interface; write test that checks that behavior and interface; run test and watch it fail (about five lines of test code) Implement interface and behavior; run test and watch it pass (about five lines of production code) Review design, identify improvements, and use refactoring to implement those improvements; make sure tests continue to pass after each micro-refactoring. Repeat with next small increment
Now here's the trick. Much of the design improvement comes during the "refactor" step. I call it reflective design
: it's a process of reviewing existing code, extrapolating its design, coming up with improvements, and then refactoring to implement those improvements. The design actually comes after the code, which is why it's reflective.
Designing after coding sounds ridiculous, I'm sure. From a phase-based perspective, it would be. But remember that design and coding actually happen simultaneously
. It's not a case of spending days coding, then days more fixing the design. It's actually a matter of creating a few lines of code that provably solve a problem, then critiquing that code and how it fits into the overall design of the system
and making immediate improvements, which are also verified by your tests.
This probably seems impossibly low-level from an architecture point of view... and it is. But I had to introduce the basic concept of reflective design before I could talk about how design and architecture works in XP.
Test-driven development alone isn't enough. Constantly reviewing your code and refactoring leads to good method- and class-level design, but it doesn't address the larger question of properly partitioning class responsibilities and relationships, or the even larger question of overarching architectural patterns.
Continuously throughout this process, the programmer should also be performing reflective design at the package and architectural level as well. Just as refactoring can eliminate duplication, clarify concepts, and improve design at the method and class level, it can solve problems at the higher level as well. And
, if the design is kept simple and the design is done continuously, the cost of these changes is low.
Thinking at multiple levels simultaneously is difficult, which is one of the reasons pair programming is valuable in XP. While one person types and thinks about the tactical issues in the current class, the other person is thinking ahead to the next set of tests and considering how the changes the pair is making influences the overall design. Is there are an opportunity to improve relationships or responsibilities? Is the current architecture sufficient to solve the problems the pair is seeing?
There's one more element: risk-driven architecture. Architectural issues are often cross-cutting and difficult to change. For example, if your code doesn't support internationalization, then adding it after the fact could be difficult.
Risk-driven architecture looks ahead to potentially risky architectural problems (such as internationalization) and directs refactoring efforts to make those risks go away, without actually implementing the speculatory feature. In other words, if your customers don't want any localization yet, you wouldn't internationalize yet. However, you might improve your design so that there was only one class to internationalize when a localization story finally did come along.
I'm condensing 45 pages of material into a few paragraphs here, so I'm afraid this still may not sound feasible. I guess you'll have to buy the book if you're still not convinced.
To answer your specific questions:
- Enterprise strategy:
If you have a overall architecture or strategy that you have to plug into, it's a constraint on your development process that team members should keep in mind as they work. When you discuss design issues as a team, this should be part of those discussions; when you do test-driven development, this should influence your decisions.
- Delivering new infrastructure:
This is largely incompatible with the idea that all deliverables should be valued by your on-site customers (the business experts on the team). You can engage in some tortuous logic that says that, since the infrastructure is important to the company, it's customer-valued, but that usually doesn't fool anybody. Often in this situation there's a schism between what the business experts care about and what the technical organization cares about... perhaps because the technical organization has not yet grokked delivering technical infrastructure incrementally.
deliver technical infrastructure incrementally using reflective, continuous, incremental design as I described above. However if people don't believe it's possible, they'll want the technical infrastructure to be developed all at once. I would push back against this, personally, but that's because I know how to do it incrementally. If I wasn't confident in my ability to deliver infrastructure incrementally, I might ask for it to be a "technical story"... but expect a lot of flack (and rightfully so) from your business experts. Introducing technical stories and insisting that they come first upsets the balance of power in agile planning: estimates are supposed to come from developers, priorities are supposed to come from business experts.
We can talk about this further if you like. There's a lot of context here.
There are a lot of different kinds of ilities. Scalability, Stability, and performance...ility are customer facing and can be scheduled with stories. "Flexibility" and "maintainability" are more abstract. In XP, however, flexibility and maintainability are built-in to the process. You're constantly improving your design, which constantly improves flexibility and maintainability.
One thing that isn't
done in XP is speculative flexibility. In other words, you don't code in hooks or plug-in points for things you don't currently need. That's because, in XP, you're constantly improving the design. Change actually gets easier over time, not more difficult, so adding code now is more expensive than adding it later. Also, speculative flexibility is sometimes wrong, and when it is, it's often difficult to fix because too many things are depending on it.
This idea--no speculative generality--can be difficult for people to swallow. One of the things that separates good designers from poor designers is their ability to see and implement abstractions. You're still allowed to see the abstractions; we just ask that you delay implementing them until there's an actual need. Sometimes waiting will lead to new information that allows you to see simpler and more powerful abstractions. (Going through this process--seeing how my abstractions were simpler and more powerful if I waited to introduce them--is what convinced me that incremental/evolutionary design was feasible.)
- Third-party solutions:
I'd have to know more about your situation to comment on this. In general, most third party code can be isolated behind an interface (or interfaces) that you can