Agility: formality vs. discipline

[Ilja Preuss]

In a recent thread in the UML forum, Gerardo Tasistro wrote:

Actually agile by its nature needs to be formal to prevent from falling into chaos.

To which I answered that it feels to me like he's confusing formality with discipline.

Here is why - any thoughts would be highly appreciated:

First I'll have to explain what I mean by "formal". The best definition I've found for this context is "being in accordance with rules explicitly established prior to use." (Gerardo, if that is not what you meant, please clarify.)

With other words, a formal process is a process with rules that are defined at the start (such as, every code change needs to be peer reviewed before check in), and at the end we can see whether we followed the process by checking whether the rules were followed.

Justifications for such formality often include making sure that no important activity is forgotten or deliberately omitted.

Now, Agile processes presume that such formality is suboptimal, that the team needs to define and follow a set of agreed upon practices, but that those practices also need to be adjusted on the fly based on current needs.

So an Agile process isn't, at its core, a set of roles, documents and workflows that an Agile team follows - it is a set of values and principles that guide the team - supported by a set of practices that are a good starting point to implement the values and principles, and that the team can use to derive it's own set of practices.

So an Agile team certainly will have some ceremonies that are closely followed, and that team members take care of that their partners keep true to. But those are not followed because of formality - because it's the process that says you should do these things. It's because the team members feel that it's important to do those things to reach their goal.

I hope this makes some kind of sense...

[Reid M. Pinchback]

I think the exposition makes sense from the standpoint of "I think I understand your explanation". One question comes to mind, and it is the statement:

Now, Agile processes presume that such formality is suboptimal, that the team needs to define and follow a set of agreed upon practices, but that those practices also need to be adjusted on the fly based on current needs.

Now, not claiming to be an Agile expert here, more I'm having a "huh, I thought I understood something subtley different" reaction to that statement. Absolutely feel empowered correct me if I'm wrong, but I thought an aspect of agility was to pick the level of ceremony that made sense given the characteristics of your current project, the risks, environment you work in, etc. In other words, don't necessarily force all projects into one process, but definitely use the process that best enables the current project. In that sense, formality is only suboptimal if the team took on more ceremony than they had to.

For example, if you are working in an environment were formality is mandated (FDA drug development being a very real example), then there is nothing sub-optimal about formality; from a strict economic optimization standpoint no external-to-the-project-team constraint is ever sub-optimal, the concept doesn't even apply. It may be *expensive* and *inefficient* but not sub-optimal.

I guess the point I'm trying to get at is I thought that formality as a concept seems potentially orthogonal to agility, not necessarily in contention with agility. Your statement seems inclined towards the latter interpretation, so I'm curious if that is the prevailing view of Agile experts. If it is, then to me it does establish a limitation in the ability to apply Agile approaches, because there are types of work with government or industry oversight that require tolerable bounds on how you change your development processes.
[ February 07, 2006: Message edited by: Reid M. Pinchback ]

[Ilja Preuss]

I can't speak for others, so the following is only my (current) opinion on the topic, although formed through countless discussions with other Agilists:

First of all, the Agility of a team isn't a binary value - that is, a team isn't either Agile or Not Agile. So even if some regulation keeps your team from being as Agile as it could otherwise, it still can work on being much more Agile than the average team.

Speaking of formality, especially if demanded by some regulation, I think it has a high probability of reducing the teams Agility. The reasoning is simple: the formality reduces the teams ability to choose the optimal set of practices, and as the formality is defined by someone outside the project, it has a high probability of not being optimal and thereby producing waste.

Formalities also most often define a form of communication that isn't face-to-face, and might even *replace* face-to-face conversation that would take place otherwise. Again this will cause a drop in Agility.

I'm not going to argue whether there are projects where it makes sense (or is necessary) to require such formality - that's too far outside of my field of experience.

But it seems to be out of question to me that formality negatively impacts the Agility of a team.

[Scott Ambler]

You should strive to be as agile as possible as your situation warrants.

As to the FDA example, having worked in a couple of organizations where this is applicable I took the time to familiarize myself with the regulations. Nowhere in them does it say you need to be inefficient, however, the so-called process experts invariably tend to interpret them that way. I was involved with a very agile project within an FDA environment, and we spent a lot of effort educating the process folks on effective software development practices. Until that point, they really had no clue.

- Scott

[Gerardo Tasistro]

Originally posted by Ilja Preuss:
First I'll have to explain what I mean by "formal". The best definition I've found for this context is "being in accordance with rules explicitly established prior to use." (Gerardo, if that is not what you meant, please clarify.)

I agree and that is what I mean too.

Originally posted by Ilja Preuss:

Now, Agile processes presume that such formality is suboptimal, that the team needs to define and follow a set of agreed upon practices, but that those practices also need to be adjusted on the fly based on current needs.

See here is the point at which we start to break up. I think that while you might use informal UML or informal documentation or come in casual wear instead of suite and tie you are definetly formal in your process. Maybe your sub-processes can have a certain degree of informality, but your agile development can't afford to have informalities at its core.

Like the over heated discussion regarding whiteboards and CASE tools. Your UML might be informal but you do make diagrams (diagram making is a set process of a formal procedure you have, along other things you listed in a post a while back). You have a formal set of TODOs while the doing of them might not be so formal.

To better exemplify this I think I'll turn to genetic algorithms. Which is something I find similar to the Agile case. You establish a genetic algorithm with a set of rules which are formal. Then you make it run and while the process might look informal and chaotic it eventually converges to a solution (it even converges better and faster than other more "formal" methodologies!!!). Now if halfway through the process you change the rules to increase mutations or tests on population you might alter the system in a way it no longer converges or simply takes longer to converge.

Originally posted by Ilja Preuss:

So an Agile process isn't, at its core, a set of roles, documents and workflows that an Agile team follows - it is a set of values and principles that guide the team - supported by a set of practices that are a good starting point to implement the values and principles, and that the team can use to derive it's own set of practices.

True, very true. But the fact that they're not written down doesn't mean they're not formal. If those sets of values, principles and guidelines were not formal then Agile could very well "mutate" to non agile or cascade development or better yet sheer chaos.

Originally posted by Ilja Preuss:

So an Agile team certainly will have some ceremonies that are closely followed, and that team members take care of that their partners keep true to. But those are not followed because of formality - because it's the process that says you should do these things. It's because the team members feel that it's important to do those things to reach their goal.

Here I lost you. How come they're not followed because of formality? How are they followed then? Hey I did read the lines you wrote. The question here is. If these "ceremonies" are not formally established then how do the team members know (in a crisp way) what they are? Without a formal established set of "ceremonies" how does the project leader know everyone is pulling in the same direction?

This is the line between agile development and chaos. While an experienced programmer like you will know this an unexperienced one can not. The unexperienced one will require this writen down or worked at. But the fact that its on paper or not doesn't make it less formal.

Without these "formal" guidelines agile development can turn into a "creative party" that as said by a post in the previous thread leads to chaos rather than development. Without these "formal" guidelines an agile developer can start to "reinvent" the processes in the name of agility. Thus spending enormous time in senseless ventures with a label of "solution space exploring" or something like that.

Along these lines comes the "eternal designer" which the culprit of our previous discussion. The "eternal designer" is a term I use for the person who is always thinking on how to design the system and refactor the project in his mind forever (or until he gets fired). For him the CASE tools are great. A tool to show "work" for all his time spent doing nothing as there is not a single line of code.

Without a set of "formal" rules in agile development the "eternal designer" can get away with it. If his concept of agile development is like a genetic algorithm which allow anything to change including the algorithm itself then he can alter the "set of values and principles that guide the team" and the "set of practices that are a good starting point to implement the values and principles".

There are other "eternal" roles I've found. Some programmers are just lazy and they think they can cut corners by calling it Agile and skipping processes. And this for me is so NOT true. You can be informal in your processes and move around things and be unorthodox. In such a way you change genes in the chromosomes of the software, but you don't change the rules under which the chromosomes themselves are changed. Those rules have to be formal.

[Gerardo Tasistro]

Originally posted by Ilja Preuss:


Speaking of formality, especially if demanded by some regulation, I think it has a high probability of reducing the teams Agility. The reasoning is simple: the formality reduces the teams ability to choose the optimal set of practices, and as the formality is defined by someone outside the project, it has a high probability of not being optimal and thereby producing waste.

But the lack of formality can free the team's ability to choose sets of practicies. Leading to the case of spending more time choosing than doing. Without a formal lockdown of these practices you might end up producing nothing.

Originally posted by Ilja Preuss:

Formalities also most often define a form of communication that isn't face-to-face, and might even *replace* face-to-face conversation that would take place otherwise. Again this will cause a drop in Agility.

Yes, but here the problem isn't formality. It is the improper choice of ways to communicate. You could very well be formal about "we have to have face-to-face conversation".

If you have a bad leader that establishes a formal requirement of means of communication which are not "face-to-face" or which replace "face-to-face". It is not an issue of "formal", but an issue of "bad leader". The same bad leader can, ouside a formal setting, establish bad requirements. And without a formal lockdown. He can soon enough change those for another set of bad requirements.

Or better yet! Going against all invariance studies, norms and procedures. He can change a bad requirement to a good one and at the same time change another bad requirement he set for yet another bad one. Half-way down the road nothing works (as usual) and he (erroneously) concludes both changes were bad. Leading to a change of both the good and the bad.

[Reid M. Pinchback]

Originally posted by Scott Ambler:
As to the FDA example, having worked in a couple of organizations where this is applicable I took the time to familiarize myself with the regulations. Nowhere in them does it say you need to be inefficient, however, the so-called process experts invariably tend to interpret them that way.

Definitely this is true, companies can take on way more of a formality burden than necessary. There is a gun-shy-of-the-FDA culture in drug QA that causes an almost lemming-like drive towards absurd amounts paperwork. Part of that is because you don't know if the FDA will accept an approach until *after* an audit, and if the FDA doesn't like what they see that can definitely become a hell-on-earth experience. As a result, drug QA people tend to copy existing processes that have been accepted previously instead of intelligently looking at current circumstances.

More what I was thinking of was that (to be specific) in a 21CFR-part 11 situation (which applies both to software development and to devices that maintain records via computer) a certain degree of audit/paper trail is required, just as it is for drug manufacturing and clinical lab practices. The FDA allows you a lot of flexiblity in your process, although not quite as much as 21CFRp11 suggests independent of additional FDA advisories and practical experience with FDA auditors. The FDA mandates that you must have enough controls in place to establish a decent audit trail that they can use to investigate safety issues. Lack of such information has pretty serious legal and financial consequences.

You could, for example, decide that a broad collection of JUnit tests is just as safe as hiring an army of QA contractors, and that is fair game (it is probably also really smart). You could decide that after-installation execution of JUnitPerf tests will be your way of dealing with performance qualification (also smart). What isn't fair game is to decide that in violation of documented company practices for regulated development you won't bother doing things like keeping audit trails, meeting minutes, training staff, etc. Documented reasons for process changes are accepted by the FDA, but not random or undocumented departures because they presume it as evidence of a safety problem. There are pieces of the formality that from a software coding standpoint are inconvenient, but exist because FDA-generated delays create, on average, a $1million-per-day loss to the company relative to patent expiry. Coding costs are almost unmeasurable noise in comparison.

Admittedly this is an extreme case, but I think underscores that projects exist in a context. Being agile enough to pick the best process for the project given its context makes a lot of sense. I think the challenge is just not to morph agility into the equivalent of the greedy algorithm in games; an overly-narrow awareness of current cost of the developer effort instead of the total cost of the project or initiative. A good process, agile or otherwise, should reduce the total burden of the work (and boy, a lot of processes don't achieve that!), not just shift it (possibly with amplified cost) to somebody else. Sometimes formality is stupid, sometimes it isn't stupid - just an unavoidable inconvenience.

Out of curiousity, Scott, in the situation you described was the resulting process ever audited? I've heard other stories about attempts to lighten the process burden for regulated development before, but when you ask the question "so, how did the audit go?" the answer has been "we haven't gotten to that point yet".

[Ilja Preuss]

Originally posted by Gerardo Tasistro:
But the lack of formality can free the team's ability to choose sets of practicies.

Well, yes - that's the point. If I understand you correctly, that is. The team is *expected* to choose the practices it uses to reach its goals.

Leading to the case of spending more time choosing than doing. Without a formal lockdown of these practices you might end up producing nothing.

As an Agile team is expected to deliver working software early and often, it has a very strong sense of needing to get work done. They can't spend a week to discuss how to implement a feature, because they would be in trouble explaining to the customer why no software was produced, why all acceptance tests still are failing.

In my experience, a team quickly finds a way to decide what practices to try. Doubly so because they don't need to lock them down - if something proves to work not well, they can still decide to do it differently in the future. In fact, good Agile teams will spend, say, an hour a week doing exactly that: reflecting on the last week, and discussing what to do better in the next.

In that sense, choosing practices *is* an ongoing effort throughout the project, but a lot of work gets done, too.

Yes, but here the problem isn't formality. It is the improper choice of ways to communicate. You could very well be formal about "we have to have face-to-face conversation".

Well, I guess you could, although I don't seem to hear often about it.

Still, I think the team is in a better decision to decide, case by case, what is the most proper way to communicate. I don't see how making it formal is going to help them be more productive.

If you have a bad leader that establishes a formal requirement of means of communication which are not "face-to-face" or which replace "face-to-face". It is not an issue of "formal", but an issue of "bad leader". The same bad leader can, ouside a formal setting, establish bad requirements. And without a formal lockdown. He can soon enough change those for another set of bad requirements.

Sure, a bad leader will do bad things, whether Agile or not - by definition...

A good Agile "leader" (aka project manager) will give the team a goal and help it to self organize to reach that goal. He won't tell the team how to work, because the most effective workers are those who can decide for themselfes how to do the work (not each for himself, of course, but in the team).

Or better yet! Going against all invariance studies, norms and procedures. He can change a bad requirement to a good one and at the same time change another bad requirement he set for yet another bad one. Half-way down the road nothing works (as usual) and he (erroneously) concludes both changes were bad. Leading to a change of both the good and the bad.

I don't get your point, sorry.

Anyway, it sounds to me as if your leader above could need some more feedback, so that he is in a better position to decide what's good and what's bad.

That's one of the things an Agile project strives for: lots and lots of early, frequent and concrete feedback. "Halfway down nothing works" isn't usual for an Agile team, because they'd notice the problems much earlier, and either fix them, or cancel the project.

[Ilja Preuss]

Originally posted by Gerardo Tasistro:
I think that while you might use informal UML or informal documentation or come in casual wear instead of suite and tie you are definetly formal in your process.

You are making a strong statement here. I wonder how you can be so sure.

Your UML might be informal but you do make diagrams (diagram making is a set process of a formal procedure you have, along other things you listed in a post a while back).

The diagram making *definitely* is *not* formal for us - believe me. We do them *very* irregularly - there is no defined point in time to make them, and in fact not even a requirement to do them at all.

You have a formal set of TODOs while the doing of them might not be so formal.

What makes that set *formal* to you? I understand that you agreed to my above definition, but either it still means something different to you, or you have a very misleading image of how we work.

To better exemplify this I think I'll turn to genetic algorithms. Which is something I find similar to the Agile case.

Agile processes are often compared to Complex Adaptive Systems - systems with a simple set of rules which result in complex, adaptive behaviour. Small changes to the rules can totally screw up the system.

So to some amount, fiddling with the practices of an Agile process is a little bit dangerous - which might lead you to suggest that they need to be formal. I don't think that's true - the team just needs to be aware of the risks, and actually needs to be able to handle the risks, because it will also need to change the practices from time to time.

True, very true. But the fact that they're not written down doesn't mean they're not formal. If those sets of values, principles and guidelines were not formal then Agile could very well "mutate" to non agile or cascade development or better yet sheer chaos.

It's not formality that keeps it from doing that, but lots of feedback.

It's not "lock down" that keeps us from endlessly discussing practices, but a look at the burn up chart and the wall of story cards, strongly suggesting that there is still a lot of work to be done for the week.

It's not formality that lets us write useful commit comments, but our coworkers - "Ilja, can you please tell me why you made this change? Your commit comment just says 'minor refactoring' - that isn't helpful at all..."

It's not formality that lets us write unit tests, but (besides other things) friendly competition - "Wow, look, Markus has already writting 50 tests today - and we didn't write one test for that code, stupid us. Can we still come up with some?"

If these "ceremonies" are not formally established then how do the team members know (in a crisp way) what they are?

By living them?

Without a formal established set of "ceremonies" how does the project leader know everyone is pulling in the same direction?

By talking to the team regularly. For example in the daily stand up meeting.

While an experienced programmer like you will know this an unexperienced one can not. The unexperienced one will require this writen down or worked at. But the fact that its on paper or not doesn't make it less formal.

As a matter of fact, we just got a new team member this month. I guess he will learn most of this simply by working together with us. Why shouldn't he? He is sitting with us in the same room, most often together with one of us on the same computer, pair programming. How would he *not* learn (and at the same time influence) how the team works that way?

Without these "formal" guidelines an agile developer can start to "reinvent" the processes in the name of agility. Thus spending enormous time in senseless ventures with a label of "solution space exploring" or something like that.

He could do that. But what would he tell at the daily stand up meeting, where everyone shortly explains what he did, and plans to do? Wouldn't he have trouble explaining to the rest of the team - and the stakeholders - what is spending his time with?

Without a set of "formal" rules in agile development the "eternal designer" can get away with it. If his concept of agile development is like a genetic algorithm which allow anything to change including the algorithm itself then he can alter the "set of values and principles that guide the team" and the "set of practices that are a good starting point to implement the values and principles".

No, he can't. The set of practices is defined *by the team* and validated by the development of Running Tested Features. A developer who doesn't contribute to a working feature at the end of the iteration had a hard stand in an Agile team.

Some programmers are just lazy and they think they can cut corners by calling it Agile and skipping processes.

Those programmers need to be educated, or separated out. Often the former will suffice.

[Gerardo Tasistro]

I find it odd that you say it isn't formal yet compare Agile with a system which has rules. You quote a site which establishes a mechanism for this. http://www.xprogramming.com/xpmag/whatisxp.htm

Please note the "core" tag before "practice". Meaning it is not an addon feature, but a central part of the system. The author is clearly establishing formally what XP is.

Overall you seem to support the fact that agile is formal than I do!

[Scott Ambler]

Out of curiousity, Scott, in the situation you described was the resulting process ever audited? I've heard other stories about attempts to lighten the process burden for regulated development before, but when you ask the question "so, how did the audit go?" the answer has been "we haven't gotten to that point yet".

They weren't audited when I was there, and I haven't kept track of what they've done since then.

My guess is that eventually an agile FDA-compliant team will have to challenge the FDA auditors at some point.

- Scott

[Ilja Preuss]

Originally posted by Gerardo Tasistro:
I find it odd that you say it isn't formal yet compare Agile with a system which has rules.

I was just comparing some aspects, I didn't say that it was identical in every aspect.

You quote a site which establishes a mechanism for this. http://www.xprogramming.com/xpmag/whatisxp.htm

Please note the "core" tag before "practice". Meaning it is not an addon feature, but a central part of the system. The author is clearly establishing formally what XP is.

That is a page for beginners to XP. People need (and typically search for) cook book approaches to things they are new to. "It depends" most often isn't an answer they can work with for their questions. See http://c2.com/cgi/wiki?ThreeLevelsOfAudience

Ron Jeffries, the owner and main author of the above site, is very well known in the XP community for the quote "the practices aren't XP, they are a way to XP". (I'm not sure whether he still says that, though.)

To quote from http://www.xprogramming.com/xpmag/jatPractice.htm - another of his articles:

There has been a long period in XP where the discussion has been around what you have to do to "be XP". Early on, I thought, and others of us thought, that you had to do all the practices to be doing XP. For the past three or four years, though, I've been saying that there is no meaning to the phrase "doing XP". XP is not the kind of a thing that can be done or not done. XP is an understanding that one has, to some extent.

[...]

To me, XP is a set of notions that inform my actions. [...]

Ah, just remembered the "just rules" quote and found some more:

http://www.xprogramming.com/xpmag/jatNotRules.htm - "They're not rules. They are, however, darn good things to know how to do, and do well."

http://www.xprogramming.com/xpmag/PracticesForaReason.htm - "You don't have to do the practices to be doing XP: You have to have done them."

http://www.xprogramming.com/xpmag/Misconceptions.htm#N37 - "While the XP "rules" are expressed very directly and in no uncertain terms, and we do recommend that teams just beginning with XP do the practices exactly, all the time, the point is to do things in these new ways, and learn from the feedback."

[Gerardo Tasistro]

Originally posted by Ilja Preuss:


That is a page for beginners to XP. People need (and typically search for) cook book approaches to things they are new to.

I rest my case.