some problems with XP ...

[paul wheaton]

Well, I've now been doing official extreme programming for about six weeks now. Now that I've done it, I think it is an excellent process, but I hit a few very painful glitches that I hope others can avoid:
1) When I started, I requested a day to tour the existing code by myself. Just to get the feel of the architecture, the group standards, they ways that folks on this team generally like to do things. I was told "no". For the first few weeks I would learn more through pair programming. I was explicity forbidden to look at the code without a pair. I did come in early one morning before anyone else and cruised the code for a couple of hours. For the two weeks that followed, the two hour cruise was more helpful than a hundred hours of pair programming. I think that pair programming is a good thing, but it doesn't have to be 100.00000000000% of the time.
2) I was told that in pair programming, the "partner" (the person in the pair that is not driving) is supposed to ask lots of questions about architecture choices and the two progress only when both parties agree on direction. This was not working at all. We ended up spending all of our time waxing on about possible directions and people were getting frustrated due to the lack of progress. It turns out that in the pink book (in the XP series), the driver picks the architecture and the partner suffers somewhat silently. The partner's job is to always support the driver and help improve the drivers productivity. The lesson here: when something isn't working, turn to the XP books.
Just felt the need to share.

[Junilu Lacar]

So now do you want to revisit your book review of XPE and give it more horseshoes?
I'm surprised you got the reaction you did about looking over the code. I wouldn't think that would be a problem at all. After all, you weren't going to modify the code, just read it. You should point out to the team that you benefitted from reading the code by yourself and that maybe they should re-evaluate their "policy".
As for the second point, you also could have tried doing what XPers call a "spike solution" to convince you and your partner that you are heading in the right direction.
Glad to see you're liking XP. I'm actually jealous because I have yet to experience it first-hand.
Junilu
[ June 21, 2002: Message edited by: Junilu Lacar ]

[paul wheaton]

I was suprised too. But the official word was "you will learn faster through pair programming".
Now that it is several weeks later, I think my opinion is not the same as my boss's opinion
XP is quite good. I see why it is becoming so popular. I might re-review that book sometime soon ...

[Ilja Preuss]

Originally posted by Paul Wheaton:
I was suprised too. But the official word was "you will learn faster through pair programming".

What do you mean by "official" - who did enforce it?
Perhaps the best answer would have been something along the lines of "I really look forward to pair programming, but I would also feel much more comfortable if I could take some hours to look at the code beforehand. What do you fear will happen when you tolerate me doing it?" Probably I would have been to surprised to find these words if it happened to me... :roll:
BTW, didn't they tell you about there System Metaphor first?

Now that it is several weeks later, I think my opinion is not the same as my boss's opinion

What is the opinion of *the team*?

XP is quite good. I see why it is becoming so popular. I might re-review that book sometime soon ...

Looking forward to read more about your experiences here...

[Ilja Preuss]

Originally posted by Junilu Lacar:
I'm actually jealous because I have yet to experience it first-hand.

What are you planning to do to get to this point? Serious qestion.

[Ilja Preuss]

I think that pair programming is a good thing, but it doesn't have to be 100.00000000000% of the time.

Pair Programming is a practice, not a rule. Most teams don't do Pair Programming 100% of the time - but most teams also report that "single programmed" code does lack some quality and often therefore require it to be at least pair reviewed, if not redone by a pair.

We ended up spending all of our time waxing on about possible directions and people were getting frustrated due to the lack of progress.

What were you discussing about? The next test to write? How to make a test pass? How to refactor the resulting code? Or something else?

It turns out that in the pink book (in the XP series), the driver picks the architecture and the partner suffers somewhat silently. The partner's job is to always support the driver and help improve the drivers productivity.

This is not how I understand XP - can you provide a pointer to the chapter of the book, please? (Page numbers won't help, except ones of the german translation )
Afaik, pair partners are supposed to be equals. They simply should find a way to compromise - if not by short discussions, then perhaps by short experiments or even by simply flipping a coin...

[Frank Carver]

This is a fascinating thread, and I'm intrigued that the discussion of methodology/process seems to become more "religious" all the time. Fundamentally, (IMHO, of course) XP is about finding what works, and doing it as much as possible, to the exclusion of things which don't work. All else is elaboration and examples.
There are now a lot of XP books out there, but "arguments from scripture" should hold little weight compared with experiment (spike solution) and recent, personal, experience (velocity, yesterday's weather etc.).
Please keep passing on your experiences - we can all learn from this sort of stuff!
I understand that it can be frustrating when people are unduly dogmatic about a particular view. Sure it's not optimal, but can you honestly say that the "100% pairing" approach was worse than the "sit down with this pile of stuff to read, and don't interrupt the workers" which has often been my experience?
As for pair responsibilities and getting off the point, I'm with Ilja. It sounds to me as if something needs a bit of tuning. Maybe the granularity of the tasks may be wrong, or the emphasis on test-first is not as strong as in my XP experiences.
Or it may be that what you have seen is just the hit of absorbing a new team member. XP doesn't claim that introducing and training up a new developer is instant, only that it is done as part of normal development.
Like many XP practices, a pairing session works best with a simple, testable goal. In the tasks which led to rambling, was the goal of the task explicit enough that you could code a test straight away, or did you (feel the) need to spend some time thinking of architecture before writing any tests? The Extreme Tuesday Club ( http://www.xpdeveloper.org ) seem in favour of "15 minute time boxes" - if you can't achieve a test pass in 15 minutes, throw both the test and code away and attack a simpler test instead.
In my experience, test-driven-development is an astonishingly subtle and powerful way of achieving good design without long pauses pondering architecture. It's a major mindset change, but a real eye-opener.
Now I'm rambling, of course. Has it taken me more than 15 minutes to answer this question ?

[Junilu Lacar]

Originally posted by Ilja Preuss:
What are you planning to do to get to this point? Serious qestion.
Serious answer (stolen from Martin Fowler): Trying to change my organization. If all else fails, I'll change my organization.
It's tough to do the first one though. I am one of those "consultants" (we self-deprecatingly call ourselves "IT whores" a.k.a. "warm bodies"), who has to do whatever it takes to make the client happy. In this business it's tough to find that rare client who will be willing to take the "risk" of changing the way they do things to try XP (good luck if you get assigned to a government client). I have wanted to try the "make stone soup" approach but it's hard to do even that when you have little control over your development environment.
The second change is a little hard to do right now too because of the economic situation.
Meanwhile I must live vicariously from content here and in the various mailing lists. I'm still looking out for good opportunities though. My hope right now is that somebody in Columbus, OH wanders into these JavaRanch forums and becomes familiar with my name. I wish I could be more active in the CoJUG but they meet on Tuesdays and I'm usually "Mr. Mom" then.

[Junilu Lacar]

Originally posted by Frank Carver:
This is a fascinating thread, and I'm intrigued that the discussion of methodology/process seems to become more "religious" all the time. Fundamentally, (IMHO, of course) XP is about finding what works, and doing it as much as possible, to the exclusion of things which don't work. All else is elaboration and examples.
There are now a lot of XP books out there, but "arguments from scripture" should hold little weight compared with experiment (spike solution) and recent, personal, experience (velocity, yesterday's weather etc.).
Since reading Alistair Cockburn's "Agile Software Development", I can see how the advice to do XP to the letter at first might be a valid approach if it can be taken. In his book Alistair talks about the three levels of learning: Shu-Ha-Ri and following the steps to the letter is the first stage. Again, this may not be feasible in all organizations so perhaps the second best way is to be flexible and do whatever works.

[paul wheaton]

What do you mean by "official" - who did enforce it?

The dude that signs my checks. Official enough for me.
What were you discussing about?
I remember a discussion about parsing. We get some strings and depending on the string received, we would do different things. I suggested a hashtable with a collection of classes implementing an interface to do the work. Cast the hashtable object to the interface, call the method that does the work.
His approach was simpler. Same sort of thing, no hashtable. Keep a vector. All methods get called and passed in the string. Each method determines if the string is for them. The observer pattern modeled a bit after the JDK 1.0 event model.
His point being that profiling is done later.
There was a good half hour of debate. In the end, I conceded so that we may write something. Later, the code was changed to what I suggested in the beginning.
The funny thing is that XP says something like "write the test before you design the code - this is test driven design" But the tests all make assumtions about the architecture of that which is not yet written.
This is not how I understand XP - can you provide a pointer to the chapter of the book, please?

Chapter 12, pair programming.
The ninth paragraph in my copy.
.....

Frank, I think the lesson here is that the XP they were using was working great for them until I got there. Every combination of pairing they did worked. But pairing with me didn't. When faced with a particular problem, their minds would often jump to the same solution. They had worked together long enough to solve similar problems together. My mind would sometimes jump to a different solution. They would tell me that I should voice this information, but hey, when you have a plan in your head and a keyboard under your fingers, do you really want the new guy to hold you up?
The solutions fournd in the pink book fixed us right up. All is smooth now.

[Ilja Preuss]

Originally posted by Paul Wheaton:
What do you mean by "official" - who did enforce it?

The dude that signs my checks. Official enough for me.

That's the wrong guy, IMO. It should be the team, perhaps under guidance by the coach. I don't think that XP works when it is forced on a team - it has to be choosen by it.

What were you discussing about?
I remember a discussion about parsing. We get some strings and depending on the string received, we would do different things. I suggested a hashtable with a collection of classes implementing an interface to do the work. Cast the hashtable object to the interface, call the method that does the work.
His approach was simpler. Same sort of thing, no hashtable. Keep a vector. All methods get called and passed in the string. Each method determines if the string is for them. The observer pattern modeled a bit after the JDK 1.0 event model.

Seems to slightly resemble the Chain Of Responsibility pattern...

His point being that profiling is done later.

So your point was that the hashtable was better because of performance? Than you were wrong...

There was a good half hour of debate. In the end, I conceded so that we may write something. Later, the code was changed to what I suggested in the beginning.

How long did it take you to implement the first solution and to refactor to the second?

The funny thing is that XP says something like "write the test before you design the code - this is test driven design" But the tests all make assumtions about the architecture of that which is not yet written.

Of course you design *the interface* by writing the test. Why does the interface differ wether you use a hashtable or a list???

Chapter 12, pair programming.
The ninth paragraph in my copy.

I will check this later...

[...] My mind would sometimes jump to a different solution. They would tell me that I should voice this information, but hey, when you have a plan in your head and a keyboard under your fingers, do you really want the new guy to hold you up?

If he has a different idea, yes - at least until I understand his solution. I think that is one of the points of pair programming, isn't it?

The solutions fournd in the pink book fixed us right up. All is smooth now.

[Frank Carver]

The funny thing is that XP says something like "write the test before you design the code - this is test driven design" But the tests all make assumtions about the architecture of that which is not yet written. I'm losing track a little, but I think this was from Paul.
I guessed that the it was something like this. I now do just about all my development by writing the tests first. And the areas where it is most useful are the ones where no code at all exists. Seems wierd? Can't think how to write tests for code which doesn't exist yet?
The trick is that test-first design forces you to start by writing down some sort of actual "requirement" in the form of a test. You have nothing else to start with! If you can't do that for even a simple but valid case (such as, what should the parser do with an empty input stream?) then you don't know enough about the requirements and should ask the customer for more information.
In practice, many of us do a sort of "zeroth" test which is to build and run an empty test to make sure the basic build/deploy/test environment works. Once that is in place, though, write the first test. Don't worry about design, that will emerge with refactoring. Just write it in the easiest way to test. That's almost always the easiest way for client code in general to use it, so you can be pretty sure you won't be writing an unusable API, at least.
Using JUnit, for example you might jot down something like.

...
public void testEmptyInput()
{
  Parser parser = new Parser();
  Object ret = parser.parse(new StringReader(""));
  assertEquals("empty string gives null result", null, ret);
}

Then you try to run it. Of course you can't even compile it, there is no "Parser" class. So write the simplest Parser class you can think of which passes the test. Maybe something like:

public class Parser
{
 public Object parse(Reader in)
 {
   return null;
 }
}

Then try to run the test again. If I haven't made any obvious typos, then it should pass. Great. Book it into the version control system - we've released our first code.
This may seem simplistic, even naive. But I would urge you to meditate on just how much design has already happened, even in this simplest of test cases. We have a class, with a name and a responsibility. We have an API with input and return types. We have an initial contract for that class. We have the beginnings of a complete regression test suite. An we have NO redundant code, premature optimization or unneeded bells and whistles. Everything it does is entirely driven by what is needed. Needed by actual client code which has to work (the tests).
Gained a lot from half a dozen lines of code, haven't we!
Now, check for duplication. I can't see any myself, but maybe you can. If you can see any, refactor to remove it, making sure the test suite still runs after every change.
Now we are ready for the next test case. So "rinse and repeat". But I guess all that took less than 15 minutes anyway, and we already have a system which passes all its tests.
[ June 24, 2002: Message edited by: Frank Carver ]

[Ilja Preuss]

Frank, very nice explanation!

[Ilja Preuss]

Originally posted by Paul Wheaton:

This is not how I understand XP - can you provide a pointer to the chapter of the book, please?

Chapter 12, pair programming.
The ninth paragraph in my copy.

Yes, it reads a little bit this way. OTOH, a couple of paragraphs later it is mentioned as the drivers responsibility to listen to his partner - and also to notice when he is "suffering silently". It is his responsibility to make sure that his co-driver always remains "on board".
I think the one thing you have to internalize for this to work is that you aren't stuck with what you try first. You can (and *will*) always refactor to a better solution. This way, it doesn't matter much who "wins" a design discussion.

[Frank Carver]

I find I have to continually remnind myself that any discussion should always be rooted in the "concrete", rather than the "abstract". By all means discuss things like "what other tests do we need for this story?" "Is this the simplest way to express this test?", "is this the simplest way to pass this test?", "is this code similar enough to this code to warrant refactoring before we move on?" etc.
In the down-to-earth XP way, if something doesn't have a test, you shouldn't be working on it. In particular, be very wary of the so-called "soft" requirements. Many things often treated as soft requirements can and should really be "hard" requirements, with automated tests. It's the job of the developer-customer dialog and "spike" experiments to convert vague desires into achievable goals, and the developers should only proceed with implementation when they know enough about a customer story to start writing tests.
Can you elaborate a little more on what drove your discussion/decision to choose between the list and hashtable approaches in your parser? In particular, were some sort of abstract "performance" or "maintainability" criteria in the pot? And if so, how did you propose to know when/if your code has or hasn't met the criteria?
I find it vital to know when I have actually reached a goal so I can stop working on it! It also helps a lot when I put something in place to ensure that future refactorings and additional features don't break it.

[Reid M. Pinchback]

Originally posted by Ilja Preuss:
How long did it take you to implement the first solution and to refactor to the second?
...
The funny thing is that XP says something like "write the test before you design the code - this is test driven design" But the tests all make assumtions about the architecture of that which is not yet written.
...
Of course you design *the interface* by writing the test. Why does the interface differ wether you use a hashtable or a list???

This is a good example of something that has been rumbling around in the back of my brain for a few weeks (yes, I know, I need a life; I'm bidding for one on Ebay right now). Frank also made a comment that I think is interesting:

This is a fascinating thread, and I'm intrigued that the discussion of methodology/process seems to become more "religious" all the time.

I've had to live through TQM and assorted process creation/improvement projects. One thing I've found is that methodology and process (which aren't the same thing) get bogged down when they become religious issues. Creating or adopting any non-trivial process is basically an engineering problem. Organizations that haven't learned that lesson do stupid things, whether they wave the XP banner or a heavyweight process banner. It sounds like Paul's boss hasn't learned that lesson.
I can't claim to be an XP expert, but I have found people use it as an excuse to validate local-optima work. Instead of learning about XP and applying it in a careful manner, and understanding the organizational context they work within, they wander around mumbling "simple is best", no matter the consequences. I've seen some very, very expensive consequences. That doesn't mean XP is flawed, the problem is in how effectively it is used.
That is partly what I mean by saying that methodology and process aren't quite the same thing. Part of architecting a process is to ensure that the outcome of following the process doesn't depend too much on the particular people doing the work. Skilled people will whiz through a process faster than the unskilled or inexperienced, but for a given set of inputs you want to see acceptable outputs in both cases.
One thing that tends to make me nervous about XP is the skill-set of its major proponents. Martin Fowler in particular is a really, really skilled designer and coder. The reason I included the quote about testing is exactly the point. Fowler creates really good tests. They focus on the interface behaviour of an object. Essentially by writing the test he has written the acceptance criteria for the implementation *and* put a state in the ground that says objects should be defined by their behavioural interactions. Not everybody knows how to do this effectively, and can undesirably bias the implementation towards some architecture or idiom.
I'm inclined to think that XP can result in decent development processes when the developers have both breadth of experience and depth of skill. Unfortunately it seems developers tend to dramatically overestimate their knowledge, particularly with respect to breadth. When that happens, I've seem XP mutate into a screw-your-neighbour game. The developer does the minimal work *they* want to do, and somebody else's job downstream is made 100 times harder.
I've also been in a pair programming situation, and have been thinking about Paul's experience. For me the situation mostly worked, I think because both of us in the pair were reasonably experienced. We each had slightly different experiences, so the combination was synergistic - each could fill in the gaps of the other. If you had a situation where one member of the pair really wasn't skilled, unless they were approaching the work as a training exercise I can see how you'd run into problems. Its one of my beefs with decisions made by consensus; consensus works best when the participants understand the issues, but it rarely generates knowledge out of ignorance. I think pair programming could face similar challenges.
[ June 25, 2002: Message edited by: Reid M. Pinchback ]

[Ilja Preuss]

Originally posted by Reid M. Pinchback:
I can't claim to be an XP expert, but I have found people use it as an excuse to validate local-optima work. Instead of learning about XP and applying it in a careful manner, and understanding the organizational context they work within, they wander around mumbling "simple is best", no matter the consequences. I've seen some very, very expensive consequences.

That sounds interesting. Can you provide some examples, please? Thank you very much!

That is partly what I mean by saying that methodology and process aren't quite the same thing. Part of architecting a process is to ensure that the outcome of following the process doesn't depend too much on the particular people doing the work. Skilled people will whiz through a process faster than the unskilled or inexperienced, but for a given set of inputs you want to see acceptable outputs in both cases.

With all due respect, I find it to be very unreasonable to expect acceptable output from unskilled people. It seems to me that to expect that, the process had to control the people, freeing them from the responsibility for the output. I don't think that this works well for software development - and I certainly wouldn't want to work with such a process.
What *I* want is a process that gives the development team (including the customer) plenty of feedback to steer the project and the inexperienced people opportunities to learn from the experienced.

One thing that tends to make me nervous about XP is the skill-set of its major proponents. Martin Fowler in particular is a really, really skilled designer and coder. The reason I included the quote about testing is exactly the point. Fowler creates really good tests. They focus on the interface behaviour of an object. Essentially by writing the test he has written the acceptance criteria for the implementation *and* put a state in the ground that says objects should be defined by their behavioural interactions. Not everybody knows how to do this effectively, and can undesirably bias the implementation towards some architecture or idiom.

IME, it is relatively easy to learn to write good tests. I have seen a team of 3 unexperienced and 2 moderately experienced developers working on an XP project under the guidance of one senior coach (who basically used some hours a week to ask intelligent questions) and producing a very well designed system. And most important: the design of the whole system got *better* through time!

I'm inclined to think that XP can result in decent development processes when the developers have both breadth of experience and depth of skill. Unfortunately it seems developers tend to dramatically overestimate their knowledge, particularly with respect to breadth. When that happens, I've seem XP mutate into a screw-your-neighbour game. The developer does the minimal work *they* want to do, and somebody else's job downstream is made 100 times harder.

Mhh, "somebody else's job downstream" doesn't sound a lot like XP to me - why wasn't this "somebody else" part of the team???
Granted, XP only works with a team that wants to use XP "to win". If it wants to use the process to do less work, at all means, don't use XP!

If you had a situation where one member of the pair really wasn't skilled, unless they were approaching the work as a training exercise I can see how you'd run into problems. Its one of my beefs with decisions made by consensus; consensus works best when the participants understand the issues, but it rarely generates knowledge out of ignorance. I think pair programming could face similar challenges.

OTOH I wonder how well non-pairing would work in such a situation... Of course the problems probably would emerge later in the game, but would that be better???
Regards, Ilja

[Junilu Lacar]

Originally posted by Reid M. Pinchback:

Creating or adopting any non-trivial process is basically an engineering problem...
Part of architecting a process is to ensure that the outcome of following the process doesn't depend too much on the particular people doing the work. Skilled people will whiz through a process faster than the unskilled or inexperienced, but for a given set of inputs you want to see acceptable outputs in both cases.

IMO, this is where the danger is: Thinking/expecting that a process can produce more or less consistent output regardless of who uses the process. This just isn't realistic. I think this is what leads to the "religious" attitudes previously mentioned.
It may be possible for a process to produce consistent output if it deals with the mechanical production of things like cars and machine parts (and even that requires some minimum level of skill), but software development is a creative process and deals with the abstract.
There are just too many variations in making software, not the least of which are the people making it. I am inclined to agree with the more "people-centric" views of process like those of Alistair Cockburn. Your process will only be as good as the people using it.
You can't really have one process that fits all projects. Each project has it's own peculiarities, including the people involved, and the process used on a project must be tailored to the needs of those using and benefitting from it. When a new situation arises (like when a new guy comes in and wants to read the code before pairing) you make the necessary adjustments.
If you really want to succeed, make sure that you have at least one person who knows what he/she is doing. Let that person lead/coach/guide the team and identify what practices will help the team become productive. Give the team guidelines but give them enough room to be creative. If you don't have at least one person who can bring it and keep it all together, you're probably screwed anyway to begin with, no matter what process you have in place.
my $0.02

[Reid M. Pinchback]

Originally posted by Ilja Preuss:

I've seen some very, very expensive consequences.
...
That sounds interesting. Can you provide some examples, please? Thank you very much!

No problemo. Not saying these were good examples of applying XP, just that people tended to use it as their excuse.
The basic model was, the team consisted of the coders. Nobody else was really thought of as part of the team, and that was the major failing. Other development teams, DBAs, release engineers, support staff, other corporations we did business with... they didn't factor at all into how many people did their work.
Support expenses increased, delivered products were uninstallable, budgets grew out of proportion to the functionality delivered, clients got fed up with the whole mess and stopped renewing contracts. Not all of this was the fault of developers, a lot of it began with management who had too little experience with creating organizational processes and managing workflow. There is nothing worse than a manager picking up a buzzword term without understanding its meaning. Net negative impact over a couple of years... easily 8, more likely 9 figures. US dollars, not lira.

With all due respect, I find it to be very unreasonable to expect acceptable output from unskilled people.

Right back atcha ... with all due respect, it is simplistic and unreasonable to argue from absolutes that don't map to the real world. Employees aren't either completely unskilled or complete geniuses. The range all over the map, and you are never sure what you get when you assign somebody to the work. The point of a decent process is to help ensure the survival of the organization and the satisfaction of its customers in spite of the variability in skills. Obviously you can't apply complete idiots to a job. You don't evaluate the merits of an approach on its ability to turn a sow's ear into a silk purse. You do try to make sure that your processes allow you to employ and grow the people you've got available. I've seen good processes and management with a broader understanding of the organization make more of newbie programmers with 1-2 years of experience than weak processes and myopic management did with experienced programmers of 10-20 years. I'm not claiming that XP is myopic; I'm just saying that I've seen the myopically-inclined quickly rally around an XP banner to justify the ineffective way they work.

It seems to me that to expect that, the process had to control the people, freeing them from the responsibility for the output.

While there are some processes that attempt to do that, that kind of control really isn't a fundamental requirement of a process-oriented approach. Frankly, I find it is a reaction I hear from people who haven't had the benefit of working in an organization that had decent processes and decent management.
In fact, generally the reverse is true. A decent process enforces accountability. That is what makes some of the pseudo-XP people I've seen really squirm. They don't want to win, as you pointed out. They just don't want to be blamed for losing. If they turn around simple work, they've done their job... to their mind. The fact that the next person in the chain can't do their job becomes almost a benefit to those who avoid responsibility (all problems are claimed as the fault of those downstream).
A lack of process usually concides with opaque work, where people can hide a lot of crap. I've seen admin staff cost an organization millions of dollars a year, because their work was so hidden the management couldn't see how their handling of purchasing issues dragged on all the projects of the entire department. I've seen the exact reverse too, where transparency and clear processes went hand-in-hand with excellent and creative work.
Don't get me wrong; I'm not saying XP is bad. I am saying that organizations have real people and real behavioural tendencies. My question is simply this: does XP work when people don't conform to a platonic ideal? I think it is a valid question, and asking it is not an indication of religious devotion to a particular approach. Decent organizational processes and process oversite are meant to provide some risk management when the ideal isn't met. In other words, in dealing with the majority of day-to-day work.
[ June 26, 2002: Message edited by: Reid M. Pinchback ]
[ June 26, 2002: Message edited by: Reid M. Pinchback ]

[Reid M. Pinchback]

Originally posted by Junilu Lacar:

IMO, this is where the danger is: Thinking/expecting that a process can produce more or less consistent output regardless of who uses the process. This just isn't realistic. I think this is what leads to the "religious" attitudes previously mentioned.

Please identify the line of text where I said "consistent output". Give me a break. I said "acceptable output". Work that meets the requirements of the business, i.e. that meets the acceptance criteria for completion of the project. Not work whose deliverables look the same, or happened on the same schedule, or had the same lifecycle properties if lifecycle issues weren't in the scope of the requirements.

Your process will only be as good as the people using it.

While there is some truth to that, in an organizational context it is also a complete cop-out. Team leads and management have the responsibility for hiring and growing their people. If your people are lacking a necessary skill for effective work, you don't create a process to accomodate their weakness, if the end result is that the work won't meet the needs of the organization. You architect a decent process that requires people to rise to the occassion, but not farther than they can realistically reach, and train or mentor them.
If the gap between the business needs and individual skills are too great, you either (a) quickly iterate the co-evolution of staff and process, or (b) hire people who have the necessary skills.

When a new situation arises (like when a new guy comes in and wants to read the code before pairing) you make the necessary adjustments.

That isn't a process issue. That is an example of having an idiot for a supervisor. Part of what makes debates over heavy-versus-lightweight process often so tiresome is that the lightweight advocates don't seem to have been exposed to even basic definitions of terms like 'process' and 'project'. They treat anything they observe that is either personally inconvenient or an example of somebody else's stupidity as a datapoint to advocate lightweight approaches. It is self-realizing behaviour.

You can't really have one process that fits all projects.

Project work and process work aren't the same thing, and it doesn't help by lumping them together as an amorphous mass. Both forms of work happen at the same time, each dealing with different issues. The thing that concerns me with lightweight advocates is that they tend to ignore that reality, and the resulting consequences.

If you really want to succeed, make sure that you have at least one person who knows what he/she is doing.
...
If you don't have at least one person who can bring it and keep it all together, you're probably screwed anyway to begin with, no matter what process you have in place.

It is true that one person can make a big difference, but there are real problems with taking this approach too far. What if a person leaves, gets hit by a bus, or isn't as good as you hoped? All those things happen in the real world. The business doesn't just shut its doors. Customers and stockholders aren't told "sorry, but we were screwed anyways".
PS: Even Cockburn, who you like to quote, admits that the amount of process oversite required for project work depends on the scale and risks of the project. That isn't the same thing as saying that processes change in an ad-hoc fashion in all project scenarios. Not to put words in your mouth, but that seems to be the conclusion you've drawn, and it is an invalid conclusion. It definitely is invalid if you do work (a) on fixed-price contracts, or (b) in environments subject to strict government regulation, or (c) situations where problems related to public health, safety, or bad P/R can put you out of business.
Definitely you change your processes as you learn and improve, but that doesn't mean that processes are inherently unsuitable just because they aren't torqued to accomodate the idiosyncratic behaviour of people working on one particular project. Sometimes it just means that the developers on a particular project didn't give a damn what kind of mess they swept under the carpet before claiming their work was done.
[ June 26, 2002: Message edited by: Reid M. Pinchback ]

[paul wheaton]

Of course you design *the interface* by writing the test. Why does the interface differ wether you use a hashtable or a list???
With the first solution, you set up a listener passing in just a listener. Your listener has a method that take a parameter that will tell you what command was passed.
With the second solution you set up a listener by passing in a string and a listener. Your listener method might not have a parameter.

[Junilu Lacar]

Originally posted by Reid M. Pinchback:
Please identify the line where I said "consistent output".
You're right, "acceptable output" is different from "consistent output". I got your words mixed in with my thoughts--Ilja knows I have a bit of a tendency to do that since he's been a victim, too . I was thinking in the same lines as Ilja: you can't expect unskilled people to produce acceptable output. But you've already answered that point.
...in an organizational context it is also a complete cop-out
Part of what makes debates over heavy-versus-lightweight process often so tiresome is that the lightweight advocates don't seem to have been exposed to even basic definitions of terms like 'process' and 'project'. They treat anything
they observe that is either personally inconvenient or an example of sombody else's
stupidity as a datapoint to advocate lightweight approaches.
Now it's your turn to give me a break. Most of the people I know of who advocate lightweight processes have "been there, done that". I think we see the same elephant here--we're just arguing over different ends of it. I am with Ilja on the point of a process: to provide sufficient feedback in order to allow the team to make adjustments in the direction of delivering acceptable output. And yes, part of that process should be to hire and train good people. I don't think it's a cop-out in any context to accept that people are what make or break processes. The cop-out is when people blame the process instead of themselves and their inability to adjust to the process or make adjustments to it to suit their needs.
There have been failures and successes on both ends of the spectrum of process weight. What makes the debate tiresome is that some of the more vocal people on either extreme can't seem to accept that there could be a middle ground. That's why I like Alistair Cockburn (yeah, I'd like to be like him when I grow up, so what? ) because he seems to be trying to find that middle ground. (I also identify with his need to constantly remind people of the correct pronunciation of his name )
My question is simply this: does XP work when people don't conform to a platonic ideal?
And right back atcha: How well does any process work when people don't conform to its ideals?
What we should ask about any process is "What kind of skills are needed to effectively use this process and how do we ensure everyone on the team has them?" But above all: "Does any of this make any sense at all?" If you can answer yes to that, I think you're at least off to a good start.
[ June 26, 2002: Message edited by: Junilu Lacar ]

[Frank Carver]

Paul wrote: With the first solution, you set up a listener passing in just a listener. Your listener has a method that take a parameter that will tell you what command was passed. With the second solution you set up a listener by passing in a string and a listener. Your listener method might not have a parameter.
From what you are saying here it seems as if the discussion took place before there were any tests for this area, and probably before there was any code to be refactored into a "good" solution.
I can't help thinking that the simplest solution wouldn't need anything to be passed in. It would hard-code the processing to pass the first test. Then, when your next few tests have caused code duplication (maybe in an 'if-else' or a 'switch') consider abstracting the common code. Even then, though, I can't see the point in doing anything other than hard-coding the handlers until there are tests which actually require different configurations of handlers.
I feel as if there is some sort of unit testing "antipattern" in here somewhere. If unit tests are applied to low-level effectively-private code which is only used to configure an object to pass its other tests, then they can act mainly as a burden to refactoring.
Please clarify for me why you feel it necessary to write tests for a "setup" API rather than just testing the behaviour of the preconfigured object.

[Frank Carver]

Reid wrote: My question is simply this: does XP work when people don't conform to a platonic ideal?
This is always a worthwhile challenge. Maybe I've just been burned by other process I have been subjected to, but to me, XP seems much more suited to the "real world" than Plato's "world of forms".
XP is an explicitly adaptive methodology, and its practices are intended to enable as much self-knowledge as possible to be fed back into the development process and the team. Learning, skills-transfer and "growth" may be informal and ad hoc (Pairing, Collective Ownership etc.) or more prescriptive (Common code style, 40 hour week etc.).
I still feel that, rather than a damning criticism of XP, Paul's experience is actually quite healthy. It shows that:

The management/development team was willing to get started and productive based on a "first iteration" of their understanding of XP. This is very unusual in the corporate world of "don't change anything, don't get blamed".

The existing team seem willing to defend their understanding fairly calmly, without resorting to ostracism, "sideways promotion" or terminhation of employment.

The team has been able to absorb someone well known (here at least) for his strong opinions, and get him reasonably integrated and productive in just a few weeks.

All of these look like good signs to me. And I bet that as Paul generates local credibility, the development process will adapt and grow to something better incorporating (and changing?) some of Paul's opinions.
I get cynical, I suppose. So many projects I am brought in to work on have no process feedback mechanism at all, and seem condemned to repeat the same expensive and frustrating mistakes forever.

[Reid M. Pinchback]

Originally posted by Junilu Lacar:

The cop-out is when people blame the process instead of themselves and their inability to adjust to the process or make adjustments to it to suit their needs.

I can agree to that, with a proviso. If 'the team' includes representation from all those who are materially impacted by the process change, making adjustments is fair game. What I've seen, and object to, is one or two individuals who "adjust" (equals "ignore" or "subvert") a process because they feel it is personally inconvenient, and be damned the impact on everybody else. This is something I see on a daily, and I do mean daily, basis, particularly by some of the staunchest lightweight advocates I know. [humourous aside: no doubt some financial folks at WorldCom once upon a time decided it was very inconvenient to treat expense-related expenditures as expenses for reporting purposes].
How well does any process work when people don't conform to its ideals?

Poorly designed processes only require one person to break it. Properly-designed processes require collusion to circumvent. You definitely can't protect against collusion (particularly if management is part of the problem).
Setting collusion aside, you can easily design processes that have easy, inexpensive checkpoints to catch individual errors or omissions. For example, having a release engineer only release what (1) they pull from a repository based on a label, and (2) passes the automated tests in the build is a perfect example. The engineer doesn't care how a team does their work, only that he/she can produce a build. People who then p*ss and moan about having to put their stuff in a repository aren't doing the job they are paid for, and you don't alter that particular process to validate their behaviour.

What we should ask about any process is "What kind of skills are needed to effectively use this process and how do we ensure everyone on the team has them?"

Definitely agreed!

[Reid M. Pinchback]

Originally posted by Frank Carver:
I get cynical, I suppose. So many projects I am brought in to work on have no process feedback mechanism at all, and seem condemned to repeat the same expensive and frustrating mistakes forever.

Amazing to see over and over again, isn't it? When I run into it always makes me wander around wondering "am I the only one with a memory?". Maybe its just because of my math background, but when I see pretty much the same input to the same process, I'm not surprised when you get a similar result. If you didn't like the old result, you aren't going to like it this time either.
The other mechanism, beyond process feedback, that is often missing is process inspection/reporting. People don't see what is happening, so they don't know where things are going wrong. I hate black-box processes (hidden and secretive work). The (lack of) quality and quantity output for easy tasks tells you that the process must be completely broken, but there is no way to observe the activities or get at hard data to collect evidence of the problems.

[Ilja Preuss]

Originally posted by Reid M. Pinchback:
The (lack of) quality and quantity output for easy tasks tells you that the process must be completely broken, but there is no way to observe the activities or get at hard data to collect evidence of the problems.

Well, isn't the most obvious problem of the process lack of communication then - the infamous Throw It Over The Wall antipattern?

[Ilja Preuss]

Originally posted by Reid M. Pinchback:
What I've seen, and object to, is one or two individuals who "adjust" (equals "ignore" or "subvert") a process because they feel it is personally inconvenient, and be damned the impact on everybody else. This is something I see on a daily, and I do mean daily, basis, particularly by some of the staunchest lightweight advocates I know.

Of course those aren't really lightweight (or agile, as we like to say) advocates, but charlatans who try to misuse the agile hype to their very personal advantage.
If they were agile advocates, they would know that agile processes have a strong focus on the One Team principle - and would live it.

Setting collusion aside, you can easily design processes that have easy, inexpensive checkpoints to catch individual errors or omissions. For example, having a release engineer only release what (1) they pull from a repository based on a label, and (2) passes the automated tests in the build is a perfect example. The engineer doesn't care how a team does their work, only that he/she can produce a build. People who then p*ss and moan about having to put their stuff in a repository aren't doing the job they are paid for, and you don't alter that particular process to validate their behaviour.

OK, lets take a look at XP - there we have
- colocated team, ideally all sitting in one room
- pair programming, all production code is produced by two developers working together on one computer
- daily stand up meetings, where the state of the project and the current tasks gets registered
- "continuous" integration and testing, code gets integrated at least daily, all developer tests have to run at 100% all the time
I think there are even more of the practices relevant, but it seems to me that these already make it hard to hide. What seems to be missing for you?

[Ilja Preuss]

Originally posted by Paul Wheaton:
Of course you design *the interface* by writing the test. Why does the interface differ wether you use a hashtable or a list???
With the first solution, you set up a listener passing in just a listener. Your listener has a method that take a parameter that will tell you what command was passed.
With the second solution you set up a listener by passing in a string and a listener. Your listener method might not have a parameter.

It seems to me that these are implementation details which should be hidden from the clients (and the tests).
IIRC you wanted a String to be parsed. Wouldn't the obvious interface be something along the lines of "parser.parse(myString)"?

[Ilja Preuss]

Originally posted by Reid M. Pinchback:
If your people are lacking a necessary skill for effective work, you don't create a process to accomodate their weakness, if the end result is that the work won't meet the needs of the organization.

Mhh, there must be a misunderstanding here, as I don't think that anyone advocated doing that...

Project work and process work aren't the same thing, and it doesn't help by lumping them together as an amorphous mass. Both forms of work happen at the same time, each dealing with different issues. The thing that concerns me with lightweight advocates is that they tend to ignore that reality, and the resulting consequences.

So can you please elaborate on where agile advocates tend to ignore the differences and what the resulting consequences are? Thanks!

It is true that one person can make a big difference, but there are real problems with taking this approach too far. What if a person leaves, gets hit by a bus, or isn't as good as you hoped? All those things happen in the real world.

You definitely suffer, regardless of the process you use. If you wouldn't suffer, that would in fact mean that you didn't make any use of his individual skills, wouldn't it?
Agile processes try to minimize this suffering by heavy cross-training. For example, in XP no task is done by an individual alone, he always has a partner on his side. Nevertheless, it would hurt to lose him - and it should, after all.

The business doesn't just shut its doors. Customers and stockholders aren't told "sorry, but we were screwed anyways".

Well, I have heard of projects that had to.

PS: Even Cockburn, who you like to quote, admits that the amount of process oversite required for project work depends on the scale and risks of the project.

I don't know anyone who would dispute that. OTOH it to easy to see your own project as "very high scale, high risk" - well, it makes it sound good, doesn't it? Nevertheless it might be very worthwhile to think about how to make it more agile.

That isn't the same thing as saying that processes change in an ad-hoc fashion in all project scenarios. Not to put words in your mouth, but that seems to be the conclusion you've drawn, and it is an invalid conclusion.

Sorry, I don't understand this. It's probably because my english isn't good enough...

It definitely is invalid if you do work (a) on fixed-price contracts

Why?

Definitely you change your processes as you learn and improve, but that doesn't mean that processes are inherently unsuitable just because they aren't torqued to accomodate the idiosyncratic behaviour of people working on one particular project. Sometimes it just means that the developers on a particular project didn't give a damn what kind of mess they swept under the carpet before claiming their work was done.

I don't follow you. What does that have to do with agile vs. heavyweight processes?

[Ilja Preuss]

Originally posted by Reid M. Pinchback:
My question is simply this: does XP work when people don't conform to a platonic ideal?

I am not sure what you mean by "a platonic ideal" in this context.
The idea of XP is *not* to impose a regulatory process on the "worker bees". Its idea is to provide a willing team a set of practices it can use to learn how to efficiently work together by living the four values of Feedback, Communication, Simplicity and Courage.
So, XP can work if you have a team that wants to use XP "to win". It probably wouldn't work if you had a team that wanted to use it to hide behind it, or even management that wanted to impose it on a team.

[Ilja Preuss]

Originally posted by Junilu Lacar:
Originally posted by Ilja Preuss:
What are you planning to do to get to this point? Serious qestion.
Serious answer (stolen from Martin Fowler): Trying to change my organization. If all else fails, I'll change my organization.

Oh yeah, been there, done that - several times... :roll:
The good thing is: I seem to slowly get a grip on the first one, so the second one might not be necessary in the near future...

It's tough to do the first one though. I am one of those "consultants" (we self-deprecatingly call ourselves "IT whores" a.k.a. "warm bodies"), who has to do whatever it takes to make the client happy. In this business it's tough to find that rare client who will be willing to take the "risk" of changing the way they do things to try XP (good luck if you get assigned to a government client). I have wanted to try the "make stone soup" approach but it's hard to do even that when you have little control over your development environment.

Yes, it's hard. Yesterday I had what I would call my first major success (after five months), so I am in a rather optimistic mood now...
When I started my current employment, I held two presentations about XP. Feedback was very good, but guess what - nothing else happened.
So I warily tried to suggest the Planning Game to the team I am working in. I even lend "Planning XP" to our project manager. Guess what - nothing happened.
So I simply started writing things to do at index cards, pinning them on our new white board under the headings "to do", "in work" and "done" - that was a couple of weeks ago. Guess what - yesterday I saw our project manager standing at the white board, mumbling something like "this is really cool. I have to write some more cards. Mhh, shouldn't we notice responsibilities on the cards, too?"

Meanwhile I must live vicariously from content here and in the various mailing lists. I'm still looking out for good opportunities though. My hope right now is that somebody in Columbus, OH wanders into these JavaRanch forums and becomes familiar with my name. I wish I could be more active in the CoJUG but they meet on Tuesdays and I'm usually "Mr. Mom" then.

Yeah, I am having most of my XP experience from a private project, too. (http://sourceforge.net/projects/bloodball) I have learned a lot by doing this, but it also sometimes is very depressing to see what would be possible at your job, too - if only you knew how to initiate it...

[Reid M. Pinchback]

Originally posted by Ilja Preuss:

OK, lets take a look at XP - there we have
- colocated team, ideally all sitting in one room
...
I think there are even more of the practices relevant, but it seems to me that these already make it hard to hide. What seems to be missing for you?

If you were talking about a small (e.g. startup) organization with only one development project, then the answer would be nothing. What you describe is fine, particularly if the organization will either stay small or the application lifecycle is short relative to the growth rate of the company. Having one team would be easy. When you are in a moderately large, geographically distributed organization, or you are mixing internally developed and externally-contracted work, the collocated team assumption doesn't cut it. You can't uproot hundreds of people in various states and countries and shove them in a room, particularly when some of those people are customer organizations who aren't accountable to you and need to be physically elsewhere to do their 'real' jobs.
Even worse is what you don't have:
- separate I/T support group per project
- seperate web site per project
- separate database environment per project
- separate data per project
- separate customer base per project
- separate delivery infrastructure per project
- separate networking infrastructure per project
- seperate security infrastructure per project
- separate regulatory context per project
When each project has its own team, and does work in dramatically different ways and ignores the complexities of what is going on around them, it radically increases the cost, frustration, and financial or legal risks borne by the rest of the organization.
The 'one team' model clearly works if all the material players are on the team. In a big organization, that gets exponentially hard to achieve. The reason why larger organizations begin to create processes and standards for work and simplify the number of technologies they use is that it allows for a divide-and-conquer approach to solving problems. I can see a 'one team' model if the one team doesn't think it is the sole determiner of standards and practices. The problem is when it does have that belief.
It is basically a complexity war. If you put yourself into a everything-touches-everything scenario, your organization has to grind to a halt as it grows. Given that, from a systems standpoint, each activity in an organization can influence many portions of that organization, you have to find a way to minimize the interfaces and monitor what happens in the transition of work across that boundary.
It is the same thing as effective component-based design. Componentization issues are trivial if you assume that there will only ever be one component (the one team with everyone in the room). As soon as you have multiple components, you have to be aware of how effectively you partition the functionality, avoid cyclic dependencies, address scalability, testability, etc. Processes in a medium-to-large (1000's of people and up) organization have similar challenges.

[Reid M. Pinchback]

Originally posted by Ilja Preuss:
you don't create a process to accomodate their weakness, if the end result is that the work won't meet the needs of the organization.
...
Mhh, there must be a misunderstanding here, as I don't think that anyone advocated doing that.

My point was that is what I've run into with lightweight advocates. Complete, total focus on creating a local optima that benefitted one person or project. Virtually zero acceptance of global consequence. Since there was no process outside the control of those teams to observe what they do, they could get away with pretty much anything they wanted. The financial consequences were enormous, but hey, their individual performance reviews were rosy so it didn't matter...

So can you please elaborate on where agile advocates tend to ignore the differences and what the resulting consequences are? Thanks!

A process is a repeatable set of activities done by one or more process workers with the goal of taking inputs and generating specific outputs. Those outputs will have known qualities that the business wants them to have. Process work is based on performing work that doesn't have a high degree of unique and expensive problem-solving. When applied to the right situations, it tends to be very high throughput, very efficient, easy to manage compared to the volume of work. Process work can be focused on small increments of state change or on creating non-unique deliverables. Learning has an immediate application to improving process work. You observe the performance of the tasks, and tweak the process or the training of the process workers to improve quality and throughput. Who the process workers are shouldn't really matter, so long as they have received the required training. Not so much art to performing the work, although there is definitely art to creating the processes.
Project work on the other hand is tasked with creating a specific set of unique deliverables or large increments of state change. The resources and acceptance criteria for the results are unique to the project. Learning has a more limited impact on project work because of the unique characteristics. Skill-building matters, but applying the skills, even identifying the skills up front, is more challenging. Lots of art to this kind of work.
Not all software engineering is project work. Sometimes people get so wrapped up in their personal tasks that they don't step back far enough to see that. Some of the work is process. Configuration management is process work. End-user support is process work. With a bit of care and fore-thought, QA is process work. Project management oversite is largely process work. Materials acquisition for a project is process work. Project approvals against infrastructure constraints is process work. Some development tasks, done often enough, become process work (e.g. do enough web pages with Struts, and you can tell people in seconds how long it will take to the team to create X pages in the web layer, and exactly what steps they'll go through to produce them).
Project work is valuable and necessary, but much more expensive and risky (because of the unknowns in a unique situation) per unit of output than process work. You do it when process work can't deal with the new situation. It isn't an either/or. What I tend to see is lightweight advocates labelling their project work as an example of process work, and claiming that all the other process stuff doesn't matter. Terminology obfuscation seems to be one of the more common debate techniques. I get the impression that lightweight approaches aren't process so much as they are a philosophy or methodology.
There are clear exceptions. Scrum is definitely a project-oversight process, just one where the amount and kinds of software output per process step is variable; the non-unique outputs (i.e. what you expect to see in a process) that drive it are the information gathered to spec out the next iteration of the process. The choice of process workers matters to productivity, but not to the generation of the information required to allow the process to head into the next iteration.
As to consequences, that is simple. All those other activities I mentioned have people employed trying to do their jobs. When lightweight advocates ignore the existence of those people and their responsibilities, it makes it difficult or impossible for them to do their jobs.

The business doesn't just shut its doors. Customers and stockholders aren't told "sorry, but we were screwed anyways".
...
Well, I have heard of projects that had to.

And you consider that acceptable? Maybe I'm unrealistic, but I expect people to be a bit more clueful about organizational survival than that.
[ June 28, 2002: Message edited by: Reid M. Pinchback ]

[Reid M. Pinchback]

Originally posted by Ilja Preuss:

OTOH it to easy to see your own project as "very high scale, high risk" - well, it makes it sound good, doesn't it? Nevertheless it might be very worthwhile to think about how to make it more agile.

Definitely people shouldn't just assume that their work is of great importance and create a lot of process just out a sense of self-importance. I class things like the Challenger shuttle blowing up as high-scale high risk, or having the FDA pull most of your products from the market and fining you a hundred million dollars. These are the real world challenges in some medium-to-large companies. If all you are working on is a corporate web site to let people subcribe to a mailing list, obviously you don't lose a lot of sleep about the consequences.

That isn't the same thing as saying that processes change in an ad-hoc fashion in all project scenarios.
...
It definitely is invalid if you do work (a) on fixed-price contracts

Why?
Because in a fixed-price contract, the subcontracting firm has to be very sure of:
(1) the requirements and acceptance criteria
(2) the division of responsibilities between subcontracting and client firms
(3) how much it should cost them to do the work.
The subcontracting firm doesn't get to adjust their deliverables in order to ensure profitability. The client firm doesn't get to change the definition of the deliverables in order to squeeze more out of the subcontractor.
In a fixed price contract you have to do the necessary work to get requirements up-front, and convert that into an accurate estimate of the effort required to honour the contract. If you don't know how to do that, you can't stay in business (as a lot of recently-created consulting firms found out in the last few years). Maybe its just the people I know, but the lightweight advocates I run into also have a high degree of fear over requirements management. One of those oh-so-inconvenient process issues. I know that there are agile processes that handle evolving requirements in a reasonable fashion (e.g. Scrum) from a developer's perspective, but contracts are written by lawyers and executives, not by the developers. The contract trumps the team. Scrum easily adjusts to new requirements, but contracts don't get re-written, and both parties hold each other stringently to the contract when a lot of money is involved.
[ June 28, 2002: Message edited by: Reid M. Pinchback ]

[Ilja Preuss]

Originally posted by Reid M. Pinchback:
Definitely people shouldn't just assume that their work is of great importance and create a lot of process just out a sense of self-importance. I class things like the Challenger shuttle blowing up as high-scale high risk, or having the FDA pull most of your products from the market and fining you a hundred million dollars. These are the real world challenges in some medium-to-large companies. If all you are working on is a corporate web site to let people subcribe to a mailing list, obviously you don't lose a lot of sleep about the consequences.

I assume as rude as it sounds to me - you probably agree that there are many projects that are much more complex than a simple subsription website but where a single failure won't cause the loss of life or ruin a whole corporation.
When working on such projects, working with an agile team would make me sleep very untroubled.

In a fixed price contract you have to do the necessary work to get requirements up-front, and convert that into an accurate estimate of the effort required to honour the contract.

Agile teams *do* gather all requirements upfront and convert that into an as accurate estimate as they can. It's just that they don't insist in following the plan when the requirements change. Of course, when working with a fixed-price-fixed-scope contract, you certainly loose this flexibility. Nevertheless you can still benefit from the early feedback of agile processes.

If you don't know how to do that, you can't stay in business (as a lot of recently-created consulting firms found out in the last few years).

Well, all of the agile consulting firms I know of are still in business. Some of them certainly do fixed-price contracts if the clients insist. Perhaps they know how to do that...?

Maybe its just the people I know, but the lightweight advocates I run into also have a high degree of fear over requirements management.

Maybe. OTOH, maybe they just use a form of requirements management you aren't used to. Perhaps they even think of you as being in fear of their form of requirements management?

I know that there are agile processes that handle evolving requirements in a reasonable fashion (e.g. Scrum) from a developer's perspective, but contracts are written by lawyers and executives, not by the developers. The contract trumps the team. Scrum easily adjusts to new requirements, but contracts don't get re-written, and both parties hold each other stringently to the contract when a lot of money is involved.

This is certainly true when you accept it. When you don't accept it, it possibly doesn't have to be that way.

[Ilja Preuss]

Originally posted by Reid M. Pinchback:
Not all software engineering is project work. Sometimes people get so wrapped up in their personal tasks that they don't step back far enough to see that. Some of the work is process. Configuration management is process work. End-user support is process work. With a bit of care and fore-thought, QA is process work. Project management oversite is largely process work. Materials acquisition for a project is process work. Project approvals against infrastructure constraints is process work. Some development tasks, done often enough, become process work (e.g. do enough web pages with Struts, and you can tell people in seconds how long it will take to the team to create X pages in the web layer, and exactly what steps they'll go through to produce them).

So, if I take the step back and see that some of my work is process work - what should I do about it???

Project work is valuable and necessary, but much more expensive and risky (because of the unknowns in a unique situation) per unit of output than process work.

What about doing project work in a very well known situation??? I think I must still be missing something...

What I tend to see is lightweight advocates labelling their project work as an example of process work, and claiming that all the other process stuff doesn't matter. Terminology obfuscation seems to be one of the more common debate techniques.

Rudeness objection - I don't find it very constructive to assume that this "obfuscation" is done mischievously.
Are you sure that your definition of "process" is the only valid one?

There are clear exceptions. Scrum is definitely a project-oversight process, just one where the amount and kinds of software output per process step is variable; the non-unique outputs (i.e. what you expect to see in a process) that drive it are the information gathered to spec out the next iteration of the process. The choice of process workers matters to productivity, but not to the generation of the information required to allow the process to head into the next iteration.

So, is the XP Planning Game process or project work in your eyes???

As to consequences, that is simple. All those other activities I mentioned have people employed trying to do their jobs. When lightweight advocates ignore the existence of those people and their responsibilities, it makes it difficult or impossible for them to do their jobs.

It's certainly not a principle of agile "whatevers" to ignore the necessities of anyone. You must know some very strange "agile advocates"...

The business doesn't just shut its doors. Customers and stockholders aren't told "sorry, but we were screwed anyways".
...
Well, I have heard of projects that had to.

And you consider that acceptable?

No. And you think that more process work and less project work would have prevented that outcome?

[Ilja Preuss]

Originally posted by Reid M. Pinchback:
If you were talking about a small (e.g. startup) organization with only one development project, then the answer would be nothing. What you describe is fine, particularly if the organization will either stay small or the application lifecycle is short relative to the growth rate of the company. Having one team would be easy. When you are in a moderately large, geographically distributed organization, or you are mixing internally developed and externally-contracted work, the collocated team assumption doesn't cut it. You can't uproot hundreds of people in various states and countries and shove them in a room, particularly when some of those people are customer organizations who aren't accountable to you and need to be physically elsewhere to do their 'real' jobs.

You don't have to - only the teams have to be colocated. If you need to have more than one team for a project, you can have one team be the "Customer" of another team (for example). There are projects which are doing exactly that.

Even worse is what you don't have:
- separate I/T support group per project
- seperate web site per project
- separate database environment per project
- separate data per project
- separate customer base per project
- separate delivery infrastructure per project
- separate networking infrastructure per project
- seperate security infrastructure per project
- separate regulatory context per project

I don't understand what you try to say...

When each project has its own team, and does work in dramatically different ways and ignores the complexities of what is going on around them, it radically increases the cost, frustration, and financial or legal risks borne by the rest of the organization.

If you know "agile advocates" suggesting you should ignore what is going on around you, please shoot them. Thanks.

It is basically a complexity war. If you put yourself into a everything-touches-everything scenario, your organization has to grind to a halt as it grows. Given that, from a systems standpoint, each activity in an organization can influence many portions of that organization, you have to find a way to minimize the interfaces and monitor what happens in the transition of work across that boundary.
It is the same thing as effective component-based design. Componentization issues are trivial if you assume that there will only ever be one component (the one team with everyone in the room). As soon as you have multiple components, you have to be aware of how effectively you partition the functionality, avoid cyclic dependencies, address scalability, testability, etc. Processes in a medium-to-large (1000's of people and up) organization have similar challenges.

I still don't understand what that has to do with one of those "components" (a team) using XP as their development process/methodology/philosophy/whatever.

[Reid M. Pinchback]

Originally posted by Ilja Preuss:
I don't understand what you try to say...

A simple point. Writing code isn't the only thing happening in an organization. For many orgs isn't even the most value-added or expense-increasing thing that is happening. Optimizing the approach to the project work can often cause a bigger sub-optimization somewhere else.
You don't always have the option of tweaking a process that is external to the team in order to suit one project, because other things much bigger and more expensive have ties to the resources managed or generated by that process. The project isn't independent of its environment, and hence is not independent of other projects and *their* methodology/process approach.

If you know "agile advocates" suggesting you should ignore what is going on around you, please shoot them.

Ahhh, the sweet smell of empowerment!

I still don't understand what that has to do with one of those "components" (a team) using XP as their development process/methodology/philosophy/whatever.

It gets back to the notion of independence (or lack of it) between teams. There is nothing wrong with XP per se *if* they augment it with something; an understanding of infrastructure and acceptance of some reasonable corporate standards. Without that, you can end up with a large collection of deliverables from small projects that are so idiosyncratic that they don't co-exist. You can end up with something that feels like a legacy code burden in a very short time.
When XP/Agile processes work, I can see that the sky would be the limit. However, processes need some protection on the downside for the sake of the organization as a whole. I fear the lightweight advocates under-emphasize this. In some respects the lightweight approach to developing projects (in the specific case where there are multiple projects and multiple project teams) reminds me of the mess that caused the US Dept of Defense to push for the development and adoption of Ada (not that I'm advocating Ada!). They had individual suppliers producing software for them with whatever tools and methodology suited the supplier. It was very efficient for the supplier. The end result for the DOD (and hence taxpayers) was a hideous mess of incompatibilities and a legacy burden straight out of hell.
Sometimes the easiest way to identify the weakness of a human endevour is to identify the strength of the approach; the blind spot of the participants will invariably be in the same place. The strength of XP and agile processes is that they serve the team. Maybe that is exactly the point of weakness; who deals with the issues and impacts that aren't made the concern of the team?

[Ilja Preuss]

Originally posted by Reid M. Pinchback:
The strength of XP and agile processes is that they serve the team.

I see three strengths in agile processes (of which I know XP best): team building, extensive feedback and (of course) agility. I don't think that these only serve the team - after all, the purpose of agile processes is to provide business value as effective as possible.

who deals with the issues and impacts that aren't made the concern of the team?[/QB]

Typically it is the responsibility of a project manager to keep the external stakeholders up to date about the project and to communicate their needs to the team and to make it their concern.
How this happens is outside the scope of XP - I don't know wether an other agile process touches this issue.