eXtreme Programming and Process

[Desai Sandeep]

Hi all,
I wish to start learning about process, extreme programming and pair programming.Am I correct to assume that this has close links with UML.
From what I gather while reading Martin Fowler's book, he had briefly mentioned about a process.As I recollect, you need to have a Developmental process in place to lay down which artifacts would be required in which phase of the project.
It will be helpful if some one can provide further information on the same.
Thank you.
Sandeep
SCJP2,OCSD(Oracle JDeveloper), OCED(Oracle Internet Platform)

[Mark Herschberg]

Originally posted by Desai Sandeep:
Hi all,
I wish to start learning about process, extreme programming and pair programming.Am I correct to assume that this has close links with UML.
From what I gather while reading Martin Fowler's book, he had briefly mentioned about a process.As I recollect, you need to have a Developmental process in place to lay down which artifacts would be required in which phase of the project.

Wow, big question. Well, here's a quick answer.
The way I describe a process to students is: everything programmers do which isn't coding. (Yes, that answer is inaccurate for all sorts of reasons, but it starts them thinking in the right direction.)
Basically, software development != coding. Development has a process. Typically that process involves the creation of a project with a simple, high level goal, e.g. "let's create ERP software for AS400 machines." For there you typically have requirements gather, in which you further define the goals, e.g. "queries returning result sets with less than 400 items should return in less than .5 seconds." After that, you go into design, e.g. Java vs. .Net, 3-tier vs. 2-tier, etc. Then you code. After coding you test. This is folled by documentation and release. Some software has deployment, e.g. sucking your customers into paying you more money to install it for them; and if you're really luck, you sell them a support contract.
Now I just described a lot of things that wasn't coding: vision, requirments, design, coding, testing, documentation, release. a process describes how these steps are done.
The way I described it above, linearly, is known as a waterfall process. This was bi gin the 80's. The problem was, the requirements would change, but you were already in the coding stage, and somehow had to change your code without changing your design. Bad Idea(TM).
In the 90's the buzzword was RAD, Rapid Applcation Development. This sed what was known as an iterative process (as opposed to waterfall), in which you repeatedly "design, code, test, repeat" so that was the requirements changed during the project, the project coul change with it. Martin Fowler took the iterative process to the extreme, and created XP.
I would not recommend going through XP as your first introduction to a process. I saw this for 2 reasons. First, to be honest, I'm skeptical of it. But even if it is viable, it is, well, extreme. It's on one end of the spectrum. That's not a good starting point. Start in the middle.
I'm going to be starting to survey books in this field, but I haven't yet, so perhaps others can suggest some titles. I do recommend two "meta-process" books, The Mythical Man-Month, and Peopleware. Neither of these books describes a process per se, but they describe mistakes made in previous projects which should give you some background when learning about, and later selecting, a process for your own use.
There's a lot I left out, but I'm out of time.
--Mark
[This message has been edited by Mark Herschberg (edited October 22, 2001).]

[Junilu Lacar]

Hi Mark and Sandeep
Thanks for trying to help prime the pump here. We've been having a bit of a dry spell in Process so your posts are a welcome sight.
Actually, XP was started by Kent Beck who wrote the manifesto for it: "Extreme Programming Explained - Embrace Change." The XPE book was based mainly on his work with Ward Cunningham (he and Beck were the originators of the CRC card technique which is also used in XP) and some other people on the C3 project at Chrysler. Martin Fowler is mentioned in the XPE book and later co-authored "Planning Extreme Programming" with Kent.
Mark is right about XP taking iterative development to the extreme. But that's not all that XP does. It tries to take everything to the extreme: if testing is good, test all the time (accomplished through test-first programming and constant integration/testing) if refactoring is good, refactor constantly. If overtime is bad, don't do any overtime. If direct communication is good, then communicate directly all the time, etc.
I have read "The Mythical Man-Month" (20th anniversay edition) and a lot of what FPB Jr. writes in "No Silver Bullet" are strikingly in line with the philosophy and practices of XP: "grow software rather than build it" -- "change is an essential property of software" -- "develop incrementally, so that the system always works" -- "the system must first be made to run, even though it does nothing useful except call the proper set of dummy subprograms. Then, bit by bit it is fleshed out..." -- "enthusiasm jumps when there is a running system, even a simple one" -- "iterative specification of requirements" and many more.
While I find the thought of doing XP very exciting, I would also be very cautious about trying to introduce it in an organization that is used to the "older" and "conventional" way of doing things. There are so many inertial forces that keep most organizations from adopting XP practices that it may be impractical for anyone other than a person with great influence to even suggest doing something remotely similar to XP.
There are lots of books and articles coming out about XP (two of them still sit on my shelf unread ) and related methodologies: Adaptive Software Development, Agile Software Development, SCRUM, FDD, Crystal, etc. Look for books/articles by Robert Martin, Kent Beck, Martin Fowler, Jim Highsmith, Alistair Cockburn, Ron Jeffries, Craig Larman, Scott Ambler and many others. Heck, even the author of "Peopleware", Tom DeMarco, thinks that the emergence of XP one of the most encouraging developments in software today.


------------------
Junilu Lacar
Sun Certified Programmer for the Java� 2 Platform

[Desai Sandeep]

Hi Mark and Junilu,
Thanks very much for the info!
This should certainly help me to jumpstart on learning about the Process.I had visited my local book stall here to find the books available on the same.Unfortunately, Extreme Programming by Kent Beck is out of stock
Found some more books of interest on the Process.They are :

1. The Unified Software Development Process by Three Amigos
   
2. The Rational Unified Process: An Introduction by Philippe Kruchten
   

Junilu, how do you rate these books?
Mark, from what you have explained, I am still not able to get how one can inculcate XP within the team (say if the firm is not an old or a conventional one and is willing to go for it!).I understand that XP makes us to do everything to an extreme.However, if I am correct, we can define levels to which one should stretch for every activity we do - let it be testing, refactoring or whatever.So, the Process could help us define the time allocation for each activity?
Also, I understand your views on Waterfall Process.Is it that XP is NOT meant for this process?I had been following the Waterfall lifecycle, while coding in VS COBOL II on the IBM Mainframe.As you mentioned, this approach is a Bad Idea(TM) from the point of view of changing requirements.I recollect reading an article not long time back which mentioned about it still being used for the Enterprise Development.And I believe it was for Java Development!!Please find the article on How does the waterfall development methodology play in the enterprise? for discussion on the same.
Infact, I had mailed the author on the cons of the Waterfall life cycle when compared to Iterative life cycle, which included the following :



1. Feedback in the Waterfall lifecycle is delayed
   
2. Flexibility for change in business requirement is not accounted for in Waterfall cycle.
   
3. Is suited more for Structured language development like COBOL.
   

However, I did not receive a reply from him.I look forward for your views on this.
Regards,
Sandeep
[This message has been edited by Desai Sandeep (edited October 23, 2001).]

[Desai Sandeep]

Hi,
I believe, you need to register (for free ) to TechRepublic to access the article I posted above.In case, you have problems please let me know.I will post the article here.
Thanks,
Sandeep
[This message has been edited by Desai Sandeep (edited October 23, 2001).]

[Paul Newton]

As Junilu says, it is nice to see some posts in this forum. Although I think a lot of the 'process' oriented discussion has been taken over by the UML forum. Maybe a merger might be a good idea?
Anyway, it was interesting to read the posts. XP is painfully a-la-mode at the moment, which can make it difficult to separate the (very interesting) ideas from the hype.
Mark gives a good summary of process - what it is and what it isn't. He states that coding != process. Interestingly, Beck (in XP Explained) actually sees a more fuzzy boundary between these two. Or at least, that was the impression I got from a first reading.
Almost all of XP is based around the programmer. It is a process designed by programmers, to help programmers write better programs. I am simplifying XP a lot, and doing a great disservice to what is a fascinating read, but XP is very much code driven. Code reacts to changes in 'stories'. If you are not sure about something, go and code it. When are you finished? Write test cases in code etc etc.
I do not want misquote Mark, because I understand exactly what he was saying. There is a lot more to developing a system than sitting down and hacking. However, there is a real danger - especially with the 'high ceremony' monster methodologies, that the code is seen somehow as a formality - something that happens at the end, as a minor part of the cycle (look at all these architectural specs and gannt charts - the project must be doing well). In this sense, XP very much takes the opposite approach - docs have no value if they don't result in code. As Beck says - a user won't thank you for pretty documents (I am not that is entirely true, but I agree with the idea).
Anyway - just a few thoughts on XP. XP is a very interesting development, and I would enjoy the opportunity to try some of his more 'extreme' ideas - particularly pair programming. As someone who has come from the RUP side of things, I tend to follow many of the ideas anyway (unit testing, iterative planning, flexible requirements etc). However, as Beck says - if you don't do all of them - then you aren't extreme...
Just one final word. I felt that perhaps the most welcome contribution of XP is that it emphasises the idea of fun. Software development can actually be enjoyable!
regards,
paul.

[Mark Herschberg]

Originally posted by Desai Sandeep:

Found some more books of interest on the Process.They are :

1. The Unified Software Development Process by Three Amigos
   
2. The Rational Unified Process: An Introduction by Philippe Kruchten
   

Junilu, how do you rate these books?

Well, you asked for Junilu's opinion, but you'll get mine, too. RUP is a relatively heavyweight process, compared to what most companies do today. Still I think it's good to have a basic idea what's on this end of the spectrum, and RUP is very wide-spread. I'd probably recommend just reading a small intro book, rather than the Three Amigos book, which I seem to recall is very big and slow reading.
Important disclaimer, both in RUP and XP, the authors often point out, you can do what you want. You don't need to include every step or task spelled out in the process-do what's right for you. So you can certainly learn RUP, and then just use parts of it. Of course, one complaint I have about XP is that although Beck talks about tailoring it for your needs, his justification of why it works seems to be something of a house of cards, in that if you remove any one element, the rest won't work.

Originally posted by Desai Sandeep:

Mark, from what you have explained, I am still not able to get how one can inculcate XP within the team (say if the firm is not an old or a conventional one and is willing to go for it!).I understand that XP makes us to do everything to an extreme.However, if I am correct, we can define levels to which one should stretch for every activity we do - let it be testing, refactoring or whatever.So, the Process could help us define the time allocation for each activity?

Well, you saw my comment in the disclaimer above on how I question splitting up XP. My other issue is how to get XP off the ground. I haven't yet read the "XP in Practice" book, which may cover this in more detail. In any case, XP doesn't scale to more than 10-12 people. Some may disagree with the exact number, but it's definately not for 20+ people.
In any case, the ideal way I would introduce XP in a company is as follows. First, convince the manager and other developers to give it a try (*big hand wave*). Then, find a new project, preferably one starting from scratch, i.e. completely new code, and not another revision of existing code. Start with 2-4 people, and slowly build up the group as the project gets bigger. By starting with a smaller group, it's more managable. Preferable have an external perosn, not in the project, but familiar with XP, who will check in every 2-4 weeks and see how you guys are doing, and will objecttively evaluate whta is and isn't working well in the group.

Originally posted by Desai Sandeep:

Also, I understand your views on Waterfall Process.Is it that XP is NOT meant for this process?I had been following the Waterfall lifecycle, while coding in VS COBOL II on the IBM Mainframe.As you mentioned, this approach is a Bad Idea(TM) from the point of view of changing requirements.I recollect reading an article not long time back which mentioned about it still being used for the Enterprise Development.And I believe it was for Java Development!!Please find the article on How does the waterfall development methodology play in the enterprise? for discussion on the same.
Infact, I had mailed the author on the cons of the Waterfall life cycle when compared to Iterative life cycle, which included the following :

1. Feedback in the Waterfall lifecycle is delayed
   
2. Flexibility for change in business requirement is not accounted for in Waterfall cycle.
   
3. Is suited more for Structured language development like COBOL.
   

However, I did not receive a reply from him.I look forward for your views on this.

I would put XP on the opposite end of the process spectrum from a waterfall process. Iteratively processes are in the middle. Obviously the spectrum is continuous, and not totally discrete. I do think there are applications for which a waterfall process is appropriate--both in software and in other fields.

--Mark

[Junilu Lacar]

My $0.02 about "Waterfall":
I think it was Robert Martin who wrote in his series of articles on "Interactive and Incremental Development" that the originator of the Waterfall method did not envision it being done the way it is most commonly portrayed. Rather, the originator envisioned that analysis, design, coding, testing, etc. would be done in separate phases but repeatedly/iteratively.
However, somewhere down the road from conception to implementation, the "Waterfall" method lost those critical iterations. And it's not hard to see how that could happen. Think back to the most unrealistic project plan you have ever seen (I can think of a few right off the bat). Chances are, those plans followed a "Waterfall" pattern with no iterations.
It never ceases to amaze me how often managers are so out of touch with reality. They come up with a "Waterfall"-based plan and expect developers to stick to it. But that is never the reality of development, is it? Unless you have somehow had the opportunity to work with users who know exactly what they want and never change their minds, analysts who know how to translate exactly what the users want into written requirements, designers who can come up with the perfect design the first time, coders who produce perfectly bug-free code, and have managed to keep this phenomenal team a secret all these years...
Well, then you're probably just like the rest of us who muddle along, trying to figure out what exactly is required and how exactly we can fulfill those requirements, at first not succeeding but trying, trying, again and again.
So why do so many managers make unrealistic plans? How can they be so out of touch with reality? Because they use Microsoft Project, that's why! No, seriously, I really think that's one of the root causes. Most managers I have met manage with detailed plans made in MS Project. Allocate resources, identify the critical path, show task dependencies, etc. It all leads them to creating the ultimate unrealistic Waterfall-based "Project Plan from Hell".
So, when the project manager presents the initial plan to upper management, he/she most probably already has the project mapped out through testing, as far as 9 months out.
Now, isn't that just ridiculous? Well, maybe not if you treat it as a very rough estimate. The problem is, once it's down on paper, it most often is just as well etched in stone. Management just doesn't seem to be able to completely grasp the concept of fluidity and change when it comes to software development project plans.
(I'll have to continue tomorrow night... it's way past my bedtime )
Junilu

[Paul Newton]

I am not sure I agree with the idea that someone somehow invented the waterfall process. This is something that has gradually emerged over the history of early computing - borrowing metaphors and practices from other disciplines (most notably engineering).
I too wince at the thought of a detailed 9 month MS Project, where everything is planned out, right down to the last line of code. However, I think it is a little unfair to lay the responsibility squarely are the feet of the managers. Don't forget that entire corporate cultures have grown up around the idea of predictive planning. Budgets are set 1 year in advance. Resources and financing must be fought over. Managers must size their project or risk losing out on the pie.
Some of the criticisms of XP that I have read, identify this as an area of concern. We can admit that projects change and evolve over time - as Beck says, we should embrace change. However, this raises the question - how do we make our managers embrace this change? And their managers? And the CEOs? And the board of directors. For those that are not used to a more adaptive IT world, I think this can be a very difficult leap of faith. I look forward to the "XP...lessons learned" stories.
regards,
paul.

[Junilu Lacar]

Paul,
This is what Robert Martin writes in Iterative and Incremental Development":

In 1970 Dr. Winston Royce published a paper entitled "Managing the Development of Large Software Systems". This paper appeared on pages 1-9 of Proceedings of IEEE WESCON. It is commonly agreed
that this paper was the progenitor of the waterfall model of software development. And in some distorted fashion, perhaps it was. It may be that some pointy haired managers ignored Royce�s words and simply used one or two of his illustrations to create and promulgate the idea that software can be completely analyzed before it is designed, completely designed before it is coded, and completely coded before it is tested. I.e. - Waterfall.
Were I Dr. Royce, I would wince at the thought of how thoroughly and inextricably Waterfall has permeated the software development community. It is not surprising, even today, to hear of managers chiding their developers about doing design while in the implementation phase. The manager wants to be able to manage the schedule by drawing an X through the design phase.

True, project managers are not to be blamed for everything (just most of it ). There are many factors that contribute to the bad situations that many developers find themselves in and even developers themselves are often part of the problem.
This leads me to the next part of my diatribe: most development efforts that end badly do so because the team never really manages to come together as one. In fact, there often is really no real "teamwork" at all. It's just a bunch of people working individually or in small isolated groups who just happen to report to the same manager.
(again, I must pause to attend a "team" meeting. Later...)
Junilu
[This message has been edited by JUNILU LACAR (edited October 29, 2001).]

[Ilja Preuss]

Desai, you might want to take a look at the "XP Evaluation Kit": http://xp123.com/cfv-xpek.htm
[This message has been edited by Ilja Preuss (edited November 07, 2001).]

[Desai Sandeep]

Hi all,
My previous post comments on Waterfall methodology and its pros and cons when compared to the RAD process.
I happened to receive a mail from TechRepublic today on the queries mailed to the author on the same.Am quoting the same for your reference.

Reply to my mail by Techrepublic on the queries posted on the article mentioned in my earlier post
Sandeep,
I just received an email that you sent to TechRepublic some time ago on the waterfall methodology. You had three observations.
Feedback in the waterfall method is delayed.
I think in general this is true. You may have very active customer involvement during the waterfall analysis phase, but then you go through a design and construction phase before
you start to show the customers any aspects of the application. In Rapid Application Development (RAD), you take a subset of the application, do a quick analysis, design and construct, and then bring it back to the customer for feedback. You then modify the
application based on their feedback. In a traditional waterfall process, the customer sees most, if not all, of the application at once. In a RAD process, you show them pieces of the application much earlier.
Flexibility for changes in requirements is not accounted for in the waterfall method.
Actually, this is not the case. It is true that changes to requirements are an expected part of the RAD process. You start off with a partial list of requirements, build a solution and show the customer. They then give you more feedback on requirements and you incorporate those into the solution. This repeats until the application is complete.
However, in a waterfall approach, you can also handle change through scope change management. Once the initial business requirements are completed, you begin work on design and construction. If new requirements are added at any time during the remainder of the project, you submit a change request. The team then determines the impact to the project of making the change. In many cases, there is no measurable impact to the project. In some cases, major changes can force a project to take more time and cost more. This is especially true if major changes are seen toward the end of the development lifecycle – say during system or user acceptance testing. However, the same thing can occur using RAD methods. You could be completing your third iteration when a requirement change comes in that will take major rework to include.
Waterfall is more suited to structured languages like COBOL
I do not see that there is necessarily a relationship between the language you develop in and the methodology you use. It is true that some technologies lend themselves better to the RAD approach. For instance, since web development is heavily oriented toward the user experience, it is probably a good candidate for the RAD process. COBOL is typically used for high-transaction online and batch systems, which may not be as good a candidate,
because there is nothing to show the user when a lot of the ransaction processing takes place in the background. So, there may be a link between the language and the development methodology, but this is an indirect link. Applications that have a heavy user orientation are more likely candidates for RAD development than applications with heavy offline processing requirements.
Summary
Both RAD and waterfall development are viable models for developing applications today. In many cases, the choice of what process to use is based on the type of application being developed. The more user interaction there is, the more that you can show the customer to get further input for more requirements. Applications that have a heavy component of behind the scenes processing may not be as good a candidate for RAD, since you have less that you can show the user. Another factor is whether you have the customer engaged enough to make RAD development work. In RAD development, you need ongoing customer involvement and engagement. If you don’t have it, RAD will not work well, even if it would be a good candidate based on the application characteristics.
Tom Mochal, PMP

Your comments on the same are most welcome.
Thank you.
Regards,
Sandeep

[Ilja Preuss]

You could be completing your third iteration when a requirement change comes in that will take major rework to include.

The point is: in waterfall you also have to rework the design of the parts to be implemented far in the future. Not so in agile development, where you just adapt the design of the already working features to current needs.

Waterfall is more suited to structured languages like COBOL
I do not see that there is necessarily a relationship between the language you develop in and the methodology you use.

I agree!

COBOL is typically used for high-transaction online and batch systems, which may not be as good a candidate,
because there is nothing to show the user when a lot of the ransaction processing takes place in the background.

So you can't show the customer that the batch system is doing what she expects it to do?

Another factor is whether you have the customer engaged enough to make RAD development work. In RAD development, you need ongoing customer involvement and engagement. If you don�t have it, RAD will not work well, even if it would be a good candidate based on the application characteristics.

How does waterfall substitute for the ongoing customer involvement???

[Desai Sandeep]

Hi Ilja,

Originally posted by Ilja Preuss:
How does waterfall substitute for the ongoing customer involvement???

Although, I am not advocating what the author had to say, I probably feel that the ongoing customer requirement/changes does not play a role in waterfall methodology which is unlike the RAD methodology.So in the case of RAD, you need to have active customer involvement which is not required in Waterfall process.

Originally posted by Ilja Preuss:
So you can't show the customer that the batch system is doing what she expects it to do?

I agree with you.I donot think if you have lot of batch processing you should shift from RAD to Waterfall process.May be the author again puts its from client involvement perspective.Since Batch processing is one time design spec (the changes ofcourse could be handled by change requests which the author proposes!), one may not require as much client involvement as you would normally do while designing applications with RAD.
May be he wants to say why do you want to show the customer what you are doing while designing batch systems?Why not show him after the whole design/construct process is over- after all it doesnot require user interaction!
In any case, I have mailed him for requesting further information on the same.
Thank you for your views,
Sandeep
[This message has been edited by Desai Sandeep (edited December 11, 2001).]

[shailesh sonavadekar]

By the way , can anybody explain what is PMP ? is it Project Management Professional From PMI , USA ?

[Desai Sandeep]

Originally posted by shailesh sonavadekar:
By the way , can anybody explain what is PMP ? is it Project Management Professional From PMI , USA ?

Yes Shailesh, that's correct!You may find more information on this Certification at PMI Certification Site
BTW, are you planning to take this Certification.I intend to take it some time in the near future.
Thanks,
Sandeep

[shailesh sonavadekar]

yes. But, you must have seen the condition of appearing for that examination. Minimum 5 yrs. of project management exp. in 5 areas if you have BE degree. Have you already registered with PMI ? When you are taking that?

Actually , I was planning to take in june / july 2001 only. I had already finished PMBOK by that time. but , due to some major problems I just missed the shot.I had to postpone my plans. So , I have to study as per new PMBOK now.
Since , you are from Mumbai , you can contact Western India Chapter of PMI for more help.
one more thing I wanted to say. Basically this certification was meant for more mature industries & engineering faculties like civil , mechanical , electrical enng. These projects are massive. You are NICMAR grad. So , you must be knowing the diff. between complexity of building grassroot refinery & software projects. Sadly , in india , people from these industries are not doing these certifications. Probably , Rs 25000 to Rs 30000/- investment in the exam. Plus , the exam is not that simple. Your experience in PM is very , very important. The questions should be answered with your exp. you won't find answers for such questions in books.
i hope this helps.

Shailesh.

[Ragu Sivaraman]

Every talks and discusses about requirements/design/development and all that good stuffs ...about waterfall and UP
What about testing strategies...??
Iterative approach allows us to mitigate risks earlier in the process.. Testing start early, techwriters write early and so on.. In a non-iterative approach development , the same people would be waiting around to begin their work making plan after plan but not making any concerete progress..
IMHO, if the application requirements are very standardized and seasoned for a while and a need for rapid development, then waterfall may help..
Ex: developing dotcom stores (most of the requirements were standardized accountprofile/catalog-store/shopping cart/credit card /shipping etc...)
(no wonder we saw dotcoms mushrooming in late 90's???)
Ragu