Software by Numbers

[Jane Cleland-Huang]

This week Mark and I will be promoting our book �Software by Numbers� on this discussion forum. Java Ranch will be giving away some free copies of our book, so some of the people who participate in the discussion, so we are hoping lots of you will join in!
Let me tell you a little about �Software by Numbers� and why its concepts are important for Software Development. A few years ago during the dot com era, organizations were willing to invest in software projects almost regardless of whether they returned clear value to the organization in the near future. The promise of future value was sufficient. Those days have changed! Organizations insist that funded projects show a clear ROI over a short period of time.
Software by Numbers describes a new approach to software development, in which the product to be developed is decomposed into chunks of functionality called Minimal Marketable Features (MMFs). An MMF is capable of generating revenue, saving costs, or returning some other form of tangible or intangible benefit to an organization.
Our book describes how to identify MMFs and how to apply the Incremental Funding Method (IFM) in order to optimize the value of the project through carefully sequencing MMFs. IFM also provides insight into other important financial metrics such as initial funding investments, time to reach self-funding and/or break even status, and the total value of the project measured in terms of its Net Present Value. Although this book covers some financial concepts, we wrote it especially for software developers, and also show specifically how to apply these concepts in both an agile and a RUP environment.

You can find out more about �Software by Numbers� at our website .http://www.softwarebynumbers.org or by browsing through the reader�s comments at amazon.com
We hope you will participate in this discussion. Mark and I will be around all week and will try to respond to any questions and comments that you might have.

[Mcgill Smith]

WELCOME!
Jane Cleland-Huang and Mark.

[Anuj Upadhyay]

I find the new terminology interesting. Have been around for quite some time, used and read different methodologies RUP, XP, Waterfall to name a few but honestly my desire to use one in totality on a project is not yet realized. I worked for a product development company with a huge market share but to my shock the formal software development process were a matter of discussion or partial implementation. We did deliver using a bridge methodology and that was most important for the team and management.
I am not sure if one will be able to adopt and implement one process, we are still far from standardizing the process like we have for the construction or automotive industry. We need a blueprint and a tried and tested implementation that all (all vendors) religiously follow, till then we will always get new terms and bridge processes.
We are not quite there. What do you say authors?

[Jane Cleland-Huang]

Interestingly when we started writing this book - we conceived of IFM as an add-on to RUP. However the more involved we became in the IFM concepts, the more we realized that IFM should be seen as a plug-in to any iterative development process, because it enhances the normal process with a financial or business perspective. We are really pleased to have received very positive feedback both from Agile developers and from project managers from much more traditional development environments.
So I guess we avoided the issue that you mentioned. I personally like to see a flexible process that is fitted to the current project, organization, and developers' characteristics and skills.

[Ilja Preuss]

Originally posted by Anuj Upadhyay:
I am not sure if one will be able to adopt and implement one process, we are still far from standardizing the process like we have for the construction or automotive industry.

Notice that as software developers it isn't our responsibility to reliably reproduce a defined product, as in car manufacturing. It is our responsibility to design new products - I am not sure that any industry has a standardized process for this, nor am I convinced that there would be value in such a thing...

[Arun Rao]

i have another question.. are there a lot of examples in the book? what i'm looking at is a situation that would apply to our organization here in the development process...
if there are examples, what industry it is pertaining to?

[Warren Dew]

The subject of the book sounds very interesting. Can we get some more specifics about what's in it - perhaps a table of contents listing, for example?

[Mark Denne]

Hi Warren,
Good evening! I think it's about time I replied to some of these posts Jane's been doing all the work today so far. I've had some meetings in the office which have prevented me from getting into the forum until now.
Amazon has kindly published a table of contents (as well as a sample chapter by the way). You can find the table of contents here:
http://www.amazon.com/gp/reader/0131407287/ref=sib_rdr_toc/002-9901654-7464003?%5Fencoding=UTF8&p=S00C#reader-link
Regards
Mark Denne

[Tina Coleman]

What's been your experience so far of the learning curve for organizations to implement the approach Software By Numbers proposes? Do folks tend to get it mostly right, partly right, or radically wrong the first time they do it? And are there any common missteps you'd caution us to guard against?

[Valentin Crettaz]

Organizations insist that funded projects show a clear ROI over a short period of time. --Jane Cleland-Huang
Don't you fear that the ever-recurring willingness to reduce time to market of a feature (or group thereof) negatively impacts its quality?

[Valentin Crettaz]

However the more involved we became in the IFM concepts, the more we realized that IFM should be seen as a plug-in to any iterative development process, because it enhances the normal process with a financial or business perspective. --Jane Cleland-Huang
What about the customer perspective? After all, a software is always made for someone or some organization who pays for it. Usually, when I pay for something I would like to see my requirements properly and accurately implemented. When the software is developed in-house, the customer is the company itself and this is where ROI usually comes in. Does your method provide some way of verifying that the customer can also measure its ROI? If not, how do you think the method could be amended or augmented to make this information available.
I personally like to see a flexible process that is fitted to the current project, organization, and developers' characteristics and skills. --Jane Cleland-Huang
And so do I, as I'm afraid that there is currently no existing process or methodology that allows to take all involved parties' concerns into account... For this reason, it is currently not possible to use only one process for developing software. All available processes and methodologies are almost never used as is, they are always somehow *adapted* to the organization that uses it. That's why, I see the real value of a process in the way it can be customized to my own needs, in short, adaptability is one of my biggest concerns.

[Anuj Upadhyay]

Originally posted by Jane Cleland-Huang:
So I guess we avoided the issue that you mentioned. I personally like to see a flexible process that is fitted to the current project, organization, and developers' characteristics and skills.

In most projects we also followed the iterative process and that was the defacto standard due to various reasons i.e. budget, scope, etc. As i mentioned earlier I find the new terminology - Incremental Funding Method (IFM) - interesting, I guess I will have to read the book to get a better understanding of the add-on / plug-in process and if possible comment.
-Anuj

[Anuj Upadhyay]

Originally posted by Ilja Preuss:

Notice that as software developers it isn't our responsibility to reliably reproduce a defined product, as in car manufacturing. It is our responsibility to design new products - I am not sure that any industry has a standardized process for this, nor am I convinced that there would be value in such a thing...

Hi Ilja,
I agree with you that it is not our responsibility to reproduce a defined product but what I meant was that we should have a minimal level of conformance.
What I meant by the example about construction / automotive industry was to reflect the way we approach something, like before we build something we make a plan - drawing - which includes detailed and measurable specification of the finished product based on know standard in the industry.
Software processes tend to provide ways to achieve these objectives apart from some specific issues local to our industry. What we lack is know standards and that is the core. All (vendors) do what suite their business objectives.
I am happy that some people are concerned and write books on engineering topics. Maybe one good idea may become a standard.
-Anuj

[Ilja Preuss]

Originally posted by Valentin Crettaz:
And so do I, as I'm afraid that there is currently no existing process or methodology that allows to take all involved parties' concerns into account... For this reason, it is currently not possible to use only one process for developing software.

I am not even sure that having it would be a good thing. After all, project circumstances are very different, so a "one and all" process probably needed to be quite heavy and needed to be adapted anyway. Modularity might be the best we can get - what's bad about using XP together with IFM and Agile Modeling if it fits my project, after all?

All available processes and methodologies are almost never used as is, they are always somehow *adapted* to the organization that uses it.

Yes. Alistair Cockburn even makes the point that every process *needs* its own, tailored process: http://alistair.cockburn.us/crystal/articles/mpp/methodologyperproject.html

[Lasse Koskela]

Originally posted by Anuj Upadhyay:
What I meant by the example about construction / automotive industry was to reflect the way we approach something, like before we build something we make a plan - drawing - which includes detailed and measurable specification of the finished product based on know standard in the industry.

How would you map this "construction/automotive industry approach" into software development?
The classical mapping is "plan first, then build", which is not applicable to software development as such for reasons that have been written down in literature for a number of decades already (starting from "do it twice" by Winston Royce back in 1970!). As long as we are unable to write down the plan correctly the first time, we won't be even near the production lines -- we'll remain in the world of designers who most definitely don't get it right the first time...

Software processes tend to provide ways to achieve these objectives apart from some specific issues local to our industry. What we lack is know standards and that is the core.

Yes, we learn to factor commonalities within a problem domain into "standards" which are realized in product offerings from vendors. That's a cycle that will never end -- a feature that's competitive advantage today is a commodity of tomorrow...

I am happy that some people are concerned and write books on engineering topics. Maybe one good idea may become a standard.

I hope not (see Ilja's previous post).

[Jane Cleland-Huang]

What about the customer perspective? After all, a software is always made for someone or some organization who pays for it. Usually, when I pay for something I would like to see my requirements properly and accurately implemented. When the software is developed in-house, the customer is the company itself and this is where ROI usually comes in. Does your method provide some way of verifying that the customer can also measure its ROI? If not, how do you think the method could be amended or augmented to make this information available. Valentin Cretazz

I really like this question - because the customer obviously is 'why' we are building the product in the first place. In fact when we first started writing this book - we "sold" it to the publisher on the basis that it would describe a Customer Oriented approach to software development. Our ultimate goal in IFM is to equip the customer to participate in the real decisions.
Developers are expert at building the system. They know how to estimate cost and effort for building a product. They can identify technical risks and design & build the product. Business stakeholders know how to project revenues, cost savings etc..... but ultimately all of this is in order to provide critical information to ALL the stakeholders.
As you pointed out - the customer is a critical stakeholder who can benefit from the information provided by IFM and can then participate in the decision making processes.
[ April 28, 2004: Message edited by: Jane Cleland-Huang ]

[Jane Cleland-Huang]

Don't you fear that the ever-recurring willingness to reduce time to market of a feature (or group thereof) negatively impacts its quality? Valentin Cretazz

Another great question! Of course this could always be a danger - and we've seen it time and again in the software industry. Organizations willing to deliver early buggy software in order to beat their competitors to the door.
I guess there is no easy answer here except to say that each organization has to set their own priorities. However, if we can decompose a system into MMFs, then we can deliver parts of the product early - and invest the necessary time into them to get the quality right.
I think there will always be some organizations that favor speed of delivery over quality, and others that are willing to invest a little extra time and effort to get the quality right. Ultimately it is probably market response and demand that will determine the acceptable balance between these two competing needs.
[ April 28, 2004: Message edited by: Jane Cleland-Huang ]

[Ilja Preuss]

Originally posted by Anuj Upadhyay:
I agree with you that it is not our responsibility to reproduce a defined product but what I meant was that we should have a minimal level of conformance.

Conformance to what - and why?

What I meant by the example about construction / automotive industry was to reflect the way we approach something, like before we build something we make a plan - drawing - which includes detailed and measurable specification of the finished product based on know standard in the industry.

You could make the argument that we are actually doing this - that the source code we write *is* the plan, which is then executed by the compiler to build the real product, the actual runnable system.
Anyway, we need to understand *why* in many industries it is good practice to do a detailed plan before starting construction. In my understanding the most important force is that construction is expensive - if after construction you need to change a detail and start over, much time and money will be wasted. Fortunately, today this force is *much* weaker in software development.

I am happy that some people are concerned and write books on engineering topics. Maybe one good idea may become a standard.

Don't hold your breath...
[ April 28, 2004: Message edited by: Ilja Preuss ]

[Ilja Preuss]

Originally posted by Jane Cleland-Huang:
I guess there is no easy answer here except to say that each organization has to set their own priorities. However, if we can decompose a system into MMFs, then we can deliver parts of the product early - and invest the necessary time into them to get the quality right.

Yes, that's a very important practice.

I think there will always be some organizations that favor speed of delivery over quality, and others that are willing to invest a little extra time and effort to get the quality right.

I'm not even convinced that, if done right, delivering quality *does* need more time...

[Valentin Crettaz]

I think there will always be some organizations that favor speed of delivery over quality, and others that are willing to invest a little extra time and effort to get the quality right. --Jane Cleland-Huang
Hopefully the ratio between the former and the later will get under 1 soon
I'm not even convinced that, if done right, delivering quality *does* need more time... --Ilja Preuss
That's an interesting point you raise here, Ilja. Agreed, today people majoritarly think that producing quality (software or whatever else for that matter) costs more. I wouldn't bet on that, though... Maybe the process we are using are flawed or are simply not mature enough, or maybe they are not even applied the right way... There are too many variables in the software development equations and project stakeholders lack the proper tools the deal with that. In fact, I think that there are often more variables than equations and this is the reason why software development project sometimes fail...
[ April 28, 2004: Message edited by: Valentin Crettaz ]

[Valentin Crettaz]

Anyway, we need to understand *why* in many industries it is good practice to do a detailed plan before starting construction. In my understanding the most important force is that construction is expensive - if after construction you need to change a detail and start over, much time and money will be wasted. Fortunately, today this force is *much* weaker in software development. --Ilja Preuss
Maybe this is because in traditionnal software development, there is a gap between the moment we gather the customer requirements and the moment we start analyzing them and start building the system. The requirements document are in natural language and analysts need to read them and *somehow* translate them in a way the computer can understand them, or at least in a form that can then be made understandable for a computer. There is a strong human factor between the requirements phase and the analysis phase, actually the human factor is the weakest link during this process. According to this, one of the biggest challenge we will soon have to face is to be able to trace the customer requirements within the software itself, so that when the customer changes his mind, we can percolate that change in the software quickly and automatically. Today, this is only a dream... even with use cases and the like.

[Ilja Preuss]

Originally posted by Valentin Crettaz:
I'm not even convinced that, if done right, delivering quality *does* need more time... --Ilja Preuss
That's an interesting point you raise here, Ilja. Agreed, today people majoritarly think that producing quality (software or whatever else for that matter) costs more. I wouldn't bet on that, though...

Some day I need to read "Quality Is Free" by Philip B. Crosby... :roll:
The problem with not producing quality is that we can't really control that we will only introduce non-critical bugs. So if we allow bugs to creep into our systems, there *will* be bugs which need to be fixed.
In my experience, fixing a bug is much more costly than not letting it creep in in the first place - so much more costly that trying to prevent *all* bugs might still be more effective than only fixing some of them.

[Valentin Crettaz]

As you pointed out - the customer is a critical stakeholder who can benefit from the information provided by IFM and can then participate in the decision making processes. --Jane Cleland-Huang
Thank you very much for your answer. Basically, I agree with what you said but I would replace all occurences of "can" with "must". This would be in an ideal world of course. As everyone knows, reality is different and "can" is appropriate

[Valentin Crettaz]

The problem with not producing quality is that we can't really control that we will only introduce non-critical bugs. So if we allow bugs to creep into our systems, there *will* be bugs which need to be fixed. --Ilja Preuss
Right, you're talking about the infamous "Yeah, whatever, I'll fix that later when the thing works"... It always makes me smile when I hear this sentence (even though I used to say it )... A program cannot work (properly) if something still needs to be fixed, or does it? A dozen "I'll fix this later" and you are in trouble without even knowing it.

[Anuj Upadhyay]

Originally posted by Ilja Preuss:
Conformance to what - and why?
[ April 28, 2004: Message edited by: Ilja Preuss ]

To keep it short and not to go off topic, what would one want to conform to when developing mission critical systems i.e. medical equipment - heart and lung support m/c or systems for an aircraft?

You could make the argument that we are actually doing this - that the source code we write *is* the plan, which is then executed by the compiler to build the real product, the actual runnable system.

I would not if I were to develop a system similar to the ones mentioned above.

Anyway, we need to understand *why* in many industries it is good practice to do a detailed plan before starting construction. In my understanding the most important force is that construction is expensive - if after construction you need to change a detail and start over, much time and money will be wasted. Fortunately, today this force is *much* weaker in software development.

Also, I would say it is weaker because we don't know what quality we must have. There is no reference to compare to; we accept software no matter how bad it may be.
Jane Cleland-Huang also addresses this:

Originally posted by Jane Cleland-Huang:
Another great question! Of course this could always be a danger - and we've seen it time and again in the software industry. Organizations willing to deliver early buggy software in order to beat their competitors to the door.
[ April 28, 2004: Message edited by: Jane Cleland-Huang ]

Don't hold your breath...

On the lighter side, they are just my views and we may have reason to disagree. As I mentioned earlier we are not quite there, for sure I will not hold my breath..:-)

[Ilja Preuss]

Originally posted by Anuj Upadhyay:
What would one want to conform to when developing mission critical systems i.e. medical equipment - heart and lung support m/c or systems for an aircraft?

I am not sure, as I don't work on such systems. I understood you to say that you think *all* software development projects should conform to some shared process. Didn't you?

quote:
--------------------------------------------------------------------------------

You could make the argument that we are actually doing this - that the source code we write *is* the plan, which is then executed by the compiler to build the real product, the actual runnable system.

--------------------------------------------------------------------------------
I would not if I were to develop a system similar to the ones mentioned above.

Notice that I am not saying the source code is the only document we ever should produce. All I am saying is that writing code is less an activity of construction, but more one of design - and that it is the most authoritative design document we have. Applying a construction metapher to writing code might not be as insightfull as it seems.

Also, I would say it is weaker because we don't know what quality we must have. There is no reference to compare to; we accept software no matter how bad it may be.

Well, I actually tend to believe that I know quite well what internal and external quality my software must have. And I can create a reference if I care to, for example in the form of automated acceptance tests. I don't have to accept bad software, if I don't want to.

[John Wetherbie]

Arguments against doing things that would help improve quality usually seem to always come back to the "we don't have time to do that" argument. In my experience this is because when you want to add tasks (walkthroughs, creating unit test suites, etc) you can quantify the time and it chews into the (usually) already short amount of time on the schedule to get the product out (usually to beat the competition). Would there be some way to apply this MMF/IFM approach to infrastructure/quality tasks to help demostrate that they help get a quality product out the door and don't add much, if any, additional time? Having proof beyond your own experiences would greatly help convince people that quality is worth the effort.

[Ilja Preuss]

Originally posted by John Wetherbie:
Having proof beyond your own experiences would greatly help convince people that quality is worth the effort.

Start by tracking how much time you spend searching for and fixing bugs.

[Valentin Crettaz]

Also, I would say it is weaker because we don't know what quality we must have. There is no reference to compare to; we accept software no matter how bad it may be. --Anuj Upadhyay
It is indeed true that there is no reference to compare to. This is mainly due to the fact that quality is often subjective. There are quality standards out there but they are difficult to apply in reality because different people have different needs. What matters is that the software satisfy the quality requirements of the person who pays for it. And customers should not accept software how bad it may be... Agreed, sometimes they have to because they need it even if bugs are haunting the software.
Well, I actually tend to believe that I know quite well what internal and external quality my software must have. And I can create a reference if I care to, for example in the form of automated acceptance tests. I don't have to accept bad software, if I don't want to. --Ilja Preuss
Well, ideally, yes. But how do you deal with time pressure and hard deadlines then? If the software has to go out by some date, you might have to accept letting some *minor* flaws stay in your software until the next maintenance or corrective release.
Notice that I am not saying the source code is the only document we ever should produce. All I am saying is that writing code is less an activity of construction, but more one of design - and that it is the most authoritative design document we have. Applying a construction metapher to writing code might not be as insightfull as it seems. --Ilja Preuss
I think I'm going to digress here, but anyway... (I might open a new thread for this discussion)
Tell me if I'm wrong but what you are saying is that "the code is the design". To me, the design is something that has the capability of giving an abstract view of the system in a much better way than the code. The design should be something very intuitive to a human being. Code is not intuitive to me. Don't get me wrong, I'm a developer, I love coding. But I still think that coding is not natural to a human being. People started coding by punching small holes in cards, later they started writing code with assembly languages. The languages have evolved and they have been made more human readable. However, maybe we got it wrong all these years, in that the end goal is not to make the computer language accessible to a human but the other way around: making the computer understand what the human means. That way we would have a much more intuitive way of developing systems. The bottom line is that we don't have to adapt to the machine, the machine must adapt to us and we have the power to achieve that since we are creating them. If you come to think about it, this is pretty paradoxal as humans have invented machines for serving them and they end up trying to adapt themselves to them. No wonder things don't work as expected...
[ April 29, 2004: Message edited by: Valentin Crettaz ]

[Lasse Koskela]

Originally posted by Valentin Crettaz:
To me, the design is something that has the capability of giving an abstract view of the system in a much better way than the code. The design should be something very intuitive to a human being. Code is not intuitive to me.

This is (once again) a question of vocabulary... I would say that the code is indeed the design, and that people sometimes need to see different views (abstractions) of that design in the form of UML diagrams etc.

[Valentin Crettaz]

This is (once again) a question of vocabulary...
Agreed, subjectivity matters and that's the reason why I wrote "to me" twice

[Anuj Upadhyay]

Notice that I am not saying the source code is the only document we ever should produce. All I am saying is that writing code is less an activity of construction, but more one of design - and that it is the most authoritative design document we have. Applying a construction metapher to writing code might not be as insightfull as it seems. --Ilja Preuss
I would say that you got me wrong. I wanted to compare the contraction process to software systems development process, this is a high level view and that is what I feel the book also addresses - a more global view of the process. I believe code is very important but *in my view* it has only fractional importance to the whole process. All through the discussion I always consider a global view and that is what MMF / IFM seems to address. Authors, please correct me if I am wrong.
I understood you to say that you think *all* software development projects should conform to some shared process. Didn't you? -- Ilja Preuss
I said that we should have a minimal level of conformance. Note - projects wary considerably in their scope.

And customers should not accept software how bad it may be... Agreed, sometimes they have to because they need it even if bugs are haunting the software. --Valentin Crettaz
I wish I had so much of choice. To give more of my personal opinion, I don't like Windows (topic for another thread) but I don't have another choice currently. I have used it and would have to use it for sometime even though it�s buggy and not a quality product.

[Ilja Preuss]

Originally posted by Anuj Upadhyay:
I wanted to compare the contraction process to software systems development process, this is a high level view and that is what I feel the book also addresses - a more global view of the process.

Ah, I think I understand your point, now... :roll:
For product development, you might be right. But I actually don't know much about how the software industry works today in this regard. All the projects I have worked on until know were for single customers - and that's probably quite a different beast!

[Ilja Preuss]

Originally posted by Valentin Crettaz:
Also, I would say it is weaker because we don't know what quality we must have. There is no reference to compare to; we accept software no matter how bad it may be. --Anuj Upadhyay
What matters is that the software satisfy the quality requirements of the person who pays for it. And customers should not accept software how bad it may be...

Yes, happiness of the customer is the final reference that counts. The art then is to get feedback from this reference as detailed, objectively and early as possible...

Well, I actually tend to believe that I know quite well what internal and external quality my software must have. And I can create a reference if I care to, for example in the form of automated acceptance tests. I don't have to accept bad software, if I don't want to. --Ilja Preuss
Well, ideally, yes. But how do you deal with time pressure and hard deadlines then? If the software has to go out by some date, you might have to accept letting some *minor* flaws stay in your software until the next maintenance or corrective release.

Yes, I might need to decide that. What's important is that I am aware of what is happening and can think about the consequences. That is, I need to actively manage quality instead of letting the software rot because, well, it happens...

[Valentin Crettaz]

Yes, happiness of the customer is the final reference that counts. The art then is to get feedback from this reference as detailed, objectively and early as possible... --Ilja Preuss
...and to be able to incorporate the feedback in the software in a reliable and productive way. Today, I'm afraid we lack the proper tools for achieving that.
Yes, I might need to decide that. What's important is that I am aware of what is happening and can think about the consequences. That is, I need to actively manage quality instead of letting the software rot because, well, it happens... --Ilja Preuss
And may I ask what kind of means do you use to "manage quality"? To be aware of what is happening is one thing, to do something about it is another. I think, human beings are not dumb, we can all be aware of what is happening, we are just helpless when it comes to dealing with complexity because the appropriate tools don't exist yet.
I admit I'm digressing
[ April 30, 2004: Message edited by: Valentin Crettaz ]

[Pradeep bhatt]

Jane,
I just wnat to want whether you book is useful for designers and developers involved in product development or is this targeted for people involved is project development.

[Valentin Crettaz]

I just wnat to want whether you book is useful for designers and developers involved in product development or is this targeted for people involved is project development.
Jane's last post in the following thread kind of answers that
https://coderanch.com/t/130382/Agile/Software-Numbers-not-wide

[Ilja Preuss]

Originally posted by Valentin Crettaz:
Yes, happiness of the customer is the final reference that counts. The art then is to get feedback from this reference as detailed, objectively and early as possible... --Ilja Preuss
...and to be able to incorporate the feedback in the software in a reliable and productive way. Today, I'm afraid we lack the proper tools for achieving that.

Do we? I think it's exactly what Agile Methods are about.

And may I ask what kind of means do you use to "manage quality"? To be aware of what is happening is one thing, to do something about it is another.

Agreed, though I think it's more a matter of care than one of tools.

I think, human beings are not dumb, we can all be aware of what is happening, we are just helpless when it comes to dealing with complexity because the appropriate tools don't exist yet.

What complexity are you referring to specifically?

[Valentin Crettaz]

Do we? I think it's exactly what Agile Methods are about. --Ilja Preuss
I agree to some extent. Agile methods go in the right direction. But we also need agile "tools" that automate agile methods
Agreed, though I think it's more a matter of care than one of tools. --Ilja Preuss
Well, when you have to care about too many things, the danger of stopping to care about some tiny details is not that far. That's why we need intelligent tools that does most of the mechanical (non-business related) work for us. I say non-business related because in the end the thing you (should) care most about is the business logic. All the rest is plumbing, and that plumbing has nothing to do with the business the software is made for. That plumbing should be the responsibility of smart tools, not humans who have to care for it and forget half the screws and bolts along the way.
What complexity are you referring to specifically?
I'm not talking about small applications. I'm talking about large-scale software systems. The tools we have now are maybe sufficient for developing small systems (up to 50-100,000 LOC ), but they are fall short when it comes to developing those large-scale applications.
The complexity I'm talking about is a mix between the great diversity of APIs available out there, the complicated interactions between the components of your system, the opacity of the behavior of a system, the impossibility to build exhaustive test cases for testing 100% of your application, etc. As soon as the design grows (or the code base for that matter), developers, architects, programmers, whoever is working on the project, start to loose the overall view of the system and this lost is a big danger for the resulting system because you have no way to guarantee that it correctly implements the requirements and no way to assess its correctness regarding to the original specification. I have yet to see one single project on earth where the resulting system was considered 100% compliant with its specification. Maybe we do not want systems to be 100% compliant, but then why do we specify the system in the first place?
[ April 30, 2004: Message edited by: Valentin Crettaz ]

[Ilja Preuss]

Originally posted by Valentin Crettaz:
Agile methods go in the right direction. But we also need agile "tools" that automate agile methods

What kind of tools are you thinking of?

Well, when you have to care about too many things, the danger of stopping to care about some tiny details is not that far. That's why we need intelligent tools that does most of the mechanical (non-business related) work for us. I say non-business related because in the end the thing you (should) care most about is the business logic. All the rest is plumbing, and that plumbing has nothing to do with the business the software is made for. That plumbing should be the responsibility of smart tools, not humans who have to care for it and forget half the screws and bolts along the way.

I don't think that will happen in the near future. There are some metrics which can help asses the internal quality, the quality of the "plumbing" of a system, but I don't think that creating a well structured system will be automated anytime soon.
But as a professional software developer, working with other professionals in a well functioning team, I feel it is my responsibility to care about the "screws and bolts", as well as caring about the functionality of the system. It is a matter of pride.
And, by the way, even in the automotive industry, highly reliable *and innovative* systems still rely on motivated and caring professionals - just think about formula one mechanics.

the impossibility to build exhaustive test cases for testing 100% of your application

Why do you think this is impossible?

As soon as the design grows (or the code base for that matter), developers, architects, programmers, whoever is working on the project, start to loose the overall view of the system and this lost is a big danger for the resulting system because you have no way to guarantee that it correctly implements the requirements and no way to assess its correctness regarding to the original specification.

Yes. The only reliable tool I see in this regard is an executable specification which you can run against the system - an exhaustive suite of tests.

I have yet to see one single project on earth where the resulting system was considered 100% compliant with its specification. Maybe we do not want systems to be 100% compliant, but then why do we specify the system in the first place?

That's an interesting question.
I am not sure where this quote is coming from, but it might have been Kent Beck who said something along the lines of "if it is not tested, it isn't implemented".