for me, the basis for all the agile methods is expressed in Agile manifesto. At least that's how I understand "being agile" in the big picture. Although I agree with most of the points from the philosophical point of view, I can also see practical problems in the effort to use agile methods. I come from a rather special background - I'm a long-time SAP/ABAP developer (now widening my horizons with Java :-)) and in effect the only developer in the company at the moment. I feel I follow the agile principles (mainly because I agree with that philosophy), but I don't use any rigorous method. There are two "whys": the first is me, not seeing any practical benefits of using any such method, and the second is the company, that doesn't feel like being pushed into following it. So the first question would, what can the company (and also the developers) gain from using XP, Scrum or such, instead of just do what you intrinsically feel is the right way? The second question is, how to find the right extent of following agile principles? For example, it's surely true that working software has greater value than rigorous documentation, but I can't imagine omitting it at all (especially in enviroments similar to mine, where there is no substitution of developers). Does it come in hand with long-time maintainability?
Andrew has said this in other threads, I'm the same way, and it looks like you lean that way too: it always goes back to the values and principles of the Agile Manifesto. For me, agile is not so much about the collection of tools and practices that are used as it is about the mindset and attitude that you need to have to use them properly. Tools and processes come and go but values and principles stay pretty much the same. An example that I mentioned in another thread is how there's a growing number of experienced practitioners who are abandoning user story point estimation. The approaches may be totally different but the principles and values behind them are the same and the goal is always to do things better.
As for finding the balance, I think that's one we have to find out for ourselves in every particular situation we find ourselves in. It will be different every time.
Something that I think doesn't get mentioned enough is Conway's Law - I think that knowing about this law and factoring it into your decisions about adopting Agile is very important because if your organizational structure is not compatible with agile principles and values, then adoption will be tough.
Tomas, you asked a really good question that gets to the heart of what agile is all about. Actually, Mike Cohn does a fantastic job of summarizing it in the forward he wrote for us in Learning Agile -- and I think it's very relevant to what you're asking about:
Early agilists agreed on a set of principles enshrined in the Agile Manifesto, and many practices were shared across multiple agile approaches. However, there was fierce debate about whether a team should start by understanding the principles of agile software development or whether they should begin by performing the practices even before developing a deep understanding of why.
Proponents of starting with practices took a wax-on/wax-off view of the world. If a team were to act agile, they would be agile. By going through the motions of being agile—pair programming, automating tests and builds, using iterations, working closely with a key stakeholder, and so on—a team would gradually develop an understanding of the principles of agile.
Proponents of starting with principles, on the other hand, contended that practices without principles were hollow. Going through the motions without understanding why did not lead to agility. Agility was (and still is) a focus on continuous improvement. The argument went that a team could not improve continuously if they didn’t understand why they were doing the things they were doing.
In Learning Agile, Andrew Stellman and Jennifer Greene do the best job I’ve encountered of stressing the principles of agile without de-emphasizing its practices. They point out that following practices without knowing why is likely to lead only to what they call a “better-than-not-doing-it” level of success. That is, implementing practices alone is helpful, but it falls far short of the true promise of what becoming agile can truly deliver.
I worry that you're falling into the trap of trying to adhere to the agile principles without really adopting the practices. And the problem is that the devil is in the details. It's easy to feel like you've valued working software over comprehensive documentation, or that you put customer collaboration over contract negotiation. But it's very difficult to gauge just how much those values are reflected in your work without the practices, because the practices are where you actually discover real-world clashes between the values and the work you have to do every day.
And the second question you asked can actually shed some light here:
The second question is, how to find the right extent of following agile principles? For example, it's surely true that working software has greater value than rigorous documentation, but I can't imagine omitting it at all (especially in enviroments similar to mine, where there is no substitution of developers).
The agile manifesto doesn't say that you should omit rigorous documentation. It says that we value customer collaboration over contract negotiation. But it also says: "That is, while there is value in the items on the right, we value the items on the left more." There's value in documentation, even comprehensive documentation. I talked a bit in other posts about where you might want comprehensive documentation -- like in a company where publicly making a mistake can end your career.
So how do you know whether you're working at a company or on a team where you may actually need to place a little extra value on documentation? Trying to adopt agile practices -- and especially a complete agile methodology like Scrum -- is a very effective way to do so. If you try to establish a Scrum backlog and do sprint planning, you may very quickly discover that your company has a contract negotiation attitude: if they don't trust you to deliver value even if the specific features are not decided until sprint planning, they'll demand detailed long-term plans up front. This is the sort of value clash that you will have a hard time discovering without the practices.
That's why we teach the practices in Learning Agile, but also give many real-world examples of how those practices only get better-than-not-doing-it results when the team and company culture don't match the principles. But we still spend a lot of time actually teaching how to use those practices on a real team in the real world, because that's how to make positive change.
Author of Head First Agile, Learning Agile, Beautiful Teams, Head First C#, Head First PMP, and Applied Software Project Management (O'Reilly)
Thank you guys for your valuable answers, such thoughts really help me to sort out things about the topic, kind of ensure me that I'm on a good way (or at least not the worst one), and motivate me to study this field further.
There were millions of the little blood suckers. But thanks to this tiny ad, I wasn't bitten once.
a bit of art, as a gift, that will fit in a stocking