Author of Test Driven (2007) and Effective Unit Testing (2013) [Blog] [HowToAskQuestionsOnJavaRanch]
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Originally posted by Lasse Koskela:
Welcome to the ranch, Jonas!
I don't think it's such a bad idea to simply go with Word documents on a shared drive (or something similar) unless you have a requirement (not a nice-to-have) that they can't fulfill.
By the way, what do you mean by "test documentation"?
Originally posted by Ilja Preuss:
For unit testing, I don't think that it is very desirable to link them to bug reports or the like. They are just of too fine a granule.
More interesting might be system level / acceptance tests. For this you might want to take a look at http://fitnesse.org , a web-based acceptance testing and documentation tool.
Originally posted by Jonas Larsson:
It turns out that there has been exactly
ZERO effort put into documenting the testing during
almost two years of development on a codebase of
hundreds of thousands lines of code. Much to the
dismay of some of my collegues I've put my foot
down and started pushing for a testdriven development
process that amongst other things features JUnit testing
and an approach for organizing and producing testing/test documents.
Originally posted by Jonas Larsson:
This is not related to JUnit testing. I'm hoping that properly Javadocing the testclasses should be enough documentation for those.
This is for the functional testing of fex GUI functionality in the form
of "Click submit in order ticket -> status should change to pending".
And if it does not a bug is reported and the results document is updated with the bugnumber and the result.
I've actually looked att FITnesse and it does not seem to adress testing *inside* the development process. Acceptance testing is more towards the end of the process right? When customers/users are involved.
Besides...we actually have no requirements document to relate to when
doing acceptance testing. There used to be one, but it is so hopelessly
out of date that it is useless. We have to rely on ad-hoc testing and
try to guess or discuss every testcases expected outcome.
The strange thing is that when all this is done we should be able to
extract a requirements document that describes the current state
of the product. Isn't that backwards to the extreme eh?
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Originally posted by Jonas Larsson:
Besides...we actually have no requirements document to relate to when
doing acceptance testing. There used to be one, but it is so hopelessly
out of date that it is useless. We have to rely on ad-hoc testing and
try to guess or discuss every testcases expected outcome.
The strange thing is that when all this is done we should be able to
extract a requirements document that describes the current state
of the product. Isn't that backwards to the extreme eh?
Originally posted by Ilja Preuss:
As long as you can discuss the expected outcome with someone who actually *knows* what he speaks about, that isn't the worst situation.
Originally posted by Mark Herschberg:
If this is a small team, it can work.
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Originally posted by Mark Herschberg:
How about using Wiki?
I admire you determination to do it The Right Way(TM), but remember that you can only lead a horse to water, you can't make him drink.
You can't go from 0 to 100 overnight. Start out simple. You need to cretae a basic framework for testing (if it doesn't already exist) and testing documention. Then teach them some basics and let them go for a month, documenting going forward (not retro-documenting). After a month, review the process and fix problems that occur. Give it another month. If after 2 months things are going smoothly, then start a small project of retro-documentation into the existing, working doc system. This could be taking a few developers to work on it full time, or it could be having all developers developer a dedicated few hours per week to this task.
The key is that this is not simply another activity to check off on the project plan. This is a cultural change. In fact, two changes (documenting tests and test first development). Go slow, be realistic, and constantly re-evaluate.
--Mark
Originally posted by Ilja Preuss:
Well, in my experience, JUnit tests can be so simple that most often they don't need any documentation at all - they *are* documentation...
What exactly do you mean by "a bug is reported"?
What makes you think so?
FitNesse has its roots in an XP environment, where the Customer is involved during the whole process (one of his responsibilities is to define the acceptance tests which get run at the end of each one-to-two-week iteration).
If you can't do that, I don't see any problems with someone else (product manager, QA or even the developers) writing the tests using fitnesse, though.
As long as you can discuss the expected outcome with someone who actually *knows* what he speaks about, that isn't the worst situation.
Does it work for you? How much guessing do you do, and how often are you wrong (and how do you know about it)?
Originally posted by Mark Herschberg:
OK, that's Very Bad(TM). Now, I'm not an idealist. Rwquirements change and the requirements doc does very often get out of date. However, this is not good.
If this is a small team, it can work. The problem is that that "someone" may not be unique. One of the biggest problems with requirements is that they are underspecified--everything thinks they agree on the behavior but it's not explicitly stated. What happens is that developers each "fill in the blanks" their own way, and it may turn out to be very inconsistent. Given no requirements, your project has a very high risk of running into this problem.
Now as you point out, you're going to define the requirements based on you test cases. Normally when we cretae requirements, we gather everyone together and spend a lot of time discussion and prioritizing behavior. I suspect what will happen here is that you will add tests "as you think of them" and will stumble across conflicting behavior; e.g. in order to make developer A's test work, dveeloper B's test will fail. Now who has preceedence? Instead of resolve this by having all parties involve consider the options, more likely is the handful of people involved will make an arbitrary decision (arbitrary meaning based on their limited information of the project). This is not good.
Originally posted by Jonas Larsson:
--------------------------------------------------------------------------------
Originally posted by Mark Herschberg:
How about using Wiki?
--------------------------------------------------------------------------------
Can you import images (screendumps) and make
tables in an easy way in a wiki solution?
Everyone thinks it's absoultely fab, as long as they don't have to do the work. So up to this point I have been doing most of the research and setup work to get it running to smooth it out.
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Originally posted by Jonas Larsson:
It definately is not. But what can I do with a product that has 300K-400K
lines of code. I'm just one person and we are NOT gonna get the resources
necessary to backtrace everything into a req document. That I know.
Originally posted by Jonas Larsson:
This is exactly how it works now. And sometimes it feels as there are
not enough hours in a 24-hour day to have the meetings required to agree
on every little detail. The level of specification has to hit a level
that works for eveyone but is not too detailed. And then there is the
thing about conflicting requirements wich seems to happen a lot...
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep
Originally posted by Amy Phillips:
I found this topic very interesting, I am actually studying it for my university dissertation with the intent to build a test management and bug tracking tool. I am unable to see why people insist on storing test cases speratly from test results and defect reports. Surely they are all linked?
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime. |