Grouping categories of unit tests and their execution

[Allan Halme]

Alright Vincent, here's a question for you that's relevant to me on my project right now.
We've inherited an existing J2EE application that we'll be maintaining and in which we'll be implementing new features as well. There aren't really any existing unit tests, but from here in we'll be writing tests for the existing code and as we write new stuff.
The tests fall into four general categories:

"Standalone" tests, i.e. that don't require any containers or scaffolding beyound the basic JUnit to be executed.

Database tests, where we're using DbUnit and connecting to an Oracle schema that we've got set up for unit testing.

Servlet and EJB tests, where we'll quite likely run the tests in the J2EE container, with the aid of a combination of Cactus and mock objects.

Functional tests that use JWebUnit (you could argue that these aren't really unit tests, but we're writing them and running them as if they were)

Do you have any suggestions or guidelines for managing, grouping, and executing this kind of sets of tests?
Our thought is that we should be able to run each of the four groups of tests separately, because some groups of tests take longer than others to execute. I don't think it will be possible to get all the tests to run in under 120 seconds -- maybe just the first set.
We could run the first set just about every time we compile, run the second and third set every time we feel we've completed a small unit of coding, and run the fourth set before each coffee break . We'll also be running the the whole shebang on our ContinuousIntegration server.
Any comments or suggestions very welcome. This is one of our number one priorities on our project right now.

[Michael Zalewski]

To manage this type of automation, put all the test classes into their own packages. Then use the junit task in ant.
You could even set up a seperate ant target for each class of test. And use a fileset to pick out all the tests that need to run in each target.

[Allan Halme]

Yeah, Ant's in use by default
And, indeed, that's what we've already got set up for the first category. We've got a naming convention whereby test classes named **/*StandaloneTest.java are of the -- drum roll -- standalone type, and get run by the junit-standalone target.
But it's a bit wordy in the class name, and I'm not really happy about having to name the other categories **/*DatabaseTest.java, **/*ContainerTest.java, **/*FunctionalTest.java, etc. But if we don't have a naming convention like this, then we have to have a fileset explicitly listed somewhere, and then whenever you write a new test class, you also have to remember to add it to the list, which is (too) easy to forget ...

[Karthik Guru]

Ok so when u meant 4 different categories, you actually dont have these tests in 4 different packages?. If that were the case i guess it w'd have been pretty straight forward to run the tests under the different categories
without specifying any complicated matchign pattern.

[Lasse Koskela]

But it's a bit wordy in the class name

Why don't you use "*DTest.java", "*FTest.java", "*CTest.java", "*STest.java", then? It's not that expressive but still quite easy to pick up. And, as suggested already, having subdirectories for each type of test is an option.
src/com/foobar/widget/MyClass.java
src/com/foobar/widget/dtest/TestMyClassThis.java
src/com/foobar/widget/stest/TestMyClassSomething.java
etc...
or
src/main/com/foobar/widget/MyClass.java
src/ftest/com/foobar/...
src/stest/com/foobar/...
src/dtest/com/foobar/...

[Allan Halme]

Originally posted by karthik Guru:
Ok so when u meant 4 different categories, you actually dont have these tests in 4 different packages?.

Right. Test classes are located in the same Java package as the class being tested. All the test classes are, however, under src/junit, whereas main code is in src/java.
For example, the same Java package may contain an EJB plus come helper POJOs that merit their own tests, thus we have 2 categories of test classes in the same Java package.

Originally posted by karthik Guru:
If that were the case i guess it w'd have been pretty straight forward to run the tests under the different categories
without specifying any complicated matchign pattern.

Yeah, I want to try to avoid the filename pattern matching if possible, but I haven't really figured out anything yet that I'm fully happy with.

[Allan Halme]

Originally posted by Lasse Koskela:
Why don't you use "*DTest.java", "*FTest.java", "*CTest.java", "*STest.java", then? It's not that expressive but still quite easy to pick up.

If we're gonna have a naming convention, I prefer the fully-qualified non-abbreviated fully expressive form, not the FQNAFEF

Originally posted by Lasse Koskela:
And, as suggested already, having subdirectories for each type of test is an option.
src/com/foobar/widget/MyClass.java
src/com/foobar/widget/dtest/TestMyClassThis.java
src/com/foobar/widget/stest/TestMyClassSomething.java
etc...
or
src/main/com/foobar/widget/MyClass.java
src/ftest/com/foobar/...
src/stest/com/foobar/...
src/dtest/com/foobar/...

Of these, the latter is an option I will consider seriously, thanks for the idea. I prefer it over the former option because I want to keep the test classes co-located in the same Java package as the class being tested, as I say above. That way it's always straightforward to find a class's test class.

[Vincent Massol]

I'd strongly suggest you use separate directories (but possibly same packages) for your different kind of tests. You'll win everything this way.
src/test/junit
src/test/cactus
src/test/database
[...]
That's because you don't want to run all these tests together so you need to be able to separate them and a filter on the name is not enough and prone to errors. It's also because this will integrate better with existing tools such as Maven, etc.

[Peter Chase]

Perhaps this is tangential, but...
You can group tests by making your own TestSuites. You do not always have to accept the TestSuites that get made by JUnit's own behaviour. TestSuites can be put into TestSuites, to make a hierarchy.
You might have a suite of all tests, containing a suite the unit tests, a suite for db tests and a suite for gui tests. You can then execute all tests or a useful subset of tests, easily.
The Swing TestRunner maps hierarchies of test suites nicely into a tree view.

[L Duperval]

Originally posted by Peter Chase:
Perhaps this is tangential, but...
You can group tests by making your own TestSuites. You do not always have to accept the TestSuites that get made by JUnit's own behaviour. TestSuites can be put into TestSuites, to make a hierarchy.

Ahhh! Now why didn't I think of that. We use the **Test stuff in ant to do this, but this is a much better approach.
L

[Lasse Koskela]

Do remember that you need to manually add a line into your TestSuite class when you create a new TestCase... Unless there's a way around this?

[Jeanne Boyarsky]

If there is any pattern, you can use reflection to generate the test suites. That takes care of the problem of not remembering to add a new class to the test suite.

[Lasse Koskela]

Reflection? Doesn't that mean that you have to write a TestSuite which traverses through the source directory tree and picks up the names of those files which extend TestCase?

[Allan Halme]

Originally posted by Vincent Massol:
I'd strongly suggest you use separate directories (but possibly same packages) for your different kind of tests.

If we can solve one last problem, then I think I'll do this.
What if a class Foo has both standalone and database tests? I don't like the idea (and don't know if it would work too well at all) about having a com.company.FooTest test under src/junit and another having the same name under src/database ...
Though I guess in these kinds of cases you'd just have the one test class under src/database and be done with it.

Originally posted by Peter Chase:
You can group tests by making your own TestSuites.

In principle, this is a great idea. However, as mentioned in a few later posts, this requires either manual maintenance or a custom TestSuite to traverse the folder structure and to filename or directory name matching. (Correct me if I'm wrong.)

[Lasse Koskela]

Originally posted by Allan Halme:
What if a class Foo has both standalone and database tests? I don't like the idea (and don't know if it would work too well at all) about having a com.company.FooTest test under src/junit and another having the same name under src/database ...

At least Eclipse shows a compilation error if the project's source directories contain multiple classes with the same fully qualified name. Also javac will complain about a duplicate class definition. There could be problems with an Ant script if you compile the two source trees in separate steps, though.

[Vincent Massol]

Allan,
Then simply use different names. For example FooTest and FooDatabaseTest.

[Allan Halme]

Originally posted by Vincent Massol:
Then simply use different names. For example FooTest and FooDatabaseTest.

Ah yes, but the purpose of the directory structuring was to avoid having to give funky names to the test classes ...

[Jeanne Boyarsky]

We use FooTest for the pure unit tests (that don't require a database or server.) Some tools, like JTest, honor this convention and will run your unit tests. These tests are in a parallel directory structure.
We put the database tests in a different package that contains the phrase test. For example, org.test.name.FooTest. This system makes searching by reflection really easy.

[Vincent Massol]

Originally posted by Allan Halme:

Ah yes, but the purpose of the directory structuring was to avoid having to give funky names to the test classes ...

There are other advantages:
- a test is not simply made of a single test class. It can be composed of static Mock object classes, resources (like xml files, etc - Although in most cases, these resource could be created from inside the test classes but that's another subject of discussions )
- when you run the test you need different runners for different types of tests (they also don't run with the same lifecycle). Thus you need to separate them. Using name is neither efficient nor foolproof. It also leads to more complex builds. More importantly, in all tools you use, you'll need to find out how to exclude some tests/classes which may either not always be trivial or not supported.
- it's simply cleaner and easier for anyone new to the project. They'll understand right away the directory structure of the project and be able to run the tests more easily. Imagine if everything is together and they're in an IDE, they'll try to run some tests thinking they are pure junit tests for example, but it won't work as they are cactus tests (say). etc.
- using Suites to group test is not so nice as you'll often forget to add test classes to the suite.

[Christian Hargraves]

Just a thought. This might seem a little strange, but you could put certain text in the javadocs or even the code of the actual tests that represents what type of test it is ( like "DATABASE", "ACCEPTANCE", "UNIT" ..). Then you could use the choose tag in the fileset to select files that only match the text provided. This way, you wouldn't have to have long names, follow a certain package structure nor create test suites and a test could be categorized in multiple ways simply by adding the appropriate text or constant from another class to the file.

[Lasse Koskela]

An interesting approach. However, that wouldn't work with Ant, would it?

[Ilja Preuss]

Another idea is to use a tag interface. Use a RecursiveTestSuite with a TestFilter filtering all the tests not implementing that interface.
[ November 12, 2003: Message edited by: Ilja Preuss ]

[Christian Hargraves]

Yes that does work with Ant 1.5 and higher.
Take a look at the
examples in the Ant manual
I use this in Jameleon ant it works quite well.
[ November 12, 2003: Message edited by: Christian Hargraves ]

[Lasse Koskela]

Originally posted by Christian Hargraves:
Take a look at the
examples in the Ant manual

Very nice!