I have a situation where I would like to repeat a set of tests but with a different configuration. For example, assume a test class with 4 test cases, and where I have three different possible configurations (all valid at the same time). What I would like to do is run the test class three times, once with each configuration and have the junit test report show a total of 12 test cases being run.
I've looked at using TestSuite and the JUnit4TestAdapter class to define the tests to run, and using a for loop to run the tests multiple times changing the configuration between runs. And that works fairly well, but reporting is lacking (shows up as only a single test in the report). I have also looked into using the @Suite.SuiteClasses annotation, but that allows running the test cases only once; I guess I could declare 3 test suite classes with the @Suite.SuiteCLasses annotation and in each one set up the configuration but that isn't much better than the workaround I am using right now (more on the workaround later).
Of course, what I have given here is a simple example. I really have about a dozen classes with well over a hundred test cases that I need to run in 5 different configurations. To make things even more interesting, I have about another 50 test classes with perhaps 6-700 test cases that are configuration independent. My goal is, in a single integration test run, to run the configuration independent tests once and the configuration dependent test cases 5 time each, each time with a different configuration, and to end up with a single report showing the total number of individual test cases run (approx 1200 test cases total).
My current workaround is for the configuration dependent test classes, I have an abstract class with the test cases defined, then 5 subclasses each with only an @BeforeClass methods that sets the configuration. This works. But it is a miantenance headache if I have to add a new configuration (need to create a lot of new subclasses) and is also a pain when in a particular test environment only a subset of the configurations are available (right now I have the @BeforeClass methods checking if the configuration is present). I do have an API where I can determine the available configurations, so ideally I'd like to do a loop that runs the configuration specific tests once for each available configuration. And as I mentioned I can run the tests in a loop, but the final report doesn't include the individual tests. (If I knew how to grab the TestResult object being used by Maven to report on the tests, that would probably work for me as I could pass it off to TestSuite.run().)
Any ideas, thoughts or suggestions would be appreciated.
By the way, I'm using JUnit 4 and running the tests within Maven.