If I run each test in the upTo5sec category separately, then tests will end without errors
If I run all the tests together then some give a timeout and some errors.
List.txt defines which tests will be executed. It can be generated by listgenerator.sh
If list.txt contains only upTo5sec/bacp/bacp-22 test will be passed.
If list.txt contains only upTo5sec/bacp/bacp-21 test will be passed
If list.txt contains another test bacp-21 will be timeout and at the same test upTo5sec/bacp/bacp-21 will be failed.
In some way timeout(bacp-22) affects another test(bacp-21).
That means you're tests don't work in isolation. Are you setting up and tearing down your test data correctly between each test? Show us the code of two tests that fail when they are run together, and your setup and teardown code.
Copying files sounds like your problem. Unit tests should not perform file system operations. If you need to read data, read it from a resource embedded in your application. The resource must NOT be altered.
Yeah... I'm sorry to say this, but you need to go back to the drawing board and redesign your tests. There are many issues with your code:
It's complex. There's no point to unit testing if bugs can be introduced in the tests themselves.
It's not clear what your test cases actually test. Tests must be small, easy to read and they must test one thing really well.
It's performing file IO. If you have a whole lot of data that you don't want to hardcode, create reusable test fixtures that read data from application resources.
It uses inheritance for the wrong reasons. Is your MinizincBasedTestUpTo5Seconds really also a MinizincBasedTestHelper? Don't use inheritance just because you want to easily add behavior to your class.
You're not using the right tools for the job. All the error messages you're throwing around suggest to me you're not using the power of your testing framework.
For now, the best suggestions I have for you to clean this up is to write a couple of test fixtures that create strongly typed objects out of resources, and to use Hamcrest's matchers to perform your assertions. I'll write an example of how to do this shortly.
Okay, let's make a simple example where we mix additive colors together. Let's limit the set of colors to just the primary and the secondary colors:
Now, we want to write a test for the mix() method. We can do this really easily with JUnit's experimental theories API.
For each theory, JUnit creates a Cartesian product of the data points it can assign to the theory's parameters. The assumptions we make at the start of the theory filter out combinations that are not valid to test that particular theory. Then we perform the method under test, and make assertions about the result.
Note that we already wrote 6 test methods to test invariants of one method under test. Each test method tests a very specific invariant, and it's clear from the code what is being tested in each method.
Now, in this case it's very easy to get the data points. But what if we have many data points that we need to generate from configuration data? You can then write a test fixture:
The nice thing about this is that once you have written your fixtures that contain types that are either mocked or loaded from a resource, you can easily fill your data points in many test classes.