System Tests

[Chris Harris]

Hi all,

I am currently using JUnit to test my code. This works great by mocking some resources I am able to run 1000s of test in seconds.

However I am know aware of people using JUnit support in order to run system tests. They are still using the Junit saying "Keep the line green and your code clean" even thought they are using system tests.

I am fear that people will start using JUnit as a system test tool and forget about the advantages real unit test give you.

I am not saying that people should not do system tests just that using JUnit may not be the best way to do it.

Does anyone know for tools for system testing that can stop people missing JUnit.

I would be real nice I there was a System Test API (JSystemTest)

Thanks

Chris.

[Lasse Koskela]

What's wrong with using the JUnit framework for launching your system tests? (yes, that's a serious question to which I'd like to hear opinions about)

[Chris Harris]

Hi Lasse,

What's wrong with using the JUnit framework for launching your system tests?

There is really nothing wrong with using the Junit framework for launching system tests as long as you are aware that what you are doing is SYSTEM TESTING.

As part of our coding standard we are asked to write JUnit test for all classes developed. This started out well however know that people have started to use the framework to do system tests they are now not writing so many unit test. This is because they claim that they have written unit test but when they have actually done is write system tests.

I am now finding that I have to set up a load of different environments in order to run all the tests. It can take time to find the required hardware.

So what is happening is that some of the tests are not run until the development is finished. This takes away the advantage of quickly running the unit tests during development.

I think that it may be unfair to say that it is a misusing Junit to launch system tests. What I really see my problem being is that people are getting confused about the difference between a system and a unit test. There are some people that think because they are using JUnit they have written unit tests.

Chris.
[ June 28, 2004: Message edited by: Chris Harris ]

[Jeanne Boyarsky]

Chris,
We use junit for two types of tests. "Pure unit tests" run without anything external. "Integration tests" require the database and/or server to be up. These integration tests test the public API of our components.

The two types of tests are in different projects. We make sure to have an All_XXX_Unit_Tests and All_XXX_Integration_Tests to differentiate the two. That way we emphasize the difference.

I like using junit for the integration/system tests because you can automate the testing better.

[Ilja Preuss]

Originally posted by Lasse Koskela:
What's wrong with using the JUnit framework for launching your system tests? (yes, that's a serious question to which I'd like to hear opinions about)

It's not exactly *wrong*, but it's nice if non-developers can read system tests (because they are in fact an executable specification for the whole system), and JUnit tests aren't very readable for non-developers.

That's why I prefer to use something like FitNesse for system tests: http://fitnesse.org/

[Jeanne Boyarsky]

Lasse/Ilja,
Can one of you guys go back and explain how you define system tests?

Thanks.

[Lasse Koskela]

I define system tests as "testing the whole software system, including any databases etc.".
Some people call them "functional tests", "integration tests", etc.

[Ilja Preuss]

Originally posted by Jeanne Boyarsky:
Lasse/Ilja,
Can one of you guys go back and explain how you define system tests?

End-to-end-tests, in contrast to testing single units.

[Chris Harris]

Hi

Thanks for your responses so far.

Is it just the projects I have worked on or does everyone fine that if you give most developer a choice to use Junit to write a unit test or a system test, 70% will always do system tests?

I can put this down to two main things:

Firstly not all developers know the full advantage of writing full unit test.

Secondly it can take a little more time in the short team to mock all external resources.

Chris

[Lasse Koskela]

Originally posted by Chris Harris:
Is it just the projects I have worked on or does everyone fine that if you give most developer a choice to use Junit to write a unit test or a system test, 70% will always do system tests?

Yeah, I've heard about such a trend... If 70% writes system tests, approximately 20% writes poor unit tests, and 10% writes good unit tests.
Anyway, the problem lies in how you design your software -- not in how you write JUnit tests.

[Chris Harris]

Lasse,

Yeah, I've heard about such a trend... If 70% writes system tests, approximately 20% writes poor unit tests, and 10% writes good unit tests.

What do you class as a poor unit test?

There a tests that may not cover all the functionality but I would not count this as poor unit tests, only that there are tests missing.

Chris.

[Lasse Koskela]

Originally posted by Chris Harris:
What do you class as a poor unit test?

There a tests that may not cover all the functionality but I would not count this as poor unit tests, only that there are tests missing.

A poor unit test
- tests something that can't break
- doesn't test something that could break
- is difficult to read (doesn't communicate intent clearly)
- runs slowly

Those are just some points that came to my mind. That's a good question though. I'd like to hear how others would classify "poor unit tests"?

[Ilja Preuss]

That reminds me of a story I heard about a project that was outsourced. The team was required by contract to do XP "by the book".

Surprisingly, the developed system was quite buggy, though having 100% test coverage. Well, not so surprising anymore when they discovered that the book didn't tell anything about the fact that unit tests actually should contain assert statements... :roll:

[Jeanne Boyarsky]

I'd like to hear how others would classify "poor unit tests"?

Interesting question:

Lack of coverage

Only tests main path (no exception cases)

Commented out tests

Only testing one piece of output. For example, tests the method returns 5, but not that it updates the cache.

Tests that don't get run

Tests that require external setup

Integration tests

I know the last two aren't really unit tests, but many people think they are because they use junit.

[Chris Harris]

When I first start this topic I was wondering about system testing frameworks that exists. After reading your comments I still believe that using JUnit to do system testing and unit test can be confusing for some people.


(Declaimer I thought of this idea while writing this message ). In order to reduce the confusion that there is about when people are writing Junit test to do system tests and writing Junit test to do unit test it may be worth thinking of extending the Junit framework. This extension would involve creating two separate TestCase Types. One for system test and another for unit test. Then using AOP; fail test if the test are using the UnitTestCase and attempt to use an external resource.

Hmm is this even possible?

Any thoughts on this?

Thanks

Chris.

[Ilja Preuss]

I am not sure that this would do much good. If the developers are dull, they will simply test less or find a workaround for the limitation. If they are not, they will feel patronized.

If a developer doesn't know the difference between a unit test and a system test, he doesn't need a better software framework, he does need better education!

[Chris Harris]

I am not sure that this would do much good. If the developers are dull, they will simply test less or find a workaround for the limitation. If they are not, they will feel patronized.

Possibly, however after thinking in more about it I think AOP could work well here. My thinking is that System Testing and Unit Test are just different aspects of Testing.

There are a number of rules that distinguish the different between a System Tests and a Unit Test. I don't see any reason why a framework could not enforce the rules.

As you say a developer would find workarounds that would allow them to break the rules. However this is always going to be the cause with some people as there are people who want to do as little as possible.

Firstly I agreed that some developer may find it patronizing however thinking a little more about I disagree. The aim of the behind enforcing the rules within the framework would ultimately make there job easier.

Chris.

[Lasse Koskela]

What would your aspect do when encountering a system test? Nothing, I presume, since system tests are supposed to be end-to-end testing.

Ok, then what would your aspect do when encountering a unit test? Replace real database connections with mocked up database connections? Otherwise a good idea, but the behavior of the mocked up database connection object is not something you can specify with a global aspect -- it's something that varies per test.

[Chris Harris]

Ok, then what would your aspect do when encountering a unit test? Replace real database connections with mocked up database connections? Otherwise a good idea, but the behavior of the mocked up database connection object is not something you can specify with a global aspect -- it's something that varies per test.

My thinking was that the Unit Test aspect would only fail tests if an attempt was made to access an external resource. I agree that it is not possible to create a global aspect that could do all mock creation for you.

Recently I have changed my total approach to mocking. In order to explain the approach I think it may be easier to understand the reasons:

When I first started to mock object, like most people I developed my own mock classes. I soon found that in order to create a mock class that could be used in all my tests would take time to develop fully.

In order to reduce the develop time I started to look around for mock frameworks and discovered MockCreator. I find this a great tool and am sure it has saved my hours as I get MockCreator to generate all my mock classes. However I have recently discovered that some of the team members are no longer creating unit tests as they find that MockCreator takes a time while to set up all expectations. In my option this set up time is justified as it forces that all expectation are defined or the test fails. However as people are no longer creating unit test I had to think a better approach.

In order to help the team mock objects I have wrapped the generate classes from MockCreator into my own mock classes. For example:

//This class reads directly for a configuration file.
public class ConfigReaderOrg {
 
  
  public String getConfigValue(){
    return configValue;
  }
}
 
//Wrapper of the mock class generated by MockCreator
public class ConfigReaderMock {
  
  //MockConfigReader is the class generated by MockCreator.  
  MockConfigReader mockReader;  
 
  //Wrapping MockCreator method into simple set methods.
  public String setConfigValue(String value){
    mockReader.expectGetConfigValue(value);
  }
 
  public ConfigReaderOrg getMock(){
    return mockReader;
  }
}
 

Then in tests I use ConfigReaderMock to mock all reading from the configuration file. Example

 
 
public class MyTest extends TestCase {
 
  public void testMyTest(){
 
    //need to mock the config reading.
    ConfigReaderMock config = new ConfigReaderMock ();
    config.setConfigValue("testValue");
    
    //Make the system use the mock object
    ConfigReaderFactory.setReader(config.getMock());
    
    //Do some testing.
  }
}
 

Hope you are able to understand all that. That is my current testing strategy. However I feel that this could be improved separating the mock population out of the test:

 
 
public class NewMyTestMock {
 
  public void mockMyTest(){
    //need to mock the config reading.
    ConfigReaderMock config = new ConfigReaderMock ();
    config.setConfigValue("testValue");
    
    //Make the system use the mock object
    ConfigReaderFactory.setReader(config.getMock());
    
  }
}
 

public class NewMyTest extends TestCase {
   public void testMyTest(){    
    //Do some testing.
  }
}
 

The idea would be to use an AOP framework to intercept the call to testMyTest. Before the method is call I would call mockMyTest within NewMyTestMock.

Thanks,

Chris.

[Lasse Koskela]

Ok. I think I get your point.
I believe the common way of doing this is to put the setup stuff into an abstract super class TestCase which all the concrete TestCases inherit. The AOP approach could be a bit cleaner, though.

[Chris Harris]

I believe the common way of doing this is to put the setup stuff into an abstract super class TestCase which all the concrete TestCases inherit. The AOP approach could be a bit cleaner, though.

Yeah, I have used the setup to do this in the past but found that it can be messy if multiple test within the same TestCase need to use different setups. For example if you want to set up the mock object with different values for two test the can no longer use the setup to populate the mock object.

Chris.

[Lasse Koskela]

By the way, it looks like Cactus 2 might incorporate AOP into its architecture...

[Chris Harris]

Thanks your the leting me know. I see they are thinking of using AspectWerkz which is a AOP I have no tried. Think I am going to give it a go.

Thanks again,

Chris.

[Chris Harris]

Do you happen to have a link to the Junit roadMap? I would be interested to see if they are also planning to go down the AOP route.

Thanks,

Chris.

[Warren Dew]

Missed this thread the first time around.

What do you guys consider an "external resource"? I don't see how to create a robust unit test for, say, a class which makes a bunch of JDBC calls without running a database. As long as the test is repeatable - for example, if it runs a standard database creation script for setup, and removes the database in the tear down - I don't see the problem.

Traditionally, unit tests only use testing stubs until the rest of the code base is written. At that point, you quit using the testing stubs and use the real thing, as that gets you a better test. I don't see why things should be different using JUnit.

[Ilja Preuss]

Originally posted by Chris Harris:
Do you happen to have a link to the Junit roadMap? I would be interested to see if they are also planning to go down the AOP route.

There is no roadmap, and I don't think you will see much changes on core JUnit at all in the future. It's meant to be small and lightweight, it is doing what it should do, it's robust - what changes does it need?

[Ilja Preuss]

Originally posted by Warren Dew:
As long as the test is repeatable - for example, if it runs a standard database creation script for setup, and removes the database in the tear down - I don't see the problem.

The problem is speed - those tests tend to be quite slow. Especially if you do test driven development, where you want to run the tests every couple of minutes, it makes a big difference wether the suite takes a couple of seconds or a couple of minutes.

[Chris Harris]

Posted by Ilja Preuss:

The problem is speed - those tests tend to be quite slow. Especially if you do test driven development, where you want to run the tests every couple of minutes, it makes a big difference wether the suite takes a couple of seconds or a couple of minutes.

I agree with this and would also like to add the advantage that if you mock the database then you no longer have to spend the time setting up the database before each test.

My tests are also run within the ant build script. This is done so that all tests are run before it is given to a customer. The distribution team would not be too happy if they had to setup a database just to build the application.

Posted by Ilja Preuss:

There is no roadmap, and I don't think you will see much changes on core JUnit at all in the future. It's meant to be small and lightweight, it is doing what it should do, it's robust - what changes does it need?

As you see from my previous posts I would like to update the existing JUnit framework. This update would separate System Testing from Unit testing and it would also be nice if it had some mocking features.

Chris.

[Warren Dew]

Ilja Preuss:

The problem is speed - those tests tend to be quite slow. Especially if you do test driven development, where you want to run the tests every couple of minutes, it makes a big difference wether the suite takes a couple of seconds or a couple of minutes.

By "suite", do you mean a JUnit TestSuite object - which I've most commonly seen used to group all tests for a single class or unit - or do you mean a regression test that includes all unit tests for the entire system?

If the former, I've only once seen a unit test take more than a couple seconds, and that case involved a poorly designed test that was soon fixed (and didn't involve a database). Setting up and tearing down a typical test database takes only a fraction of a second in my experience, with Oracle and MySQL.

If the latter, I guess it's a matter of different coding styles - I don't have a problem if a regression test for all the modules takes a couple minutes, and I don't think I'd really trust a set of tests that didn't use the actual database to parse the embedded SQL code handed to JDBC.

[Ilja Preuss]

Originally posted by Warren Dew:
If the latter, I guess it's a matter of different coding styles - I don't have a problem if a regression test for all the modules takes a couple minutes, and I don't think I'd really trust a set of tests that didn't use the actual database to parse the embedded SQL code handed to JDBC.

Me neither. That's why I would want to have *both* styles of tests: the fast ones (using mocks or something) which I can run in a matter of seconds to catch the most blatant bugs, and the slower ones which wouldn't be run as often (perhaps by an integration server or in nightly builds) but would catch the more subtle bugs, too.

[Lasse Koskela]

Originally posted by Ilja Preuss:
There is no roadmap, and I don't think you will see much changes on core JUnit at all in the future. It's meant to be small and lightweight, it is doing what it should do, it's robust - what changes does it need?

One change I'd possibly like to see would be some standard way to tell the JUnit TestRunner that a given test class should or shouldn't be included in a test run of type X. For example, it would be nice to mark your JUnitPerf tests with a "label" that excludes those tests from regular unit tests but includes them when doing a "full" test.

Another possible enhancement might be an attribute-driven approach to writing test cases like the one NUnit is using.