testing vocabulary - do you agree with this vocabulary?

[paul wheaton]

This thread is dedicated to the first section of this unit testing article focusing on unit testing vocabulary. Agree? Disagree?

After all, any putz with a web site can stand up and declare a vocabulary. But this is an area riddled with so many problems, I think it would benefit all of us if we could say "yes, that's spot on" or "I agree with all of it except" or ... whatever. Let's figure out at least what we do agree on!

There will be several threads tying back to that article. This thread is only about the vocabulary at the top of the article.

[Ernest Friedman-Hill]

One statement I disagreed with: "Functional test suites are usually run with JUnit." I guess it depends what kind of code we're talking about, but I've never written functional tests with JUnit. Functional tests are application tests, and as such, use the application's interface, not the coder's interface. Now, I'm unusual, maybe, in that most of my projects include a scripting interface of some kind. But I've got three different projects now with large functional test suites, all of which use the same shell-script driven test harness. Not only don't we use JUnit, but it would be exceedingly awkward to do so.

[Ernest Friedman-Hill]

Another disagreement is "A unit test never uses the filesystem". If this were an article about testing Ruby, maybe I'd buy that. For Java, though, following this would force me into an unreasonable corner, if only because somehow encoding test data into Java code (in one case, finite element mesh data in NetCDF format) would be so difficult it'd be a strong deterrent to testing.

I guess you could argue, then, that those tests should become functional tests. but I think deciding that a test is a functional test because of the form of the test data is artificial and odd.

[paul wheaton]

Originally posted by Ernest Friedman-Hill:
One statement I disagreed with: "Functional test suites are usually run with JUnit." I guess it depends what kind of code we're talking about, but I've never written functional tests with JUnit. Functional tests are application tests, and as such, use the application's interface, not the coder's interface.

I think what you are calling "application tests" are what I was calling, in the vocabulary, "component tests." So a functional test is possibly exercising the component as whole, but, in my experience, it is usually exercising a small part of a component. A good example would be a DAO (java code on top of database stuff): A JUnit test would pass stuff into the DAO to store some fake data, and then use the DAO to get the data back out. This is definitely not the whole component. And is definitely a java interface.

For component level testing, yes, some form of test harness is typically used. Sometimes junit will do something if the entire component provides a java API to some other component.

[paul wheaton]

Originally posted by Ernest Friedman-Hill:
Another disagreement is "A unit test never uses the filesystem". If this were an article about testing Ruby, maybe I'd buy that. For Java, though, following this would force me into an unreasonable corner, if only because somehow encoding test data into Java code (in one case, finite element mesh data in NetCDF format) would be so difficult it'd be a strong deterrent to testing.

I guess you could argue, then, that those tests should become functional tests. but I think deciding that a test is a functional test because of the form of the test data is artificial and odd.

The file system is a different "unit". It is already tested and verified to be working properly. What needs to be tested is your business logic that uses that unit. The file system would be mocked, and you could verify that your business needs are being properly passed to the file system.

Testing your business logic with the file system would be more than one unit, and therefore, according to the vocabulary I laid out, a functional test.

Further, since a test with the file system would run much slower than one without, thousands of such tests would end up taking much longer than 10 seconds. Therefore, as you build a rich test suite, people might (probably will?) run the tests less often.

[Jeanne Boyarsky]

I think there are subcategories within "functional test" that could use a different name. For example, we have back end functional tests. They test a single DAO/session facade method and its interaction with the database. They aren't unit tests because they access the database. But they need to run fast so they run often. We currently have about 400 tests in 5 minutes.

Then there are the longer functional tests that could take hours to run. We differentiate the two by calling them "integration tests" vs "end to end tests" or "shorter tests" vs "longer tests" depending on the project and exact nature of the tests.

[Jeanne Boyarsky]

We have another type of test that doesn't really fit anywhere in the vocabulary.

These tests read in all the JSPs and check for certain accessibility rules. They run very quickly and run on both Windows/Linux, so we treat them as unit tests. They access the file system so technically they aren't unit tests. But they aren't functional/higher level tests either.

[Lasse Koskela]

Regarding the term "mock object", there's been some push within the community to start using the term "test double" instead which I think communicates this type of usage more clearly.

[Pradeep bhatt]

-file/network I/O or file system;
- another application;
-the console (System.out, System.err, etc.)
- logging

Paul,

I dont understand why Unit tests can't use the above items. Why can't it use the console API or do logging.

[paul wheaton]

Originally posted by Pradip Bhat:


Paul,

I dont understand why Unit tests can't use the above items. Why can't it use the console API or do logging.

Are those things already tested?

Are those things the business logic that you created which need more testing?

You can test your code working with those things - but that would be a functional test.

Besides, if you want to make sure that under the right conditions, information is getting into a log, I/O, the console, etc., the easiest way to do that is mock that entity and examine the mock after exercising your code.

AND .... all of those things are slow! If you have 10,000 tests using file I/O, your tests will be much longer than 10 seconds. But if you mock out all of that I/O, your tests will easily run within 10 seconds.

[manuel aldana]

my thoughts.

unit test: very fine grained test (on isolated class level).

integration test: testing software-artifacts (modules,classes etc.) working together.

functional test: a test very close to a use case of your specification (very often servers's session facade methods).

system test: test of a deployed application (still in your software lab).

acceptance test: test of a deployed application in a real environment (customers who are using software in production, installed on computers software is run from).

JUnit: is a tool which helps you running tests automatically. it is still up to you, what and how fine grained you test. so you can use JUnit for integration,functional or unit testing. i think here is the biggest misconception often taken, where every JUnit test is a unit test.

i think unit-test are allowed to use other apis. a good example is spring framework. of course i use its ioc-container to build up my (isolated) unit test. that's one of this framework's advantages. in your conception a test done with help of spring still couldn't be a unit-test?

[Katie Portwin]

I thought this was really interesting.

I think I'm writing tests at a range of granularities, but it hadn't occurred to me that coarse ones might really be "functional" rather than "unit".

I agree that coarse grained/functional ones are brittle. I would also add that they take more mental energy to write - and to read! I want a simple world where each little method has its little partner testMethod (or two).

I think the fact that it's impossible (ok, not impossible, but ugly) to test Private methods might encourage people to write overly coarse grained tests. What do you think?

I notice that to force in your Mock/Shunt you end up promoting a private method... I've been doing this sort of thing.. but feeling guilty about it..

[Joel McNary]

What is the name of the type of test done by the developer to ensure that his code meets the business requirements? If I'm given a requirement to implement / bug to fix, before I check in the code I run a test to ensure that the bug is fixed -- I start the system (running with all databases/application servers in place) and test the functionality that I have added and/or modified.

I've always called that Unit testing, because I am testing a single unit - the business requirement.

The QA personnel then perform functional testing via their test scripts (which are written from the use cases), and then system testing. Finally, the users perform User Acceptance Testing before the system hits productions. After installation in production, a final test is performed in the production environment (here I've had it called Integration Testing, because the new version of the application has been integrated into a new environment).

Should these be called with different terms, or am I just using an entirely different methodology here?

[Lasse Koskela]

Joel, I've always seen the term "unit test" being used to refer to technical units rather than "functional" units.

Also, the same goes for "integration testing" (as in "integrating technical components").

[Warren Dew]

Originally posted by Ernest Friedman-Hill:

One statement I disagreed with: "Functional test suites are usually run with JUnit." I guess it depends what kind of code we're talking about, but I've never written functional tests with JUnit. Functional tests are application tests, and as such, use the application's interface, not the coder's interface.

I think the confusion arises from Paul's usage of "functional test" being different from yours. To Paul, a "functional test" is like a unit test, but without mock objects behind the unit, so that it may exercise code in other units. To you, a "functional test" is a test of the application, not of the unit - it's a higher level test, perhaps testing something specified in the systems "functional specification".

I think your use of the term is closer to what most people think it means; I think Paul's use is confusing. For the types of testing Paul is talking about, I'd use the following simpler terminology:

Unit - a class or closely related set of classes in a package. What used to be called a "module".

Unit test - a test that tests part or all of the functionality in a unit, but does not exercise code outside of the unit - such outside code is replaced, for testing, by stubs or "mock objects".

Integration test - a test that exercises code in more than one unit, typically by calling into the unit, but allowing the unit to make some or all calls to other units without stubbing or mocking. Such tests test the integration of the unit with other units.
[ March 22, 2006: Message edited by: Warren Dew ]

[Buster Gooding]

I've been confused by differences in terminology at my current workplace. Business analysts use the word "unit-test" to describe any test against a use-case (which I assume is their "unit"). This test may be, but usually isn't, tool-based. But developers tend to mean anything you can run in JUnit...

... including tests on databases. I must admit I think of these as unit-tests. These are the usual CRUD tests, (can you create, read, update and delete an entity?), and are testing one class method. Other database tests might be checking that a method works that lists all outstanding orders, for example, (which in turn tests the underlying database query).

Regardless of what you call these tests (and what *do* people call them?), does anyone have strong views on how to run this sort of test? I tend to use a setup() to insert some test data, then query it, then tear it down again with a delete. Is there a better way?

[Matthew Baranowski]

The file system is a different "unit". It is already tested and verified to be working properly. What needs to be tested is your business logic that uses that unit. The file system would be mocked, and you could verify that your business needs are being properly passed to the file system.

As a rule of thumb unit tests should not violate the integrity of an objects public interface if possible. If the class does not provide public access to the means by which it accesses the file system then the test cannot help but test the filesystem.

public class MyDatabase {
 
public MyDatabase( String fileName ) {
   ... some code that does I/O ...
}
  ... some more code ...
}
 

Paul, your definition would suggest that it is impossible to unit test this class unless you advocate that unit tests cannot violate the integrity of the interface that they are testing.

This issue seems to apply to a number of the other priniciples that you have outlined.

Thanks,

Matt

[Dave Churchville]

Paul,

I agree with most of your definitions, although the level of detail (C.A.T vs. S.A.T, etc.) is beyond what I feel comfortable using in everyday discussions.

For me the 3 interesting levels of testing are:

1. Unit tests - test a single class without depending on the system environment or on correct operation of other classes. The best way to facilitate this is to "inject" dependent classes or service interfaces rather that creating them inside the class under test. You can then pass in mock implementations as your example shows.

Of course, EJBs and DAOs themselves are necessarily environment-dependent, so can't technically be unit-tested. So that brings up...

2. Functional tests - tests that test the operation of a class in a real or simulated target environment and that use the actual code that will be run in production (not mocks or stubs). I see these more as tests of business layer functionality, data access, EJB logic, etc., more so that end-to-end use case testing.

3. Acceptance tests - tests that exercise the system from the perspective of a user of the system (human or computer). Tools like FIT, Selenium, HTTPUnit, etc. might be used here to make them less brittle (at least to underlying code changes). These are also the ones your customers care most about, and should have input into defining (maybe with the help of a QA team).


Finally, one small issue with your code example - I prefer to pass in a business interface or facade to the servlet rather than exposing individual EJB interfaces. This makes for easier testing since I don't then need to support dozens of mock EJBs.

--Dave

David Churchville
ExtremePlanner Software
http://www.extremeplanner.com

[Ron Horton]

I totally disagree about the unit test which you have mentioned.According to me daily reading of news paper and editorials will help a lot to improve over vocabulary and its the way of testing ourself whether we are perfect in our vocabulary or not.
[ January 03, 2008: Message edited by: Lasse Koskela ]

[Faisal Fuad]

hehehehehe....although this post is very old but would love to say that "Ron Horton" <- You are the Man !!!

My mood is now so good after reading this post.....you guys are really Super ;)

Many many thanks...

[Brian Burress]

This is a pretty old thread but as I happend to see it while heading down the rabbit hole, I figured I'd drop in my two cents.

1) Whether you agree with the documented specifics of the vocabulary or not, agree that it is important for you and your organization to agree and understand what you mean when you use certain words and phrases. It is an excellent point that to someone in a testing/quality assurance role that a "unit" test may correspond to a requirement which could translate into a component or system test to a developer. While the word used may be confusing, stating boundaries of what is involved should help get folks on the same page.

2) I do disagree on the statements regarding where JUnuit can and can not be used. To me that is up to the organization and what they want to accomplish. I was involved with a project which went light on the unit tests (as defined in the doc) and focused on more functional/integration/component testing. We did have minimal unit tests as defined, but we fell into the "rut" where a majority of our issues were with how we were integrated with the backend application (use of the APIs, how data was passed, etc) such that we had developer coded "atomic" integration tests meant to focus on fine grained actions so that we could identify a problem point (and all tests did not fail as a result of an issue) coupled with more course grained "acceptance" tests also written in JUnit but which focused on a broader set of the data and in some cases involved confirming that the orchestration of various API calls to the back end behaved as expected. These acceptance tests were developed with specific requirements in mind, general system use cases, etc. Per our set up, the intent was that we would see 100% passage of acceptance tests to release a build and that we were able to document failures of the other tests as "known issues".

3) The mock/shunt boils down to how the code is designed and implemented. If approaching a solution using OO principles and you tend toward maximizing re-use and keeping the pieces of your solution simple, I think you would tend to implement the solution such that you need minimal "special" code to test. While there will be exceptions to every rule, use of helper classes, utility method, etc which can be exposed and then integrated into the overall solution maximizes what can truly be "unit" tested as defined here and result in an overall stronger solution.

[Tharpa Roberts]

Stress testing - the definition you gave is not what I understand by stress testing. Stress testing, as I understand it, has to do with quantity of demands. That is, you "stress" the system. Suppose, for example, you have a program that processes files in a folder. For many kinds of tests, you would simply process one file placed in the folder at a time. Suppose, though, that in normal use, you can imagine that up to 100 files would be placed in the folder. So for stress testing, you would place 1000 files in the folder, far more than you would expect in real use.