Shipping Tests

[Mike Clark]

What do y'all think of shipping tests with commercial software?

I was lucky enough to work in a small company where developers were also tech support. Often the support calls were recurring problems with installation or deployment issues. Imagine a user inadvertently changing permissions on a directory and the application acting strangely. Sometimes troubleshooting that took a while on the phone. The application was eventually made more robust to handle these situations, or at least report the problem more clearly, but in the meantime it would have been valuable to drop a simple JUnit test suite into the installation. That test suite would include tests like checking whether certain directories had the correct permissions, checking that the version of the JRE was supported, etc. The suite could be grown to include common support problems or issues you might find in an FAQ. Think of the test suite as being like sending a tech support person to the field, except that nobody has to leave the office. I offer examples of how to write and distribute these diagnostic tests in the book.

The installation diagnostic tests can be optionally deployed when a customer experiences a problem relating to the environment in which the software is being used. Additionally, unit tests can be shipped with the product as a way of advertising quality and pride of ownership. In some circumstances the unit tests may even fail, though probably less often than the installation tests. Realizing that the number of tests is no indication of the quality of tests, they unit tests still might be valuable.

Yet although many open source projects ship unit tests, I've yet to run across a commercial product that ships unit or installation diagnostic tests. I'd love to hear of projects that do ship tests, and what others might think of shipping and receiving tests with commercial software.

Mike

[Ernest Friedman-Hill]

We "ship" the application tests for one internal software product. Although the users rarely use them, it's very convenient when one of us is called to a user's desk to diagnose a problem. If they've got the app, they've got the application tests, and these can often find installation and configuration problems for us.

[Mike Clark]

Originally posted by Ernest Friedman-Hill:
We "ship" the application tests for one internal software product. Although the users rarely use them, it's very convenient when one of us is called to a user's desk to diagnose a problem. If they've got the app, they've got the application tests, and these can often find installation and configuration problems for us.

Interesting. Would you say that "shipping" the tests has been valuable? In general, what kinds of problems have you seen the tests find?

Thanks!

Mike

[Lasse Koskela]

Many if not most laptop vendors are shipping some kind of diagnostics software along with their configurations with the intent of avoiding unnecessary FedEx'ing due to configuration problems that could've been solved over email or phone.

[Mike Clark]

Ah, that's a great observation, Lasse. I hadn't thought of that when I started thinking about shipping tests. On a similar note, the getting started instructions for my new printer includes a step that prints out a diagnostics report.

So, in general, we're accustomed to having self tests included with hardware. But, from my perspective, the practice of shipping self tests has yet to permeate software deployment and installation. That is, when a software vendor gives me their latest and greatest JAR file, for example, I don't expect it will contain tests I can run before integrating it into my software system.

Would folks here find value in having self tests (unit and installation diagnostic tests) distributed with vendor products? How would you use each type of test?

Thanks!

Mike

[Bear Bibeault]

Would folks here find value in having self tests

Since the topic of this forum is build tools, I'm assuming the 'consumer' in this case is usally a tech-savvy developer who can make heads or tails out of test results. In this case, I'd love to have tests that could verify the installation, configuration and execution of a product. (Just look at the JSP, Servlet and Tomcat forums to see how many people have trouble setting up even a fairly simple tool like Tomcat).

For the general public, I'm not so sure that consumer-run testing is feasible. Now, having the tests handy on the remote system upon which the software is installed so that a help desk tech can have the end consumer run the tests, is a different story.

[Ernest Friedman-Hill]

Originally posted by Mike Clark:


Interesting. Would you say that "shipping" the tests has been valuable? In general, what kinds of problems have you seen the tests find?

This particular application interacts with a lot of other software -- it can run various things with Runtime.exec() for example. The application tests provide a nice report that the application itself doesn't provide as coherently: "You don't have application XYZ installed, you have an unsupported version of application PDQ, and although I searched for and found application ABC, it's not on your PATH, so the software can't run it." It's a lot easier to make these distinctions from the tests because they're mostly shell scripts and can do things that aren't so easy to do in Java (like looking at environment variables.)

[Lasse Koskela]

Originally posted by Bear Bibeault:
For the general public, I'm not so sure that consumer-run testing is feasible. Now, having the tests handy on the remote system upon which the software is installed so that a help desk tech can have the end consumer run the tests, is a different story.

May I ask why or in what scenarios you think consumer-run tests aren't feasible? Certainly not all products are suitable for such self-diagnostics but there's so many examples of working -- and useful -- applications of the concept that I'm having hard time seeing your point.

[Bear Bibeault]

May I ask why or in what scenarios you think consumer-run tests aren't feasible?

I guess I'm think of people like those in my neighborhood (for whom, of course, I have ended up being tech support). They have a hard enough time figuring out how to run simple programs. Running and interpreting test results is far beyond their ken. But as I said, having the tests available so that they could run them under the direction of help desk personnel (or tech-savvy neighbors) who would do the result interpretation would be very handy.
[ September 22, 2004: Message edited by: Bear Bibeault ]

[Lasse Koskela]

Originally posted by Bear Bibeault:
They have a hard enough time figuring out how to run simple programs. Running and interpreting test results is far beyond their ken. But as I said, having the tests available so that they could run them under the direction of help desk personnel (or tech-savvy neighbors) who would do the result interpretation would be very handy.

That's actually what I've been thinking all along -- instead of trying to squeeze out meaningful replies from the consumer in trouble, the help desk can say "double-click that icon to run some diagnostics and read the results out loud for me".

[Mike Clark]

Originally posted by Lasse Koskela:
That's actually what I've been thinking all along -- instead of trying to squeeze out meaningful replies from the consumer in trouble, the help desk can say "double-click that icon to run some diagnostics and read the results out loud for me".

Amen. As you probably know, in the book I talk about writing and fielding installation diagnostic tests using JUnit as the example. It's fairly trivial to do, but simple tests that flush out common problems can really pay off, and yet it's not often done.

I think if the tests that fail report what they expected in some human-readable format with a recommendation for what might be the problem, that technical users might be able to resolve the problem on their own.

That doesn't necessary mean that the tests are in JUnit, or if they are that the JUnit results should be displayed directly. For consumer users, the tests themselves could be run by some process that interprets the results and then presents the problem/solution in an intelligible fashion.

Mike
[ September 22, 2004: Message edited by: Mike Clark ]

[Mike Clark]

Originally posted by Ernest Friedman-Hill:
It's a lot easier to make these distinctions from the tests because they're mostly shell scripts and can do things that aren't so easy to do in Java (like looking at environment variables.)

Ah, that makes sense. Scripting languages in general are well suited for this kind of work.

For Java applications that are deployed on disparate platforms, it's difficult to predict what tools would be available. Moreover, installing another tool (scripting language) for the purpose of running diagnostics can be problematic. This might be an area where Groovy could shine in that it can be relatively easy to install alongside a Java runtime environment.

Thanks for tickling my brain.

Mike

[Jeanne Boyarsky]

This thread was a really interesting read.

I too like the idea of shipping tests with code. Parasoft doesn't include tests, but they have some features that are useful for tech support. It zips up the configuration files which you send to tech support. What would be cool would be an ant script that sent the files (or test results) along when you clicked a button!

[Mike Clark]

Originally posted by Jeanne Boyarsky:
What would be cool would be an ant script that sent the files (or test results) along when you clicked a button!

In the book, I call that a "crash report". It collects relevant evidence, bundles it up, and sends it over the web after you get a chance to look at what it will send. Some folks have gone so far as to automatically take the information from the crash report over the web and enter it into their issue-tracking system.

Mike

[Craig Demyanovich]

Mike,

Have you shipped any tests with any of your software, like JDepend? If so, has anyone communicated using them to solve a problem with installation/configuration? Or, have you shipped tests only on projects on which you've consulted?

Thanks,
Craig

[Mike Clark]

Originally posted by Craig Demyanovich:
Have you shipped any tests with any of your software, like JDepend? If so, has anyone communicated using them to solve a problem with installation/configuration? Or, have you shipped tests only on projects on which you've consulted?

I ship unit tests with both JDepend and JUnitPerf. In the case of JUnitPerf, I've had a couple of users email with failed test reports. This is usually due to timing issues or peculiarities with versions of Java. Off the top of my head, I don't recall any installation-related test failures for JDepend, but I'll have to check on that.

By virtue of shipping tests with both of these open source tools, I've had users submit bug reports, proposed patches, and enhancement requests that included unit tests. I suspect because the tests were readily available these end-users chose to use tests as a way to 1) highlight a corner case that illuminated a bug, 2) communicate that their patch worked and didn't break existing functionality, and 3) to demonstrate how an enhancement would work if the software were changed to make the test pass. Each time I've received tests like this, I've found it to be compelling and quite helpful.

As another data point, while maintaining the JUnit FAQ I've seen recurring installation problems related to the onerous classpath. A while back I wrote a Java class called WhichJUnit that asserts that certain critical classes can be found on the classpath. Although I can't quantify how effective that has been, we seem to see less classpath-related questions on the JUnit mailing list. WhichJUnit is available at

http://junit.sourceforge.net/doc/faq/faq.htm#running_2

Apologies for the long answer to a short question. I'm using this to work through some ideas on shipping tests, and your questions are helping! :-)

Mike