Testing Framework

[Andrew Och]

Dear Cedric and Hani

I have been working as a software developer in China for just under 5 years in two separate and large international companies. Both companies are here to get the "China Price" for research and development and both appear to have very similar problems, i.e. quality.

Both companies see testing, as the answer to their quality problems, but theory and execution seem be miles apart.

Problem 1, Testing is boring. It must be because in my previous company and current, if you ask a tester if they like their job, they say "no, it's boring" and they would rather be developers. In fact my previous company would do a written exam during interview day and those with high scores became developers, those with low scores became testers. In my current company I asked where the tests are located on our Clearcase server and was told there aren't any. Development time pressure meant there was no time to write any. The project I joined is 7 years old :-)

So will this book make testing sound sexy?

While this question may sound flippant, it is not, until testing is seen as sexy and cool and fun (like software development), it will not attract talented people. Since you are both talented, what attracted you to testing?

Problem 2, Confusing plethora of testing extension. I used HttpUnit for testing an Oracle ADF application (now Apache Trinidad) and it failed to deal with JavaScript. HtmlUnit saved me, as did Javaranch :-). It was also a J2EE application, but our team was never given enough time to setup Cactus to test it properly. We had JMeter setup a little to test performance. Cynically I feel that our testing, test plans and test reports were contained in our project to ensure we passed our CMMI assessment.

Does this new framework provide one type of interface for all of this testing (the xml file), thus reducing test extension setup time?

I really hope so.

Andrew

[Hani Suleiman]

Hi Andrew,

Great questions! My personal experience actually is very similar to yours. Developers do find testing boring. Despite always being told that testing is sexy, I think it is a reality of real life projects that testing (unless accounted for up front) is something that always ends up falling by the wayside.

There are a couple of ways to combat this. One of the most important is to not think of testing as separate from development. It's just another tool to use *during* development. People dont complain that debuggers are boring to use, do they? Writing tests should feel the same.

So it is a matter of slowly adopting good testing habits. For example, dont enforce a testing period. Instead, when your QA guys (or users) give you a bug report, then and only then create a test. So start by enforcing tests as a method of capturing bug reports.

You'll probably first end up with a functional test, which is OK. As you debug it, you'll probably find that its taking too long to get to the bit where things are going wrong (the actual bug). So from this, you can naturally evolve a unit test. Before you know it, you'll have a suite of functional and unit tests that are driven by users, not by developers deciding what they should test (and usually getting it wrong!) We discuss this approach in the book by walking through a typical 'enterprise' type application, showing how despite not designing for testing or factoring it in up front, we can slowly introduce it into a project.

For the second issue, I think what would help you tremendously is test groups. Thanks to TestNG's runtime vs compile time separation, you can decide what set of tests you will run by modifying your xml file. Another nice trick is to have multiple runtime configuration files, and use the one that's most appropriate. The way this works is that every test is 'tagged' with what groups it belongs to. The xml file then specifies what groups to run. So for your example, you could have 'server', 'cactus', 'performance' as groups, and can then decide for example to only run server + cactus tests.

A side note to using Cactus, I'd strongly recommend (at least going forward) you try to design your components to be testable outside of a container. Cactus is not the most friendly tool to use, and the overhead of having to go to a server to run tests will discourage developers from writing tests or running them, thus defeating the purpose entirely! The book does discuss in-container vs out of container testing, along with recommending and illustrating tools to use for both (for example, htmlunit).

[Cedric Beust]

Hi Andrew,

Originally posted by Andrew Och:
Dear Cedric and Hani

I have been working as a software developer in China for just under 5 years in two separate and large international companies. Both companies are here to get the "China Price" for research and development and both appear to have very similar problems, i.e. quality.

Both companies see testing, as the answer to their quality problems, but theory and execution seem be miles apart.

Problem 1, Testing is boring. It must be because in my previous company and current, if you ask a tester if they like their job, they say "no, it's boring" and they would rather be developers. In fact my previous company would do a written exam during interview day and those with high scores became developers, those with low scores became testers. In my current company I asked where the tests are located on our Clearcase server and was told there aren't any. Development time pressure meant there was no time to write any. The project I joined is 7 years old :-)

So will this book make testing sound sexy?

You are touching a topic that is dear to my heart :-)

Whenever I talk at a conference on this topic, my first few slides claim exactly this: that testing is sexy again. Based on your reaction, it looks I might be overstating the case somewhat, but I don't think it's wrong to be overly enthusiastic about it. After all, the more people say that testing is cool, the more people will eventually take it seriously...

I do believe that silly ideas such as "real developers don't test" and "good developers write code, bad developers write tests" are finally coming to an end. We are seeing more and more material (books, mailing-lists, frameworks, forums) which hint that testing should be part of any serious developer's daily tasks.

This material is also making it clear that there are a lot of problems to solve in the testing area to the extent that sometimes, it's harder to write the tests than it is to write the business code, a challenge that should appeal to any developer who likes to solve hard problems.

I think you will find that the book describes techniques and concepts that should make test skeptics pause and realize "yes, that's a pretty nice piece of code right there", so from that perspective, I do believe that our book makes testing sexy.

Now, I'm curious about the case where you are working with an entire team of people who just won't touch testing. I'd like to ask them what makes them interested in writing code in the first place. Are they really passionate about coding or is it just a nine-to-five job? Why do they think writing tests is boring? Maybe they've only had to test boring code, how about you challenge them to test a piece of code that has deep dependencies into legacy code? (okay, maybe it's not the sexiest challenge). How about exposing them to data-driven testing? Or challenging their code they and tell them "prove me that I can invoke this code that you wrote from multiple threads and nothing bad will happen".

Having said that, Hani and I are pragmatic and we know that deadlines sometimes put pressure on a development team. You won't hear us say things such as "people who don't write tests should be fired" or that "there is no excuse not to write tests" (statements we've heard quite a few times from test extremists). I believe that there are times where a new functionality should take precedence over a new test, but it's all about balance and judgment: testing should be taken into account in the early phases of the project and possibly be put on the backburner when the deadline approaches.

Please take a look at the table of contents that I posted in the welcome thread and let us know if there is anything in there that piques your curiosity...

While this question may sound flippant, it is not, until testing is seen as sexy and cool and fun (like software development), it will not attract talented people. Since you are both talented, what attracted you to testing?

Problem 2, Confusing plethora of testing extension. I used HttpUnit for testing an Oracle ADF application (now Apache Trinidad) and it failed to deal with JavaScript. HtmlUnit saved me, as did Javaranch :-). It was also a J2EE application, but our team was never given enough time to setup Cactus to test it properly. We had JMeter setup a little to test performance. Cynically I feel that our testing, test plans and test reports were contained in our project to ensure we passed our CMMI assessment.

Does this new framework provide one type of interface for all of this testing (the xml file), thus reducing test extension setup time?

I really hope so.

Andrew

For your information, TestNG is almost four years old, so it's not exactly "new" :-)

There are several reasons for the existence of the XML file (which I'll describe shortly in the next paragraph), but I don't think any of these address your question, which I'm not sure I understand. Would you mind rephrasing it?

I think it's important to have an external file describing your tests for a couple of reasons:
- So you can share it with coworkers easily, or save several files describing different test configurations (testng-database.xml, testng-frontend.xml, testng-servlets.xml, etc...)
- So you can rerun different set of tests (using TestNG's "test groups") without having to recompile anything

Please let me know if you'd like me to expand on either of these points.

--
Cedric

[edited to fix quotes tags]
[ December 17, 2007: Message edited by: Jeanne Boyarsky ]

[Cedric Beust]

Okay, I messed up the QB and BLOCKQUOTE tags in my previous post... Make sure you read the entire article since my response is hidden in there :-)

--
Cedric

[Jeanne Boyarsky]

Originally posted by Andrew Och:
Testing is boring. It must be because in my previous company and current, if you ask a tester if they like their job, they say "no, it's boring" and they would rather be developers. In fact my previous company would do a written exam during interview day and those with high scores became developers, those with low scores became testers.

Are you asking about development level testing or Quality Assurance level testing? There is a difference between making sure your own code (or that of your team) works and making sure that another team's code works. While I enjoy testing, I don't think I would like it as much if it was my whole job. At the same time, I know people who feel the opposite way (that work in Quality Assurance.)

Originally posted by Cedric Beust:
Okay, I messed up the QB and BLOCKQUOTE tags in my previous post... Make sure you read the entire article since my response is hidden in there :-)

Fixed to make it easier for me to read your response

[Andrew Och]

Dear Hani, Cedric and Jeanne

First of all thank you very much response to my questions, I didn't expect it :-)

To Jeanne, my questions are all framed at developer level testing. I am very cynical when it comes to Quality Assurance level testing, mainly because of my experience, the Quality Assurance Managers' bonuses were based on getting a higher CMMI level during assessment and so for them quality means ticking boxes on checklists. To be fair R&D centers in China are full of fresh inexperienced graduates and a high CMMI rating implies that they are mature enough to get the job done.

To Hani, I really struggled with Cactus and turned to HtmlUnit, but that meant that the test cases generated were leaning towards functional tests rather then unit tests. At the moment I am working on a lot of legacy code (mixture of Korn Shell scripts and Perl). We are now creating a lot of functional tests, just as you say. Our users/customers give us bug reports :-) and our new policy is to create an automated test case first using Perl's Test::Unit, then fixing the bug or developing the enhancement. So I feel we are definitely moving in the right direction.

To Cedric, I worked for Siemens in Nanjing and now work for HP in Shanghai. As I say my colleagues are generally fresh graduates and I believe they are genuinely passionate about coding. You have however hit the nail on the head, we are testing legacy, boring code. It is the nature of the business I believe, to reduce development costs by moving support and maintenance of older code/products to low cost countries.

My second question was phrased badly. Let me put it into context, testing and process. We write test plans, then test cases, then execute the test case and write test reports. There is a lot of duplication of effort, e.g. the textual description of a test case goes into the test plan, test case and test report (sure you can cut and paste). However having a centralized xml file as a source manuscript for all would be ideal. One that works for TestNG and also perhaps Apache Continuum and is the bases of my test plan and is the descriptive input to my test report, output as a pdf. Thus the bureaucratic aspect of testing is covered by one or two xml files and a set of tools. A testing framework should offer that shouldn't it?

Although I have to admit my original question was hinting towards the fact that this is another xml/configuration file, on top of many others and may pull people away from using conventions, defining convention busting test cases simply because they can using. I guess I am still a little skeptical of the xml panacea ;-)

Finally I do feel reinvigorated about testing and will during my lunch breaks talk up testing as the new cool thing and those who don't test "just don't get it".

Once again thank you very much for your answers, they are very much appreciated by me.

Andrew

[Nathan Syngrem]

Jeanne Boyarsky wrote:

Originally posted by Andrew Och:
Testing is boring. It must be because in my previous company and current, if you ask a tester if they like their job, they say "no, it's boring" and they would rather be developers. In fact my previous company would do a written exam during interview day and those with high scores became developers, those with low scores became testers.

Are you asking about development level testing or Quality Assurance level testing? There is a difference between making sure your own code (or that of your team) works and making sure that another team's code works. While I enjoy testing, I don't think I would like it as much if it was my whole job. At the same time, I know people who feel the opposite way (that work in Quality Assurance.)

I don't agree with you. If you have proper knowledge about testing then you will surely like it. If you talking about development the working hours required for that are lot more plus, many a times you have many people working on a single module, so this can be very hectic as you need to co-ordinate with all and a mistake by a single person can lead to huge problems.

[edited to add missing close quote so it is obvious what I had said]

[Jeanne Boyarsky]

Nathan,
This is a bit philosophical, but I disagree that everything one knows about, one likes. I have knowledge of how to clean my house. Does it follow that I would enjoy doing so as a full time job?

And I do like testing as part of my job.

[Jeanne Boyarsky]

ps - Welcome to JavaRanch!