Who does the unit testing? Who SHOULD do it?

[Esther Schindler]

I'm working on an article for Software Test & Performance magazine (stpmag.com), and I'm interested in getting your input.

Some stages of quality testing definitely belong in the QA Department's hands: the professional testers know how to read (and write!) functional descriptions, they know how to use the fancy testing tools, and they know how to exercise the features of an application.

Some stages, on the other hand, aren't quite as clear. One of them is unit testing -- the step in which the developer makes sure that each "chunk" of code works as designed. (I wrote about "how big is a unit?" in the September 2005 issue; you can download the PDF -- of the entire issue, be warned! -- at http://stpmag.com/issues/stp-2005-09.pdf).

In your shop, who does the unit testing? Is that the responsibility of the individual developer? Or is QA responsible for that part of the testing cycle?

Whichever answer you choose: is that the way you prefer it to be? Why? If the answer is "It depends," what does it depend upon? Do software methodologies change the answer?

Please let me know how I can refer to you in the article. If necessary, send me a private e-mail message ([e-mail]) to let me know your real name, company affiliation, and geography.

Esther Schindler
Contributing editor, Software Test & Performance magazine

[Ernest Friedman-Hill]

My main project -- small at about six developers -- practices TDD, "Test Driven Development." The developers do all the unit testing, and the tests are written before the code. I think you'll find a number of people in this forum doing the same thing.

I work for Sandia National Laboratories in Livermore, California, USA.

[Esther Schindler]

Originally posted by Ernest Friedman-Hill:
My main project -- small at about six developers -- practices TDD, "Test Driven Development."

Have you ever done it another way? In what ways was it different? (The unit testing, I mean. TDD may change EVERYthing, so this may be a dumb question.)

[Gerardo Tasistro]

Esther I believe the units should be tested by the developer.

a) Because it reduces the amount of problems QA will have to deal
b) Benefits code reusability <-- I think this is the most important reason you don't want to reuse bad code.

[Jeanne Boyarsky]

Originally posted by Ernest Friedman-Hill:
I think you'll find a number of people in this forum doing the same thing.

We have the same developer write the code and the tests, but don't necessarily write the tests before the code.

We started out with nobody testing the units. Developers informally tested their chunks in any way they saw fit. Since then, we've move to writing unit tests. This works much better and none of us would ever want to go back.

We often have to sell this concept to new team members. I use a quote from the Pragmatic Programmers book when people argue that it isn't their job to test. The Pragmatic Programmers point out that it is your job to create working code. The developer written unit tests just help out with that.

[Ernest Friedman-Hill]

Originally posted by Esther Schindler:

Have you ever done it another way? In what ways was it different? (The unit testing, I mean. TDD may change EVERYthing, so this may be a dumb question.)

Doing it first does change everything. I've run plenty of projects in the past where developers wrote tests after the fact. These invariably were more application tests than unit tests, because if you don't design for testability, you often end up with code that's hard to test "in the small". You get big tests with lots of set-up that don't isolate faults and tend to give worse test coverage.

Writing tests first leads to better designs because it forces you to think about an interface for each class which allows that class to stand and be tested in isolation.

[Ilja Preuss]

Hi Esther!

Originally posted by Esther Schindler:
Some stages of quality testing definitely belong in the QA Department's hands: the professional testers know how to read (and write!) functional descriptions, they know how to use the fancy testing tools, and they know how to exercise the features of an application.

Sorry, I don't agree that this is clear!

Two reasons:

- Having a separate "quality testing stage" defers feedback; the earlier you get feedback, the better you can work with it.

- Having a QA department that separates testers from the rest of the development team hampers communication; the testers can't contribute as well to the project as if they were a colocated part of the team.

See the second part of http://www.xprogramming.com/xpmag/are_we_doing_xp.htm and the discussion started by http://groups.yahoo.com/group/agile-testing/message/6663

In your shop, who does the unit testing? Is that the responsibility of the individual developer? Or is QA responsible for that part of the testing cycle?

In our shop, it's the responsibility of the developers.

Whichever answer you choose: is that the way you prefer it to be? Why?

Yes, I prefer it that way. When I'm writing code, I need to know wether it works - as Jeanne pointed out, that's my job. And not only the code I just wrote, I also need to know wether I broke some existing code. So I need to have new tests for my new code as early as possible, and I need to be able to run all the tests whenever I want. (In fact there is an Eclipse plugin that runs all the tests of a project automatically as part of the compile cycle that I like to use.) So early feedback is part of the why.

Another important reason is that writing the tests helps me writing well designed code. I am too practicing TDD, and writing the unit test first *forces* me to write testable code. And the most important property of well testable code is that it is decoupled, so that you can test units in isolation. Writing the tests in parallel to the code also helps with getting a high code coverage.

Do software methodologies change the answer?

I think I'd want to use TDD in every project, independently of the methodology used.

Please let me know how I can refer to you in the article.

Ilja Preu�, Software Developer at disy Informationssysteme GmbH, Germany

[Al Caplan]

Esther
In my shop developers are 100% responsible for writing unit tests, as well as fixing them on the occassion they fail. There are many pro's I've seen for this standpoint

1) Unit testing is faster than integration testing. Imagine that you are working on a web application. To test a new page, or a bug fix, to test a fix you need to write the fix, compile/deploy the fix, start up your app server, navigate to the page, and perform a set of actions. Maybe your fix worked, maybe it didn't. I've watched as developers go through this cycle ad nauseum. (yes, I know there are some environments like MyEclipse that take care of the compile/deploy, you still have to do the login/navidation etc). With Unit Testing, you define what the correct behaviour is. You can then write your code, and if you're using an editor with JUnit integrated, you just keep clicking that run button until you see a green bar. Only then are you usually ready to deploy to the server and attempt an integration test.

2) If you leave it to QA to write the test, what if it fails? Who is responsible for a failure, when the developer could have moved on, and is not looking at the previous issue?

3) Unit Tests make for an additional level of documentation. Sometimes there are questions on how to use an API, JUnit tests are easily provided as working examples.

[Scott Selikoff]

Personally I think Unit Testing is more about PR and accountability from a management perspective than it is about quality, control, or good coding. I've seen managers talk about it as a buzz word in the same way they talk about J2EE. It reminds me of a Dilbert cartoon at best. Most of the time, they have no notion of what unit testing is nor can they properly formulate the desires to the developer as far was what to test and how.

Before I give an example, I will say the benefits are that it makes sloppy developers think first before they code. That being said, I think real testing should be done entirely by a QA department and that test-driven development is fundamentally flawed. I especially say it is problamatic in J2EE where service context, database dependency, and transactional requirements make it unrealistic to run good unit tests. For example, if a line of code was only supposed to be run inside a transaction, it is very difficult to successfully unit test.

But skipping all that (I could go on for hours), here's my 'real-life' example...

Super High level management came in and demanded unit testing on all code for projects that were months behind schedule as a way to make customers feel safe. Middle management says "sure" and puts the burden on the developers to do it with the obvious condition "it wont add any new time to an all ready delayed project". On top of that, in many cases true unit testing is worthless, for example in a database-driven environment you cannot run unit tests since the majority of your data cannot be contained to single method calls. In this situation, you really want integration system tests.

In my scenario it took months to decide the difference between unit, system, integrating, service, and UI testing, at which point when middle management finally figured it out said "ok, well why don't you the develop just do a little bit of ALL of those tests and still finish on time".

I think the people who thought of test-driven development were smart and had excellent intentions, I see the practical application of it being a complete mess and giving management a false sense of security at least in the near future.

[Scott Selikoff]

Originally posted by Al Caplan:

2) If you leave it to QA to write the test, what if it fails? Who is responsible for a failure, when the developer could have moved on, and is not looking at the previous issue?

While I may dislike test-driven development I fully support Agile development, which is where you attempt to do vertical requirements/design/code/testing all at once. In an Agile environment, the designated QA tester would be testing pieces of the developers code as soon as they were ready.

Ultimately its QA's job to protect the client, not the developer's. Oh, "and if the developer consistently writes bad code, they should be promoted which would solve the problem since only good developers would be writing good code." Often when a manager notices a particular developer writes really good code, they don't them to stop writing good code, so they'll keep them writing code as long as possible.
[ November 08, 2005: Message edited by: Scott Selikoff ]

[Ilja Preuss]

Originally posted by Scott Selikoff:
While I may dislike test-driven development

Interesting - what do you dislike about it?

In an Agile environment, the designated QA tester would be testing pieces of the developers code as soon as they were ready.

In fact, in many Agile teams today, the tester writes the automated test even before the developers start implementing the functionality.

Ultimately its QA's job to protect the client, not the developer's.

I see that differently. QA's job is to help the whole team (including the developers and the customer) to get better feedback about how close the developed system is to what the customer actually requested. In my eyes, that is more of a win-win situation than being about protecting one side from the other.

[Madhuri Chanda]

Hi Esther
I work as a developer in a team of 8 members among which none are testers. We do all the initial phase of testing .

But we do not follow TDD.. we do all the testing after development. Our project mainly requires to eliminate the bad data at the user entry level itself. So even writing the test cases has to be done after the development stage.
According to me, the developer knows better what part of the code might break and so tests with that kind of direction in mind. But most of the errors will be eliminated with the completion of the development itself. How can we say our development work is finished when it has flaws and our code is breaking something else. By the time, Unit Testing stage has reached, good coding should not produce any functionality and major errors.

I do not say there is no point in testing by QA.. after we(developers) test from our perspective, I think QA can test better from the user perspective of formatting, spelling, layout requirements etc..
--------------------------
working as a developer at IDPH, IL.

[Jeanne Boyarsky]

Originally posted by Ilja Preuss:
I see that differently. QA's job is to help the whole team (including the developers and the customer) to get better feedback about how close the developed system is to what the customer actually requested.

Ilja,
I think that is a nice goal for QA and would love to see it in practice. As I see it, QA does tend to represent the client more than anyone else. It's harder for developers to say the application isn't ready. QA tends to be more of a neutral third party. Also, QA is less attached to the code and can view testing as more of trying to break the code than developers can. (statistically - not for all developers)

Madhuri,
It sounds like it is back to the question of what "unit testing" means.

"By the time, Unit Testing stage has reached, good coding should not produce any functionality and major errors" - This implies that you are doing some pre-unit testing. It may not be formal testing, but it is some type of testing.

[Ilja Preuss]

Originally posted by Jeanne Boyarsky:
As I see it, QA does tend to represent the client more than anyone else.

Well, I don't have any problems with testers *representing* the customer. But shouldn't it also be the developers *wish* to make the customer happy? So isn't helping to make sure that the customer gets what he is expecting also *helping* the developers?

It's harder for developers to say the application isn't ready. QA tends to be more of a neutral third party. Also, QA is less attached to the code and can view testing as more of trying to break the code than developers can.

I agree. Still I would see that as a service to the whole team. "Protecting the customer" sounds like the developers actually would prefer to ship him crap, which isn't true in my experience.

[Al Caplan]

Originally posted by Scott Selikoff:


Ultimately its QA's job to protect the client, not the developer's

Is it? QA has their test cases to run. Where do these test cases come from? If they cover 100% of all test cases that all customers will use then that is a fair statement, however it's been my experience rather that it's QA's job to protect the product via the test cases they have been given, and not necassarily the customer.

[Jeanne Boyarsky]

But shouldn't it also be the developers *wish* to make the customer happy?

Definitely. It's also the developers wish to make clean maintainable code that implements the requirements by a certain date. Meanwhile, QA can focus purely on quality. I guess what I'm saying is that QA has a more specific focus on quality, not that developers don't want to make the customer happy.

developers actually would prefer to ship him crap, which isn't true in my experience.

Ship - no. Put into the early server environments - yes. In some places there is a strong emphasis on being "done" by the defined date. We've gotten into heated debates about what "done" means.

[Ilja Preuss]

Originally posted by Jeanne Boyarsky:
It's also the developers wish to make clean maintainable code that implements the requirements by a certain date. Meanwhile, QA can focus purely on quality. I guess what I'm saying is that QA has a more specific focus on quality, not that developers don't want to make the customer happy.

If by "more specific focus" we mean that they mainly work on it because of their skills, I agree. I wouldn't want them to be ignorant of internal code quality or the deadline, thought. I think that won't lead to a better product, but to stress in the team, and therefore an overall *less* effective team. The developers need feedback on how well they are doing, not all kinds of contradicting messages from the outside about what they need to concentrate on (such as "the date" vs. "quality").

Ship - no. Put into the early server environments - yes. In some places there is a strong emphasis on being "done" by the defined date. We've gotten into heated debates about what "done" means.

In my experience, this emphasis is typically not inherent to the needs of the developers, but tought by management. The developers simply have learned by experience that making the deadline is the most important thing and that compromising quality is acceptable in the eyes of the typical project manager, but compromising the date or functionality leads to suffering.

I think that QA testers actually can help getting project managers to accept reality and find a better definition for "done", thereby giving back some freedom to the developers to really produce a quality product. I don't know any developer worth his salt who wouldn't like to do that, deep in his heart.

[Jeanne Boyarsky]

Originally posted by Ilja Preuss:
In my experience, this emphasis is typically not inherent to the needs of the developers, but tought by management.

Agreed!

[Ilja Preuss]

Originally posted by Jeanne Boyarsky:

Agreed!

I hate it when this happens - it means that the discussion is over...

[Esther Schindler]

I use a quote from the Pragmatic Programmers book when people argue that it isn't their job to test. The Pragmatic Programmers point out that it is your job to create working code. The developer written unit tests just help out with that.

I like that. It keeps everybody's eyes on the goal.

--E

P.S. I apologize for posting-and-running. I spent part of last week at the DevConnections conference in Las Vegas. Not gambling, I assure you; I kept myself busy by writing about C++ (and whether Microsoft is trying to take it over).

[Esther Schindler]

Originally posted by Ilja Preuss:
I think that QA testers actually can help getting project managers to accept reality and find a better definition for "done", thereby giving back some freedom to the developers to really produce a quality product. I don't know any developer worth his salt who wouldn't like to do that, deep in his heart.

I agree that this is a shared goal... actually most managers would like to have a picture of reality that they can believe in.

Some developers are better at it than are others. What do you think those developers do right? (Which takes us far off the topic of unit testing, but I can't help myself.)

[Scott Selikoff]

Originally posted by Ilja Preuss:
Interesting - what do you dislike about it?.

The immediate implementation, not the theory. I see people rushing to it as if it will allow projects to finish bug free without really understanding it. Also, I find the notion of unit testing in J2EE perposteruous.

Originally posted by Ilja Preuss:
In fact, in many Agile teams today, the tester writes the automated test even before the developers start implementing the functionality.

This I like, alot. I'm in favor of test driven development in such that the testers get involved with testing more and more. I don't see QA testers having the knowledge to write low level unit tests. Most QA testers I've met couldn't read code at all, but knew how to write good test cases so in practice I'm not sure how often this would be theasible.

Originally posted by Ilja Preuss:
I see that differently. QA's job is to help the whole team (including the developers and the customer) to get better feedback about how close the developed system is to what the customer actually requested. In my eyes, that is more of a win-win situation than being about protecting one side from the other.

Then you're not really talking about unit tests. True unit tests are of single functions in isolation and rarely does a single function mean anything from a customer perspective. System tests would be what customers would be most interested in, not unit tests.

While I admire your desire to view the situation from a more positive perspective, it is about protection. QA's entire role is to protect bad code from getting to the customer. If QA is doing its just properly I should be able to write terrible code (on purpose or otherwise) that never gets to the customer. QA is a form of accountability, or in other words blame.

[Scott Selikoff]

Originally posted by Esther Schindler:

Some developers are better at it than are others. What do you think those developers do right? (Which takes us far off the topic of unit testing, but I can't help myself.)

"Tests (not neccessary formal tests) of boundary conditions"

But let me diverge for a minute...

There are some cases when I am coding that I would not want to write test cases because they were far too simple and other cases, especially ones that have intense logic or calculations I would always write test cases. But, there are some developers I would prefer to see test cases from no matter what they were doing, since their code has been prone to errors time and time again.


Back to the mention of "boundary conditions". I think some developers finish a piece of code and don't really test it. By test it, I don't mean write unit tests I mean just run the application a few times. For example, if you're writing a program that searches for books in a database, once you're done you should perform a number of test queries often using boundary conditions such as passing in an empty string, passing in a really long string, passing in string for which there are no books in the database, etc.

Running simple boundary conditions tests that take only as much time as it takes for your program to load, are extremely valuable and should always be done. Developers that only perform simple tests with no boundary conditions often find out there are problems later when there are no books or matchings in the database, or when someone enters numbers for integers and visa versa, etc.

And just to reiterate, I'm not advocating formal tests cases which may take longer to write then the real code (and often do) I'm just saying simple boundary condition testing is often missing because of lack of forsight and imagination on to what real users of the code might do.
[ November 13, 2005: Message edited by: Scott Selikoff ]

[Ilja Preuss]

Originally posted by Scott Selikoff:
I see people rushing to it as if it will allow projects to finish bug free without really understanding it.

Mhh, and you think they are motivated to do that by doing TDD?

Also, I find the notion of unit testing in J2EE perposteruous.

You do? Why?

I don't see QA testers having the knowledge to write low level unit tests. Most QA testers I've met couldn't read code at all, but knew how to write good test cases so in practice I'm not sure how often this would be theasible.

Yes, I was more thinking about tests defined via something like http://fitnesse.org/

Then you're not really talking about unit tests.

Correct. This was in response to the paragraph about QA testing.

While I admire your desire to view the situation from a more positive perspective, it is about protection. QA's entire role is to protect bad code from getting to the customer. If QA is doing its just properly I should be able to write terrible code (on purpose or otherwise) that never gets to the customer.

Well, yes. And the programmers should try to write code that QA never finds a problem with. If QA *does* find a problem, the whole team should take it as valuable feedback to improve the process.

QA is a form of accountability, or in other words blame.

I think I'm not understanding what you are trying to say here...