This week's book giveaway is in the Agile and Other Processes forum. We're giving away four copies of Darcy DeClute's Scrum Master Certification Guide: The Definitive Resource for Passing the CSM and PSM Exams and have Darcy DeClute on-line! See this thread for details.
I've had instances where my code is 100% unit test covered, only to later have a bug appear in production that shows I missed a key test case. Is there a heuristic I can follow to help prevent this kind of thing? Thanks.
I am afraid you didn't have 100% coverage; an incomplete test is worthless. You need to test with inputs from the expected range and abnormal input, e.g. 0, negative numbers. I recommend that if you get such abnormal input, or an arithmetic overflow or similar, you should not allow execution to proceed, but throw an exception
You mean, you've had instances where your code had 100% coverage. If your code was 100% unit tested, you wouldn't have any bugs. That's also practically impossible. Your code will always contain a bug somewhere. It's a fact of life.
Code coverage measures how many of your execution paths are covered by unit tests. If you have 100% coverage, that means that every execution path through your code was triggered by at least one unit test.
The problem is that even if all execution paths are covered, that doesn't necessarily mean that your code is completely tested. After all, you can be missing code that handles a specific edge case, AND you can be missing a unit test for that non-existent execution path.
I'm not sure if there are simple guidelines for ensuring all your code is properly unit tested. Testing software, like programming, is not just science. Part of it is an art as well.
Campbell Ritchie wrote:I am afraid you didn't have 100% coverage; an incomplete test is worthless.
Well, that seems harsh. From my perspective, it's nearly impossible to get truly complete coverage - yet there's still value to each test case you do have. Don't let the quest for perfection get in the way of making useful progress.
I agree with Campbell's other comments, and all of Stephan's.
Jose, it sounds like maybe a tool reported your code coverage as 100%. And yet, as you've found, that doesn't necessarily mean it's really 100%. This depends in part on what definition of code coverage is used, and what tool is used to measure it. 100% might mean that 100% of methods have been executed at least once, or 100% of lines have been executed at least once. But there are different things that can happen within a line, too.
That will behave differently if message was null, than if it wasn't. Did you test both? Or:
That will call different methods, depending on the results of each method called. Did you test all possible variations?
You can use something like JaCoCo which measures instruction-level coverage. Which means, at the bytecode level, how many of the bytecode instructions are getting called? That's pretty good. Even then, though, things can be missed. What happens if a method call throws an exception? Or if a mathematical operation throws an ArithmeticException? These possibilities are generally not covered by most code coverage tools, as far as I know. In part because in principle, any method call could thrown an unchecked exception at any time, for unknown reason. Tools (and humans) can't really cover all the things that might theoretically happen. Generally humans have to make their best guesses about things that could happen, based on what they understand about how the code is used. And when you find out you missed something, add another test (or group of tests) to help fill the remaining gaps.
Campbell Ritchie wrote:Yes, I think I was too harsh on those tests: sorry.
Definitely too harsh. Obviously, the more bases you can cover, the better, but you're probably not allowing for gamma-ray bursts through the RAM, just as an example. To say nothing of the user whose keyboard is set to Latvian or the Leap Second come December. You can over-test, but Murphy's Law will still get you in the end. Better imperfect tests than no tests, though.
The secret of how to be miserable is to constantly expect things are going to happen the way that they are "supposed" to happen.
You can have faith, which carries the understanding that you may be disappointed. Then there's being a willfully-blind idiot, which virtually guarantees it.
We don't have time for this. We've gotta save the moon! Or check this out:
a bit of art, as a gift, the permaculture playing cards