How to know that I have covered all key paths in my code regardless of the code coverage percentage?

[Jose Almonte]

I've had instances where my code is 100% unit test covered, only to later have a bug appear in production that shows I missed a key test case. Is there a heuristic I can follow to help prevent this kind of thing? Thanks.

[Campbell Ritchie]

Welcome to the Ranch

I am afraid you didn't have 100% coverage; an incomplete test is worthless. You need to test with inputs from the expected range and abnormal input, e.g. 0, negative numbers. I recommend that if you get such abnormal input, or an arithmetic overflow or similar, you should not allow execution to proceed, but throw an exception

[Stephan van Hulst]

You mean, you've had instances where your code had 100% coverage. If your code was 100% unit tested, you wouldn't have any bugs. That's also practically impossible. Your code will always contain a bug somewhere. It's a fact of life.

Code coverage measures how many of your execution paths are covered by unit tests. If you have 100% coverage, that means that every execution path through your code was triggered by at least one unit test.

The problem is that even if all execution paths are covered, that doesn't necessarily mean that your code is completely tested. After all, you can be missing code that handles a specific edge case, AND you can be missing a unit test for that non-existent execution path.

I'm not sure if there are simple guidelines for ensuring all your code is properly unit tested. Testing software, like programming, is not just science. Part of it is an art as well.

[Rob Spoor]

A famous quote:

Edsger W. Dijkstra wrote:Program testing can be used to show the presence of bugs, but never to show their absence!

The only way to handle this issue is to keep your tests, and every time a bug is found:

Write a new test first, that should test the desired behaviour with the actual input. This test should fail, because the code failed.

Fix your code to make this test succeed. Don't change anything else, especially not tests.

Verify that your existing and new tests all succeed.

This is a procedure followed by several large projects, including OpenJDK. I've also had to write such tests for some pull requests for Apache Commons IO; without them, pull requests are rejected.

Changing existing tests should only be done if you're 100% sure that the test is indeed incorrect.

[Mike Simmons]

Campbell Ritchie wrote:I am afraid you didn't have 100% coverage; an incomplete test is worthless.

Well, that seems harsh.  From my perspective, it's nearly impossible to get truly complete coverage - yet there's still value to each test case you do have.  Don't let the quest for perfection get in the way of making useful progress.

I agree with Campbell's other comments, and all of Stephan's.

Jose, it sounds like maybe a tool reported your code coverage as 100%.  And yet, as you've found, that doesn't necessarily mean it's really 100%.  This depends in part on what definition of code coverage is used, and what tool is used to measure it.  100% might mean that 100% of methods have been executed at least once, or 100% of lines have been executed at least once.  But there are different things that can happen within a line, too.

var message = message != null ? message : defaultMessage;

That will behave differently if message was null, than if it wasn't.  Did you test both?  Or:

boolean isAuthorized = isLoggedIn() && (isAdmin() || isRegularUser() && !isBanned());

That will call different methods, depending on the results of each method called.  Did you test all possible variations?

You can use something like JaCoCo which measures instruction-level coverage.  Which means, at the bytecode level, how many of the bytecode instructions are getting called?  That's pretty good.  Even then, though, things can be missed.  What happens if a method call throws an exception?  Or if a mathematical operation throws an ArithmeticException?  These possibilities are generally not covered by most code coverage tools, as far as I know.  In part because in principle, any method call could thrown an unchecked exception at any time, for unknown reason.  Tools (and humans) can't really cover all the things that might theoretically happen. Generally humans have to make their best guesses about things that could happen, based on what they understand about how the code is used.  And when you find out you missed something, add another test (or group of tests) to help fill the remaining gaps.

[Campbell Ritchie]

Yes, I think I was too harsh on those tests: sorry.

[Tim Holloway]

Campbell Ritchie wrote:Yes, I think I was too harsh on those tests: sorry.

Definitely too harsh. Obviously, the more bases you can cover, the better, but you're probably not allowing for gamma-ray bursts through the RAM, just as an example. To say nothing of the user whose keyboard is set to Latvian or the Leap Second come December. You can over-test, but Murphy's Law will still get you in the end. Better imperfect tests than no tests, though.