I was very pleased to find this Patrick Smacchia’s blog entry where he shows how his quest for 100% code coverage helped him finding a bug. His other post presents deeper arguments in favour of striving for 100% code coverage, and I think the most important point is that 100% test coverage protects from coverage erosion. Exactly! And I have a fresh example that illustrates this point. I am responsible for maintaining and extending a library of .NET utility classes that are shared across company projects. Since these classes are frequently used, they just can’t fail, so I keep them well covered by tests, and two of four assemblies have 100% code coverage. With other two it is difficult to reach the absolute maximum, but the coverage is still pretty good. Recently I extended one of not fully covered assemblies with new classes and did it in a rush, without writing many tests for newly added classes. The coverage remained over 95%, so the library looked well covered. Now if I don’t take my time and write more tests for new code, in a few months nobody will probably be interested in doing anything with this assembly. There are other important things to do, there are projects with coverage less than 20%, shouldn’t we take them instead? Well, it depends. If we instead of using project metrics take the class metrics, then we will have to admit that we introduced a poorly tested class, perhaps with less than 15% coverage. Or we can take one step further and have a look at method coverage. In so poorly tested class there may be methods with 0% coverage. These are the consequences of what Patrick Smacchia calls coverage erosion: once we don’t strive for maximum coverage, we open the door for poorly tested code, but our metrics will continue looking impressive.
I often hear an argument that trying to reach 100% code coverage is not practical and finding an acceptable code coverage level is a result of a compromise. I think this argument is dangerous, because it is mostly used not by people who stop writing automated tests when they see the rest is difficult to cover, but by people who won’t even try to cover as much as they can, because you know… it is not practical… we have other things to do. Last year I worked with a consultant whom I asked to write more tests for a module that deals with customer data retrieval. He spent several days and reported that the work was done. When I checked the module code coverage, if was about 60%. I spent few more hours and it increased to 80. I asked him why he decided not to move on with writing more tests, and he answered that since the average code coverage across company projects was approximately on that level, he thought this would be sufficient. To me this sounds like if we stop fixing bugs because the world is imperfect.
So if code coverage analysis tool exposed that the statement “a = b;” is not covered by automated tests, what could be the reasons for leaving this line uncovered?
- Is this because this statement is not used? Actually this is quite possible, code coverage reports often help figure out obsolete code that can be removed. Then we should confirm this and remove it.
- Is this because this statement is supposed to be tested during manual QA session? Then what is less expensive and more reliable: assign the test task to a human being or instruct a computer to run it every night or even on every check-in?
- Is this because this statement is hard to reach? Well, we should refactor it then. This is another advantage of code coverage reports: they often help us make code more accessible.
- Is this because this statement is triggered on external event that only occurs in production? So how do we know it will work? We probably have an access to an external test environment, but it’s not for automated tests invoked thousands times a day. Then we can mock it, write a simulator, there are well known methods to deal with this stuff.
If you consider various options to reach untested code, you will find that in most cases it is not that hard. And most of it is one time investment: you acquire technique to reach most of your code, and you will be writing code that is easier to reach.
So when people insist on practical aspects of attempting to reach maximum code coverage, I can ask them about practical aspects of whole unit testing initiative. It’s the same, really. The main reasoning behind automated testing is that we can’t beat a computer in accuracy and punctuality. If nobody seems to be willing to accept these days that a software product is developed without automated tests, then why should it be different for a class? For a method? For a single statement? Look at such statement and ask yourself a question: why am I willing to release this line of code in production without running it through an automated test? You should have very good arguments if you are willing to do so.
And if you are still not fully convinced, please read a relatively old blog post by Eric Sink, a founder of SourceGear. Well, he is a smart guy, you agree? So even though he says that he would personally stop short of a recommendation that coverage must be 100%, look at his NCover report. 100%, not less! And he writes that “getting full coverage has been worth the effort”. You won’t find in sotware industry many other people who combine such brilliant development and business skills, so I am pretty sure that when he says “I am somewhat fanatical about automated testing and code coverage, I enthusiastically recommend using them”, he has certain practical reasons for such statement.
And if you convinced, then let me share with you a piece of my personal practice: if I have two projects, one with 20% code coverage, and the other with 95% code coverage, then assuming that their quality and readiness is of equal importance, I take a look at the one with highest coverage first. If it’s so close to completion, then bringing its code coverage to 100% takes it to another level where it is protected from coverage erosion. And it’s worth it.