Wednesday May 03, 2006

Tests Embody the Results of Thinking

I was talking to one of the engineers on my team the other day about bugfixing and unit tests.

We've reached the bugfixing stage in our project. To fix a bug, one must do a lot of thinking about the case that fails. Clearly, one must change the behavior in a particular case from incorrect to correct. But one must also ensure that the correct behaviors that occur in all the other cases are preserved. So, the goal isn't just to make some test change from fail to pass; it's to make sure that all the cases are handled correctly at the same time.

It's surprising how often people forget this. Sometimes a developer will make a change, and if the failing test succeeds, they think they're done. If they haven't ensured that all of the other cases are still correct, they're being irresponsible. (A greater irresponsibility is with managers who don't understand this.)

When dealing with code -- whether writing it initially or fixing bugs in it later -- a developer has to do a tremendous amount of reasoning, analysis, and thinking about all the cases the code has to handle. Unfortunately, the results of this thinking are rarely written down anywhere.

Sometimes these results are written into design documents. These are usually separate from the code, they get out of date, and nobody reads them anyway. Sometimes these results are written into comments. People sometimes reads the comments, but they're so often out of date that they aren't to be believed anyway.

“When the comments and the code differ, both are probably wrong.” -- attributed to Dennis Ritchie

The best place to write down these results is in unit tests. A good suite of unit tests acts as a reviewer who checks all the "other" cases for you automatically. If you run the unit tests all the time, they don't get out of date, either.

Next time you're reading a piece of code and you gain a significant understanding of it, don't throw away that understanding. Write it down in the form of a unit test.