Tuesday, April 14, 2009

Your unit tests will lie to you.

I've become a very big proponent of unit tests. I don't practice strict test-driven development, in which you write the test and then the code that meets the test. However, I do write JUnit tests for almost all of my classes. (High-level OpenJUMP classes in the core are an exception to this. They can be very hard to write unit tests for.) I find unit tests are especially worth the pay-off when writing low-level libraries and parsers, which I seem to do a lot.

Did you know that your unit tests can lie to you?

It is the truth. Consider the following method of a Java class, which compares two make-believe CustomColor objects:

public boolean customColorsAreEqual(CustomColor argCompareTo)
// Method implementation goes here.

Now consider a unit test that tests this method. It creates two CustomColor objects that should be equal, and then uses the above method to compare them. If the boolean value returned by the method is true, the test passes, if the method is false, it fails. Our test method might look like this:

public void testCustomColorsAreEqual(CustomColor argCompareTo)
CustomColor myGreen1 = new CustomColor("My Green");
CustomColor myGreen2 = new CustomColor("My Green");

boolean areEquals = myGreen1.customColorsAreEqual(myGreen2);

if(areEquals == false)
fail("The method did not recognize two CustomColor objects that are equal.");

The programmer runs the unit test, which passes, and assumes the method is bug-free. But the unit test lied.

The programmer neglected to have his test repeat a comparision with two CustomColor objects the programmer knew to be unequal. He should modify his test to add this comparision as well. If he doesn't, a bug could result in the method under test indicating that two unequal CustomColor objects are equal.

Most tools I know of to examine JUnit test coverage deal with the program as a whole. The programmer also needs to think about the test coverage of indivdual methods under test. You should test the method for all likely scenarios, not just the first scenario that pops in your head. This often means running the method under test multiple times from the same test method, but each time with different arguments or program state.

I'll post some more comments on unit testing in Java later.

The Sunburned Surveyor