How to modify untested existing code safely
July 8th, 2008 at 2:56 pm by Jan Willem Tulp
You may know, or not know, that there are various kinds of software tests that may help you ensure that your code meets the requirements and behaves as intended, like Unit Testing, Scenario Testing, Integration Testing, etc. And the better your test coverage, the lower the risk and the safer you feel modifying your code since you can always run your tests to make sure you haven’t broken anything. Now, what do you do if you work on existing code without Unit Tests? Making a change can be a risky because you can’t run any Unit Tests. How do you make sure that your change won’t break anything?
One possibility is to create Characterization Tests. Characterization Tests are tests that describe (characterize) the behavior of existing code. It’s important to realize that characterization tests therefore do not verify the correct behavior of your code, but only the existing behavior. So the purpose of Characterization Tests is to make sure that if you make a change, your change doesn’t affect the existing behavior of your application.
So basically when you create a Characterization Test, you observe the output for a certain input of a method, and then you write a test to assert this. For example, if you observe that the output of the method sum(2, 3) is 5, then a test can be created that asserts 5 is the output for the input 2, 3. When you make a change to the code, the result of your test should still be the same. Now the fun thing is, since Characterization Tests are based on existing code, you could generate some Characterization Tests from your existing code. JUnitFactory seems to be such a tool, but at this time the website seems to be down.
A best practice for writing Characterization Tests is only to write Characterization Tests for code that may be affected by your change. If methods are very long, have lots of side effects, and if the object or the method has lots of dependencies, then it may not be very easy to create a Characterization Test. Two factors that influence the testability of your code are:
- how easy is it to construct the Object Under Test
- how easy is it to execute the method to be tested in order to assert something
You can start writing Unit Tests now, even for existing code. And Characterization Tests really reduce the risk and make it safer to change existing code. Writing Tests also gives you more insight into existing code, and your tests will make your colleague developers happy for sure: they feel safer to make changes as well if you wrote a test.
To read more about Characterization Tests, see:
- Working Effectively With Legacy Code
- Working Effectively With Characterization Tests
- Working Effectively With Characterization Tests - Part 2
- Working Effectively With Characterization Tests - Part 3
- Fun With Automated Characterization Test Generation
Popularity: 488 points
July 9th, 2008 at 7:32 am
Thanks for the very helpful information.
Best Regards
Offshore Software Development
http://www.softwebsolutions.com
July 9th, 2008 at 2:56 pm
My first thought when I read this was ‘great idea’. Then I began to think about how to implement the test code generator and came to the conclusion that it would be pretty difficult to write an automated tool that really reverse engineers a test case.
For instance the following class:
public class SecretNumber{
private int secret = 0;
public setSecret(int v){
this.secret=v
}
public boolean check(int guess){
return secret==guess;
}
}
I wonder if the tool is able to generate a test case for this scenario, because the output of check is based on the state of the class. The same input results in different outputs depending on the state of the object(the value of secret).
To bad i can’t check out there website right now, but my guess would be that it only support true functions not state based methods. Or is there some trick for this ?
July 9th, 2008 at 3:36 pm
Apparently their website is down because they have re-marketed as Agitar Automated JUnit Generation:
http://www.agitar.com/solutions/products/automated_junit_generation.html
July 9th, 2008 at 4:28 pm
@Arjen: a generator could generate some boundary checks (Integer.MAX_VALUE and Integer.MIN_VALUE), and perhaps some checks for special cases like 1, 0 and -1. Remember that characterization tests should describe existing behavior. These would all be valid testcases that test the check() method in isolation, and all based on the (initial) state of the SecretNumber class. This basic testing does not involve different internal states though.
With reflection a smart generator could of course infer that the internal state is changed by another method of SecretNumber, and thus a more complex testcase could be generated. I do agree that this is more challenging but not impossible. If you look at the demo at http://www.agitar.com/downloads/demos/junit_gen/junitGen.html you can see that some testcases depend on field values that are set using a constructor. Also, the more complex your methods, the more dependencies, the more side-effects, the harder it is to instantiate your class, etc. the harder it is to generate test cases (and also to manually create test cases).
And even if your generator only does some basic testing, it’s still better than having no tests at all.
July 9th, 2008 at 8:28 pm
Well don’t get me wrong I think products like this definitely have some potential, but they will never be anywhere near as useful as handwritten tests. For several reasons…
- The tests only covers a anonymous snapshot of the current behavior so the test code has no meaning (it wont tell you the ‘why’-part of the implemented algorithm section ). The test names and comments won’t tell you the reason for the behavior. And therefor it would also write tests to snapshot the behavior of a bug
- Nor will it be able to work with classes with state. Like the example I gave, but especially with more interaction and larger states.
Although the more intelligent a parser gets the more accurate these tests will be..
The example in the movie actually only covers tests for constructors and tests for methods without state.
But maybe if I find the time I’ll check out the product on my current project. Because I do believe they have a value (for instance to track the ripple effect of changes on code without tests)
August 13th, 2008 at 12:31 pm
I found your site on technorati and read a few of your other posts. Keep up the good work. I just added your RSS feed to my Google News Reader. Looking forward to reading more from you down the road!
September 1st, 2008 at 7:10 pm
I very much like your idea of “characterization tests.” Without doubt, you cannot maintain legacy code without thorough, automated regression tests. The best way to do this in the Oracle world is through Quest Code Tester, which I designed. You describe expected behavior, and it generates PL/SQL packages that exercise your code and verify correctness.
http://www.quest.com/code-tester-for-oracle