Vigilant Software Blog

A new bug report landed on your desk this morning. Maybe it's from an customer in the field, maybe QA found it, or maybe you found it and reported it yourself. (You are using issue tracking software, right?)

If you're experienced in this sort of thing, your first step is usually to attempt to reproduce the issue. How can you know if you've fixed the problem without first seeing it fail? Maybe you can't reproduce it and need more information, or maybe it's actually working as intended and the report is mistaken.

For this story, let's assume the bug report is a real defect and you are able to replicate it. What's your next step?

If you said "write a test", you get a cookie. Write a test and watch it fail. (You have a test suite, right?) Do not pass go; don't start debugging, don't start hacking around in the code, and definitely don't commit anything.

Write a test, watch it fail.

Once you have a failing test, you are free to start changing things. Diagnose and fix the bug, re-run your test suite. Not only should all the old tests pass (verifying your change didn't break anything), but the new test should pass (proving your "fix" actually fixes something).

This approach accomplishes a few things:

1. It forces you to stop and actually reproduce the issue.
2. It ensures that you've done a before and after test, and that your change actually fixes the issue.
3. Finally, it prevents this issue from re-appearing in the future - you get a regression test for free by following this process.

It's frustrating to squash a bug, only to have it reappear later, right? This is how you make sure they stay dead. It's actually a form of test-driven development, an approach that is well-suited to bug fixes.

Maybe you're passionate about programming; you like puzzles and enjoy the process of finding the best solution. You know the satisfaction of replacing 500 lines of spaghetti with 50 lines of elegance. You're pragmatic^TM: you don't repeat yourself and never leave broken windows behind you.

That mentality is important; it's what keeps you sharp, looking for new ideas and solutions, never satisfied with “good enough”. A programmer who rests on his laurels quickly finds himself a dinosaur facing the proverbial meteor.

Time to test

You've crafted something beautiful, it's time to make sure it works by writing some tests. If you're like most people, writing tests can feel mundane, something we do out of necessity rather than because we really want to.

I'm here to tell you this attitude has consequences. Assigning test code second-class status in your mind affects the way you develop tests. Writing tests becomes a special kind of programming where the normal rules don't apply: the code isn't part of a deliverable, quality isn't a concern, efficiency is an afterthought, and maintainability is usually ignored. If you think about it, this doesn't make any sense.

Quality matters

There are reasons that the best practices in software development have evolved over time. These reasons don't just apply to deliverable products because they affect the entire process of developing any non-trivial software.

Consider:

1. Test code is still code.
2. Tests are maintained and improved over time, like any other code.
3. Tests are used (run by developers) every day.
4. Tests are relied upon to prove the quality of your application code.

The consequences of these facts:

Maintenance matters. Your tests will generally be around as long as the application code they exercise. They're anything but throw-away. Changes to the application often mean changes to the test — if the inputs or outputs of a function change, the tests need to be updated to reflect that. If you have 100 tests that all copy-and-paste the same logic, updating the tests will be a miserable affair.

Also, just like your application code, you should write your tests with the assumption that someone else will have to read the code some day. When a new developer has to update your tests, it is just as important that you follow good development processes. Use clear names and informative comments, write reusable utilities, factor out dupliation. There's nothing special about application code that requires it to be maintainable while your tests are allowed to be a mess.

Performance matters. Test that take 20 minutes to run will not be run often, because it would seriously cut into productivity. There is still a place for large system-level tests, but you should have a set of comprehensive unit tests that take less than a minute to run. This allows developers to run the tests prior to every commit and minimize regressions.

Reliability matters. Your tests should be reliable. They should depend on very little aside from what they test — don't count on some database being available on the network, or some file on NFS, or even other major components of your application. Keep tests focused on the component they test, and they will function well for a long time. A change in an unrelated component shouldn't break your test, otherwise that test creates extra maintenance work proportional to how often it breaks.

Learning to love test code

You already know how to write great code, sometimes you just need to be reminded why it matters.

The best way to address this is to cultivate a love of your tests. Hold your test code at the same level of esteem as the rest of your code. If you are able to see beauty even in your tests, writing them will become fun again, and you will want to invest yourself in doing them right.