Vigilant Software Blog

A new bug report landed on your desk this morning. Maybe it's from an customer in the field, maybe QA found it, or maybe you found it and reported it yourself. (You are using issue tracking software, right?)

If you're experienced in this sort of thing, your first step is usually to attempt to reproduce the issue. How can you know if you've fixed the problem without first seeing it fail? Maybe you can't reproduce it and need more information, or maybe it's actually working as intended and the report is mistaken.

For this story, let's assume the bug report is a real defect and you are able to replicate it. What's your next step?

If you said "write a test", you get a cookie. Write a test and watch it fail. (You have a test suite, right?) Do not pass go; don't start debugging, don't start hacking around in the code, and definitely don't commit anything.

Write a test, watch it fail.

Once you have a failing test, you are free to start changing things. Diagnose and fix the bug, re-run your test suite. Not only should all the old tests pass (verifying your change didn't break anything), but the new test should pass (proving your "fix" actually fixes something).

This approach accomplishes a few things:

1. It forces you to stop and actually reproduce the issue.
2. It ensures that you've done a before and after test, and that your change actually fixes the issue.
3. Finally, it prevents this issue from re-appearing in the future - you get a regression test for free by following this process.

It's frustrating to squash a bug, only to have it reappear later, right? This is how you make sure they stay dead. It's actually a form of test-driven development, an approach that is well-suited to bug fixes.

Maybe you're passionate about programming; you like puzzles and enjoy the process of finding the best solution. You know the satisfaction of replacing 500 lines of spaghetti with 50 lines of elegance. You're pragmatic^TM: you don't repeat yourself and never leave broken windows behind you.

That mentality is important; it's what keeps you sharp, looking for new ideas and solutions, never satisfied with “good enough”. A programmer who rests on his laurels quickly finds himself a dinosaur facing the proverbial meteor.

Time to test

You've crafted something beautiful, it's time to make sure it works by writing some tests. If you're like most people, writing tests can feel mundane, something we do out of necessity rather than because we really want to.

I'm here to tell you this attitude has consequences. Assigning test code second-class status in your mind affects the way you develop tests. Writing tests becomes a special kind of programming where the normal rules don't apply: the code isn't part of a deliverable, quality isn't a concern, efficiency is an afterthought, and maintainability is usually ignored. If you think about it, this doesn't make any sense.

Quality matters

There are reasons that the best practices in software development have evolved over time. These reasons don't just apply to deliverable products because they affect the entire process of developing any non-trivial software.

Consider:

1. Test code is still code.
2. Tests are maintained and improved over time, like any other code.
3. Tests are used (run by developers) every day.
4. Tests are relied upon to prove the quality of your application code.

The consequences of these facts:

Maintenance matters. Your tests will generally be around as long as the application code they exercise. They're anything but throw-away. Changes to the application often mean changes to the test — if the inputs or outputs of a function change, the tests need to be updated to reflect that. If you have 100 tests that all copy-and-paste the same logic, updating the tests will be a miserable affair.

Also, just like your application code, you should write your tests with the assumption that someone else will have to read the code some day. When a new developer has to update your tests, it is just as important that you follow good development processes. Use clear names and informative comments, write reusable utilities, factor out dupliation. There's nothing special about application code that requires it to be maintainable while your tests are allowed to be a mess.

Performance matters. Test that take 20 minutes to run will not be run often, because it would seriously cut into productivity. There is still a place for large system-level tests, but you should have a set of comprehensive unit tests that take less than a minute to run. This allows developers to run the tests prior to every commit and minimize regressions.

Reliability matters. Your tests should be reliable. They should depend on very little aside from what they test — don't count on some database being available on the network, or some file on NFS, or even other major components of your application. Keep tests focused on the component they test, and they will function well for a long time. A change in an unrelated component shouldn't break your test, otherwise that test creates extra maintenance work proportional to how often it breaks.

Learning to love test code

You already know how to write great code, sometimes you just need to be reminded why it matters.

The best way to address this is to cultivate a love of your tests. Hold your test code at the same level of esteem as the rest of your code. If you are able to see beauty even in your tests, writing them will become fun again, and you will want to invest yourself in doing them right.

It's been said that learning to program is something like acquiring a super power. You get to tell computers what to do and they have to listen to you. Feats that are impossible to the average person are not only possible for you, they're easy.

This power can be used for trivial, every day things, like customizing your computer's or smart phone's interfaces (especially if they're open source). Or you can use this power to analyze 80 years of stock market data and try to get rich predicting the next move. Or you can use it to scrape eBay and craigslist for good deals on antiques.

Given how versatile and useful this power is, it's remarkable to me that developers often neglect to use it.

Warning Signs

Does it feel like developing your project is an uphill battle, constantly taking steps backward and struggling to get a foothold? Do you have a great development team, yet your application quality wavers and things feel more unstable than they ought to be? Does every bug feel like a heisenbug?

Hopefully, you have a development process that involves some rules about what developers should or shouldn't do. This could be things like:

• Test that you haven't broken anything prior to committing new changes.
• Update the changelog after a commit.
• Never use platform-specific APIs.
• Format code according to a coding standard.

Developing a nontrivial software project is a constant battle against complexity and entropy. The holy grail is the ability to develop new features while introducing zero regressions and spending zero time on maintenance.

Without automation, the normal state is a sort of controlled chaos. Everyone makes a best effort to follow the procedures, but, being human, they occasionally forget. Or they each interpret the rules differently. Or ther code works on their systems, but fails somewhere else (or everywhere else).

Why is this happening?

Here's the problem with rules that people need to manually follow: every single time, the person must overcome inertia and summon the energy required to perform some (probably tedious) tasks. Even if the manual steps are small, they still represent some energy threshold that must be crossed to make them happen. That means that there will be occasions where someone is tired, distracted, or forgetful enough, and a shortcut is taken.

Even when you are mostly successful at manually applying rules like the above, it is a drain on your productivity. If you have to perform manual testing every time you make a change, you are draining energy away from developing new value and pouring it into simply maintaining the status quo.

Super Cow Powers?

It's time to bring that super power to bear on this problem. Conveniently, things that are boring and tedious are perfectly suited to being performed by a computer (at least until they become self-aware). These are problems that we can solve with software, usually with a minimal amount of scripting and tools like CruiseControl or Jenkins.

We can use automation to not just overcome inertia, but create a new steady state. That is, create a development process that is self-sustaining and increases your product's quality with nearly zero overhead.

Here's what to do:

• Have a test suite that can be run at the push of a button. It should run quickly and every test should pass every time.
• Automate your tests and run them (at least) every night. Even better, use continuous integration and test after every push.
• Automate enforcement of any coding standard that can be enforced programmatically. Ideally as a pre-commit hook.
• Automatically generate documentation from code wherever possible.
• Add regression tests whenever you fix a bug, so that bug can never reappear.

Automating development and testing tasks is an almost magical thing. It brings a certain kind of peace of mind to get a clean, passing test suite every day. You can sleep at night. And it makes inertia your ally rather than your foe: the new steady state is one that maintains stability and doesn't drain your energy in the process.

Since the Sentry 3.0 release, we've been enjoying the new plugin system that lets us (and our users) write new scanners quickly in Python. It's really satisfying to be able to find new defects so easily, and as a result, we've been adding quite a few!

One key change is that our results, which are identified by a unique error code, can now be registered with Sentry at run-time. This means any scanner can define new result kinds (and usually more than one), and then start reporting errors with these codes. This is true for both the scanners we ship as well as the ones you write yourself.

Figure: Analysis profile dialog

As long as the plugin is present, Sentry will load and make available the result kinds that are defined there. These result codes are available for use in your analysis profiles, which means you can enable just the results you care about.

This kind of fine-grained control means you can tell Sentry to use only the parts of a scanner that are useful to you. For example, if someone shares a new scanner with you and it turns out that it produces some spurious results, you can just disable that result kind in your profile while leaving the rest of the scanner's functionality enabled.

For me it's exciting, because this is where static analysis becomes truly agile. Now you can quickly develop new scanners, share them with your teammates, and customize their output to your liking -- without the need for any updates to Sentry itself.

Sometimes a small feature goes a long way. We've been scaling up our open source efforts recently, and it became obvious that the ability to plug into a continuous integration (CI) process makes life a lot easier.

Several months ago, we developed a new feature to compare two runs and identify the differences between them - results that were added or removed between the two runs. This enables you to quickly determine if a patch fixes a defect found by Sentry, or if it introduces a new one.

It seemed like a straightforward, useful feature, but it turns out to be essential to a good CI process. Because run diffs are available from the command line, it's a simple matter to write a script that analyzes a project on a regular basis and only notifies you if new defects have been found.

The upshot of this feature is that we can integrate dozens of open source projects into a continuous analysis process, yet spend very little time digging and inspecting results. For Sentry users, this basically means you get all the benefits of automated testing with almost no extra time spent in your development process!