Vigilant Software Blog

Sometimes the simplest obstacles can stop you from doing something right (or doing it at all). The battery in the TV remote is dead — do you get up and change the channel, or do you suddenly develop a fascination with Snuggie infomercials? You would go to the gym, but by the time you get home from work you're too tired to go back out.

Often, these trivial inconveniences are sufficient to stop you in your tracks. Or, even if they don't stop you, they require you to summon some mental energy to overcome them. (As sad as that sounds, it's how our minds work, for the most part. We have a finite amount of willpower to use each day, so consider it a precious resource.)

Rather than force yourself to overcome these obstacles, sometimes you can make a systemic change that eliminates the obstacle altogether. For example, bring your gym clothes to work, and go straight to the gym from work. This way, you've simplified the effort involved in getting to the gym (packing for it in the morning, when you have plenty of energy), and you've changed the system so that success is more likely.

Obstacles in software development

This concept has numerous applications to the area of software development. In fact, I can give you a heuristic to recognize the situation any time it occurs: "ugh!"

If you're in the zone, the development process flows naturally, your work gets done, and you barely notice the passage of time. However, occasionally you hit a step where the flow stops, and you say (mentally or aloud), "ugh!"

This is your red flag; it's where human nature will occasionally cause us to cut corners. Instead of writing proper tests for the new feature, you may just try a few examples by hand and consider that good enough. It's also where you're more likely to make mistakes due to inattention, trying to get through the painful part quickly. Or maybe you'll decide it's a good time to take a break and read reddit.

What can be done? When you have the "ugh" reaction, take a step back and ask why this step is so painful, and how it could be improved (or eliminated if a systemic change can be put in place).

For example, maybe writing tests is harder than it should be. Sometimes it takes 100 lines of setup code before you can get to the point where you can even test your new feature. It may be that you simply have to force yourself to get over that obstacle and write the test anyway.

However, it may also be that you can rearrange your code to be more testable. Why is all that setup code necessary? Can you spend 15 minutes refactoring and make the application significantly easier to test? If you can, you will remove the obstacle not just now, but each time someone needs to work on this chunk of code.

Conclusion

Sometimes trivial obstacles can sap your energy, slow you down, or stop you from doing the right thing. The solution isn't always to bite the bullet and slog your way through. Instead, try to occasionally listen to that little voice that says "ugh" and ask how you can change things and remove the obstruction from your path.

What are the obstacles in your development process?

Nice summary of the state of systems programming in this InfoWorld piece, Hail the return of native code and the resurgence of C++:

"Last week, the latest version of the ISO C++ Standard was approved by unanimous vote. It's the first major revision of the language in 13 years. Now officially known as C++11, the new standard introduces features designed to make it easier to develop software for modern parallel processing architectures, including lambda expressions and new data types for concurrent computing.

"Not that C++ really ever went away. With its older cousin C, it remains one of the most popular languages for systems programming and for applications that call for performance-intensive native code, such as 3D game engines."

It's nice to see the importance of native code being recognized; not everyone can afford the performance tradeoffs associated with interpreted languages. Continue reading the article here.

Maybe you're passionate about programming; you like puzzles and enjoy the process of finding the best solution. You know the satisfaction of replacing 500 lines of spaghetti with 50 lines of elegance. You're pragmatic^TM: you don't repeat yourself and never leave broken windows behind you.

That mentality is important; it's what keeps you sharp, looking for new ideas and solutions, never satisfied with “good enough”. A programmer who rests on his laurels quickly finds himself a dinosaur facing the proverbial meteor.

Time to test

You've crafted something beautiful, it's time to make sure it works by writing some tests. If you're like most people, writing tests can feel mundane, something we do out of necessity rather than because we really want to.

I'm here to tell you this attitude has consequences. Assigning test code second-class status in your mind affects the way you develop tests. Writing tests becomes a special kind of programming where the normal rules don't apply: the code isn't part of a deliverable, quality isn't a concern, efficiency is an afterthought, and maintainability is usually ignored. If you think about it, this doesn't make any sense.

Quality matters

There are reasons that the best practices in software development have evolved over time. These reasons don't just apply to deliverable products because they affect the entire process of developing any non-trivial software.

Consider:

1. Test code is still code.
2. Tests are maintained and improved over time, like any other code.
3. Tests are used (run by developers) every day.
4. Tests are relied upon to prove the quality of your application code.

The consequences of these facts:

Maintenance matters. Your tests will generally be around as long as the application code they exercise. They're anything but throw-away. Changes to the application often mean changes to the test — if the inputs or outputs of a function change, the tests need to be updated to reflect that. If you have 100 tests that all copy-and-paste the same logic, updating the tests will be a miserable affair.

Also, just like your application code, you should write your tests with the assumption that someone else will have to read the code some day. When a new developer has to update your tests, it is just as important that you follow good development processes. Use clear names and informative comments, write reusable utilities, factor out dupliation. There's nothing special about application code that requires it to be maintainable while your tests are allowed to be a mess.

Performance matters. Test that take 20 minutes to run will not be run often, because it would seriously cut into productivity. There is still a place for large system-level tests, but you should have a set of comprehensive unit tests that take less than a minute to run. This allows developers to run the tests prior to every commit and minimize regressions.

Reliability matters. Your tests should be reliable. They should depend on very little aside from what they test — don't count on some database being available on the network, or some file on NFS, or even other major components of your application. Keep tests focused on the component they test, and they will function well for a long time. A change in an unrelated component shouldn't break your test, otherwise that test creates extra maintenance work proportional to how often it breaks.

Learning to love test code

You already know how to write great code, sometimes you just need to be reminded why it matters.

The best way to address this is to cultivate a love of your tests. Hold your test code at the same level of esteem as the rest of your code. If you are able to see beauty even in your tests, writing them will become fun again, and you will want to invest yourself in doing them right.

It's been said that learning to program is something like acquiring a super power. You get to tell computers what to do and they have to listen to you. Feats that are impossible to the average person are not only possible for you, they're easy.

This power can be used for trivial, every day things, like customizing your computer's or smart phone's interfaces (especially if they're open source). Or you can use this power to analyze 80 years of stock market data and try to get rich predicting the next move. Or you can use it to scrape eBay and craigslist for good deals on antiques.

Given how versatile and useful this power is, it's remarkable to me that developers often neglect to use it.

Warning Signs

Does it feel like developing your project is an uphill battle, constantly taking steps backward and struggling to get a foothold? Do you have a great development team, yet your application quality wavers and things feel more unstable than they ought to be? Does every bug feel like a heisenbug?

Hopefully, you have a development process that involves some rules about what developers should or shouldn't do. This could be things like:

• Test that you haven't broken anything prior to committing new changes.
• Update the changelog after a commit.
• Never use platform-specific APIs.
• Format code according to a coding standard.

Developing a nontrivial software project is a constant battle against complexity and entropy. The holy grail is the ability to develop new features while introducing zero regressions and spending zero time on maintenance.

Without automation, the normal state is a sort of controlled chaos. Everyone makes a best effort to follow the procedures, but, being human, they occasionally forget. Or they each interpret the rules differently. Or ther code works on their systems, but fails somewhere else (or everywhere else).

Why is this happening?

Here's the problem with rules that people need to manually follow: every single time, the person must overcome inertia and summon the energy required to perform some (probably tedious) tasks. Even if the manual steps are small, they still represent some energy threshold that must be crossed to make them happen. That means that there will be occasions where someone is tired, distracted, or forgetful enough, and a shortcut is taken.

Even when you are mostly successful at manually applying rules like the above, it is a drain on your productivity. If you have to perform manual testing every time you make a change, you are draining energy away from developing new value and pouring it into simply maintaining the status quo.

Super Cow Powers?

It's time to bring that super power to bear on this problem. Conveniently, things that are boring and tedious are perfectly suited to being performed by a computer (at least until they become self-aware). These are problems that we can solve with software, usually with a minimal amount of scripting and tools like CruiseControl or Jenkins.

We can use automation to not just overcome inertia, but create a new steady state. That is, create a development process that is self-sustaining and increases your product's quality with nearly zero overhead.

Here's what to do:

• Have a test suite that can be run at the push of a button. It should run quickly and every test should pass every time.
• Automate your tests and run them (at least) every night. Even better, use continuous integration and test after every push.
• Automate enforcement of any coding standard that can be enforced programmatically. Ideally as a pre-commit hook.
• Automatically generate documentation from code wherever possible.
• Add regression tests whenever you fix a bug, so that bug can never reappear.

Automating development and testing tasks is an almost magical thing. It brings a certain kind of peace of mind to get a clean, passing test suite every day. You can sleep at night. And it makes inertia your ally rather than your foe: the new steady state is one that maintains stability and doesn't drain your energy in the process.

Are you interested in code quality? This post is part of a series called Defect Spotlight. In it, we'll be bringing you instructive examples of real-world defects found by static analysis. Learn from the mistakes shown here!

Most of what we do revolves around finding software defects. They're everywhere, so it's not usually hard to find them. One thing we have learned is that Sentry spots not only the hard-to-find defects, but also the ones hiding in plain sight.

We've been analyzing many open source C and C++ programs on a nightly basis. As we continue growing the list of projects we inspect, we hope to help improve the quality and security of open source software.

When we add a new project, we tend to ignore all of the existing defects Sentry reports. This isn't to say that Sentry didn't find a defect worth reporting to the project, it most likely did. The issue is, developers care much less about old defects than they do about new ones. Especially brand new ones, like the ones committed last night.

In fact, in our experience, this is so important that reporting bugs immediately is the key factor in motivating developers to fix them. When someone is actively working on the code in question, it's generally easier (and safer) to fix a bug than it would be several months later.

Recently, we added Audacity to the list of C++ projects that we analyze nightly. Sentry found 305 potential defects. I casually reviewed some of the memory leaks reported by Sentry and found one that was pretty blatant. I decided in this instance to report this old issue.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

double AudacityProject::NearestZeroCrossing(double t0)
{
   int windowSize = (int)(GetRate() / 100);
   /* [Variable dist allocated but never returned or deleted.] */
   float *dist = new float[windowSize];
   int i, j;

   for(i=0; i<windowSize; i++)
      dist[i] = 0.0;

   TrackListIterator iter(mTracks);
   Track *track = iter.First();
   while (track) {
      /* [Code that doesn't touch dist snipped for brevity] */
      for(i=0; i<windowSize; i++) {
         if (windowSize != oneWindowSize)
            j = i * (oneWindowSize-1) / (windowSize-1);
         else
            j = i;

         dist[i] += oneDist[j];
      }

      track = iter.Next();
   }

   int argmin = windowSize/2; // Start at default pos in center
   float min = dist[argmin];
   for(i=0; i<windowSize; i++) {
      if (dist[i] < min) {
         argmin = i;
         min = dist[i];
      }
   }

   return t0 + (argmin - windowSize/2)/GetRate();
}

The memory leak that Sentry found was committed here. Take a look at the function NearestZeroCrossing above. It allocates the variable dist with the new[] operator, but never returns or deletes it, so the associated memory will always be leaked. In this case, the Audacity team was happy to accept a patch to fix it.

The interesting thing to note here is not that Sentry found a memory leak, it's that we found and fixed a memory leak that was eight years old.

Software does not fix itself. It takes good programmers to improve software. Even then, software defects are not always easy to spot, especially when the program appears to run well. Unit, integration and system testing are great ways to shake out defects and ensure software behaves correctly. Static analysis makes a great addition, helping you spot those hard to find bugs that don't occur on the happy path. Or, as in this example, it can find those eight-year-old bugs hiding in plain sight.