Debugging isn't magic! Patience and common sense goes a long way

When you work for as long as I have and on as many projects and software problems as I have encountered, it stands to reason that you will tend to end up working with other bright engineers. I have had the pleasure of working with developers on a great many applications that I have not only been proud to attach my name to, but that I am pleased to say are still out there in the wild working as expected, performing as designed, and making clients quite happy in the process.

When it comes to troubleshooting systems... I am sometimes really surprised where things can take a turn. I have often heard this described as a special skill set, something that some are better adept at than others. The more cynical folks I have met will ascribe it to a darker missive, one where developers are either lazy, incompetent, or simply blame the first thing they don't understand.

I don't believe its either some wizard's skill attained climbing the magic mountain, nor will I trash those who sometimes (often) struggle with troubleshooting distributed multi-user multi-threaded systems. I am not suggesting it does not require skill and knowledge -- like all engineering pursuits, a great deal of time and energy pursuing this craft is required, along with the desire and passion to do, and be, the best of the best.

But lets face it... we are human beings after all, and therefore fallible. The reason we unit test our code, we have peer based code reviews, and we aim to have independent QA teams test our code is because at the end of the day, it is difficult to challenge our own preconceived notions of what to look for when things go off the rails, and even more so when things silently fail.

Let me illustrate what I am talking about. Check out the following video and follow the instructions. Don't pause or fast forward, just play it all the way through.

Were you paying attention to everything? Did you catch the gorilla, or were you too busy just focused on the things you were only interested in i.e. the number of ball passes? That's how most developers debug, and why so many miss problems in the systems I described above.

I was recently working with a colleague on a debugging issue where he could not, for the life of him, figure out why no matter how many changes he made to his code, the code was not getting updated in his build. I should point out this was the first time he was testing with a Release build instead of the usual Debug build, and that should have been the first clue. Unfortunately, it has become mantra in the .NET world that other than differences in optimizations and JIT settings, and unlike in C++, there is no difference between Debug and Release. That would be true if we were talking about code generation in a literal sense, but there are other aspects to code generation, such as where the code is ending up.

When I looked at the project settings, I noticed that someone had blindly copied the output path for one of the assemblies to point to the Debug folder for both Debug and Release. As you can imagine, all of the assemblies in Release were being created in the Release folder except for the assembly he was making his code changes in. The error was his assumption that there was no difference, and so he never thought to look at the project file settings, a somewhat obvious debugging step to an outside set of eyes (mine).

Now, I'm not suggesting that I am somehow smarter or brighter. In point of fact, some years ago I spent far too many hours chasing down such a problem and vowed I will never get burned by the this again. Making mistakes is forgivable... making the same mistakes over and over is malfeasance. I choose to course correct after my mistakes 😊.

Patience and common sense is a nice way of saying we need to be careful of the ticking trap of cognitive dissonance. Debugging only appears to be magic to those unable or unwilling to challenge their own preconceptions.