Debugging for Fun and Profit

I tend to call any troubleshooting process "debugging." A week or so ago I debugged our lawnmower when it wouldn't start. Today I spent a lot of time (unsuccessfully) debugging a mysterious apache child crash related to a php application. Tomorrow I will undoubtedly debug some software at work and some other things in "real life."

I would guess that roughly 40% of my life as a professional programmer and system administrator has been spent debugging. By that I don't mean to imply that I only write broken code (although I probably do); what I mean is that like most writing is rewriting, most programming is debugging. A first attempt at solving a problem rarely works correctly. If it was that easy to solve it must not have been much of a problem in the first place right?

For me debugging software/hardware/whatever is mostly about thinking of the most common things that could be causing the malfunction and then systematically proving that these things are or are not the cause of the current problem. When my dad taught me how to keep our crusty old lawnmower running when I was a kid he gave me the checklist "fuel, fire, air." This magic formula has pointed to the cause of 90% of the problems I have ever had with engines.

  • Does it have gas in the tank?
  • Is the gas getting to the cylinders?
  • Is the battery/magneto/alternator working?
  • Is the charge making it past the points/distributor?
  • Is the spark plug fouled?
  • Is the air filter clogged?
  • Is the choke stuck closed?

The order here is important too. On each branch of the decision tree we start at one end of the subsystem and walk towards the other end checking the key weaknesses in between. The branches are also arranged in order of most frequent failure. All this ordering hopefully get you to the root problem earlier rather than latter. You need to tailor the tree based on prior experience. When working with the beater Datsun I drove in college, fire came before fuel because it had constant alternator and distributor problems.

Software debugging works just like the mechanical troubleshooting process for a malfunctioning engine. The only difference is the decision tree. If I'm working on a network connectivity problem the quick checklist usually goes something like "interface, firewall, routing."

  • Is the physical link in place?
  • Is the interface active?
  • Is there a local firewall block?
  • Is there an upstream firewall block?
  • Is there an outbound route?
  • Is there an outbound route from the next hop?

Just like the lawnmower to car differences, this list gets modified based on what I'm working on. This order worked best for border connectivity issues on the frame relay networks I used to work with. The network I'm on today calls for the firewall checking first because 50% of the time that's where the problem is.

So how do you construct this sort of list when you run into a problem you've never seen before? I tend to use first principles or cartesian doubt technique. Based on the nature of the failure you are seeing and the knowledge you have of the system, think of everything that could have gone wrong. Start with a likely cause and follow it out until you are satisfied that particular subsystem is working. Move on to the next and repeat. If the problem is in code that you (or someone you know) wrote think about the things that are typical problems in your code. Do you often make fencepost errors? Are you working in a language that is prone to API mistakes like calling a method with the arguments in the wrong order? Are you working in a threaded environment and experiencing a concurrency issue?

I wish I had something better to say at the end of all this, but maybe some part of it is non-obvious: think about what could be broken, verify that it's not broken, repeat. Sounds pretty simple when I put it like that, but there's some magic in there that I can't quite elucidate.

What I Read On My Summer (OK Spring) Vacation

There's no central theme, but it turned out that there were a lot of books dealing with themes of computer and human intelligence. There was also a lot of just plain good reading and a handful of crazy crackpot tinfoil hat thrown in for good measure.

The Stephenson novels were re-reads of two of my all time favorite books. I'd love to work on a project to make something like the Young Lady's Illustrated Primer.

Mystery IP Address

A couple of months ago Steve noticed a lot of hits to the new company website from three IP addresses in the 38/8 block. By a lot of hits I mean something like 75% of the total traffic. The MaxMind database we use to generate region and ISP information for the traffic reports said that these addresses belonged to "Performance Systems International". Steve googled for this company and came up with a whole lot of blog and forum posts that claimed this company was home to a rogue bot that has plagued a lot of people.

I decide to dig a little deeper because most of the links that Steve sent me seemed to be a little under informed. Here's what I found:

"Performance Systems International" is the original name of PSINet which was an early tier 1 ISP. PSINet was eventually acquired by Cogent. Cogent currently owns the 38/8 ip space.

Apparently Cogent hasn't done a swip mapping for the particular addresses: 38.98.136.241, 38.98.136.242, 38.98.136.243 that the bot is coming from. Cogent does however run their own rwhois server that records the sub network assignment:

$ whois -h rwhois.cogentco.com -p 4321 38.98.136.241
%rwhois V-1.5:0010b0:00 rwhois.cogentco.com
38.98.136.241
network:ID:NET-266288E01B
network:Network-Name:NET-266288E01B
network:IP-Network:38.98.136.224/27
network:Org-Name:Ambiron, LLC
network:Street-Address:120 N LaSalle St Ste. 1250
network:City:Chicago
network:State:IL
network:Postal-Code:60602
network:Tech-Contact:ZC108-ARIN
network:Updated:2007-09-18 17:09:29
network:Updated-by:jknowles

This shows that the network block is assigned to "Ambiron, LLC". More googling leads to a press release on the Trustwave website announcing that in March of 2005 Trustwave and Ambiron merged. So the hits are coming from ips addresses owned by the company that is performing our PCI DSS security scans.

Nothing to see here folks. :)

I'm NOT turning into a Republican

Once you start down the dark path, forever will it dominate your destiny.

Yoda

So I was standing in line at the grocery store today waiting to buy some goodies for dinner. The customer in front of me had a huge cart of what I would mostly consider junk food: 4 half racks of soda pop, case of cup 'o noodles, sugary cereal, and a lot of frozen meal stuff. She's got some serious ink on her arms, bleached and teased hair and designer sunglasses. The clerk comments on her fancy french nails and the lady beams and explains how they took two and half hours to do. The total for her cart full of crap is $180. Scratch that, $169 and change because she's paying with her food stamp debit card.

For a moment (that I'm extending now because I'm taking the time to write about it), I was completely pissed off. The thing that set me off here wasn't that the lady obviously had money and time to put into her appearance while still on public assistance; it was the things that she was buying with "our" money. It's shallow and judgmental and in many ways none of my business, but it still rubs me the wrong way. Jones Soda and Sugar Smacks are not the building blocks of a healthy society.

Canada Two: The Revenge

If things go according to plan, my youngest brother will take over Canada sometime in the next two decades. Details are a little sketchy as to whether this will be the result a bloody revolution or if he will be welcomed with open arms following a massive ingestion of tainted poutine. The means of his ascension really doesn't matter; we know that it will happen eventually.

Once he takes power the basic plan it to set up a utopian society; a socialist meritocracy guided by a benevolent dictator. The first order of business will be to build a large wall to separate Canada II from the normals in the rest of the world. Then everyone wishing to remain inside the wall will have to take an intelligence test. Those who do not meet a minimum requirement will be ejected from the country by means of a large trebuchet. All future applicants for citizenship will be subject to the same test and penalty for failure.

The benevolent dictator part will be ensured via a lottery system. Every 24 months a call for applicants for the office of High Poobah will be made. Anyone who applies will be ejected via the trebuchet. Everyone left will be assigned to the lottery pool and the (un)lucky winner will be declared dictator for life. Obviously they will have to make the best of the two years left to them.

My only concern with the plan, other than hoping that I pass the entrance exam, is that Canada is a long way from the South Pacific. I'm going to be very torn between my dream of subsistence level farming in the tropics and hanging out with Adam.

Rubik's Revenge

I've got a Rubik's problem. Tom got me a Rubik's Revenge for my birthday. It's a 4x4x4 version of the evil cube. With 7.4E45 permutations possible for the 56 components of the cube there is a lot of twisting to do.

Interestingly, it's not too much different to solve than the 3x3x3 cube. Basically what you can do is reduce the 4x4x4 to the same problem as the 3x3x3 by solving the centers and pairing up all of the edge pieces. Once that is done, the two inner layers can be rotated in tandem and solved normally.

Solving the centers and edges only takes a handful of new algorithms. Once you solve a couple of centers that part becomes pretty easy. The first 6 edges are cake too because you have room to store them out of the way once you've matched the pieces. After that things get a little trickier.

It's sitting on my desk at work right now. I've got the cheat sheet for solving it in my "man purse." Sweet fancy Moses I am a dork.

A Major Award

Chili Champ 2008My wife and some of her friends put together a chili cookoff/chili feed party last night. The idea came from a conversation with Laura and Ben over a batch of tasty but too hot to eat comfortably chili I made a month or so ago. Five of us have January birthdays, so we decided that a chili cookoff would be a fun way to get together and celebrate.

The turnout for the party was great. There were 11 batches of chili up for judging submitted by nine cooks. The judging was a head-to-head taste test with everyone at the party voting for their top three picks. A first place vote was worth 5 points, second favorite got 3 and third got 1 point. The entries varied from cookbook recipes to custom creations. There were ground beef, beef, pork and chicken entries. Since these were eating chilis rather than professional competition recipes, every entry contained beans or rice. If I had it to do over, I think I'd leave the beans out of one of my submissions.

I made two batches of chili to enter in the contest, one was a Texas style red chili and the other was a chili verde. The green chili won the contest by a landslide. I think this was a combination of the uniqueness of the recipe among the other entries, which were all some red chili variation, and my secret weapon: a cilantro lime crema topping.

Several people asked for the recipe, so I'm going to try an record what I did to make the winning entry. I used the pinch of this and dash of that method to put it together, but I think I can still remember all of the key ingredients. If you're interested, use the instructions below and lots of tasting and tweaking to put together a batch for yourself.

Turkey Chili Verde with Cilantro Lime Crema

2008 Capricorn Chili Cookoff winner
(c) 2008 Bryan Davis some rights reserved

  • 4c Chopped Sweet Onion (2 big onions)
  • 2T Vegetable or peanut oil
  • 1t Kosher salt
  • 1t Ground coriander seed
  • 1t Ground cumin
  • 1t Ground thyme
  • 1t Garlic powder
  • 1T Chili powder (homemade is best)
  • 2t Crushed chili flake (green if you can find it)
  • Fresh ground black pepper
  • 1 - 1 1/2lbs Dark turkey meat (I used legs, see notes below)
  • 2t Kosher salt
  • 4T All-purpose flour
  • 1t Ground coriander seed
  • 1t Ground cumin
  • 1t Ground thyme
  • 2t Garlic powder
  • 1T Chili powder (homemade is best)
  • 2t Crushed chili flake (green if you can find it)
  • Fresh ground black pepper
  • 2T Vegetable or peanut oil
  • 2 cans Whole tomatillos (or 10-12 fresh tomatillos if you can find them)
  • 2 cans mild green chilis
  • 1T Brown sugar
  • Vegetable or poultry stock
  • Small bunch of cilantro
  • 1 Lime
  • 1c Sour cream, plain yogurt or crème fraîche
  • 2 cans white beans

Saute onions in oil in a heavy sauce pan with salt, coriander, cumin, thyme, garlic powder, chili powder, crushed chili flake and a few cracks of fresh ground pepper. Cook onions until they start to take some color and about half the water has cooked out of them. Dump the cooked onions into a crockpot to wait for the rest of the goodies.

Chop turkey into bite size pieces. I used turkey legs because I wanted a lot of connective tissue for the deliciousness that a long stewing can give. If you use legs, be prepared for a lot of knife work. You need to bone out the leg to get the meat. A turkey leg has 7 or 8 large tendons that will need to be removed and several tough layers of connective tissue. I think you could substitute turkey thighs or chicken thighs instead if you want to save some time. Don't use white meat though because it won't survive the stewing process; you'll just end up with a bunch of turkey dust in your chili. Once the turkey is chopped, salt liberally.

Mix the second batch of spices and the flour and toss with the turkey to coat. Add some oil to the same sauce pan you cooked the onions in and cook the turkey in batches until lightly browned (4 or 5 minutes). Cook in batches so you get frying and browning instead of stewing or steaming of the meat. Dump each cooked batch in the crockpot with the onions.

Dice the tomatillos and add to the crockpot. Add the cans of chilis and the brown sugar as well. Give the pot a good stir and add some stock if it doesn't look moist enough. The onions still have a lot of water in them that will cook out as everything stews in the crockpot, so don't go too crazy. The contents should look moist, but not be standing in liquid.

Turn on the crockpot, put on the lid and take a break. Very the temperature of the pot depending on how long you have to cook it. Leave it on low if your going to let it stew all day. If you want to eat in a couple of hours, turn it up higher. The slower and longer you leave it to cook the better the turkey will taste.

When you've caught up on your tv watching, make the crema. Pick the leaves off of the cilantro stems and finely chop them. Don't throw away the stems because we're going to use them in a minute. Add the cilantro to the sour cream in a small bowl. Zest the lime and add that as well. Finally juice the lime and add that to the sour cream mixture. Stir well and season with kosher salt and fresh ground black pepper to taste. When I make it, it comes out looking like a loose spinach dip.

Very finely chop the cilantro stems you reserved during the crema step. Add these to the slow cooker about an hour before you want to serve the chili. Rise, drain and add the beans at this point too if you're using them.

Taste your chili at this point and add any of the seasonings to balance the flavor. The hardest part for me is putting in enough salt. Tune the spices in small increments, because you can go from bland to overwhelming really quickly, especially with the chili powder and flake.

Writer's Block

I actually though I had several ideas for blog posts when I started this thing, but it turns out that I don't drink enough to be a good writer. Maybe I'll start drinking more.

Knowing vs Thinking

A lot of the blogs that I read are about programming and project management. They are predominantly written by people that have a lot of online credibility as "experts" in their field. It dawned on me recently, that although I don't have the interwebz cred that they do, I actually have as much or more real world programming and project management experience as most of these "experts".

The funny thing about getting old is you don't realize it.

Dolly Parton (and I'm sure a lot of other people but that's the google hit I got for the phrase)

Anyway, if being 30+ years old and having worked as a professional programmer for 10+ years qualifies some other guys to pontificate about the business of programming and hit the front page of digg for doing it, I might as well do some of it too.

"I think ..." is a phrase that strikes fear into my heart when uttered during discussions about debugging, troubleshoting or project status. I think means that the speaker is not confident enough to say I know. I know means that it is a dead certainty that the following statement is fact. It should mean that the statement has been verified before sharing it with the team. I think means that something is being vaguely recalled or assumed based on history, pride or prejudice.

I heard the dreaded I think phrase three times at work today. Two out of the three times the speaker was soon proven wrong via empirical research. That in and of itself is not a big deal. People are wrong all the time; it's to be expected actually in the software world. The dangerous thing about I think is that it costs valuable time. A potential failure mode has been identified and the question has been asked if it could be the true cause of the current problem. By thinking this is not the problem it is discarded as a root cause and other avenues are searched. Unless someone else forcefully believes that the cause which is now thought by the group to be disproved it will not be revisited until many other options have been exhausted.

In my experience, it usually only takes a few minutes to know something instead of just think it. These few minutes spent upfront can be critical when an emergent problem is being dealt with. Troubleshooting (at least good troubleshooting) is the process of running through an n-ary tree with a depth first spanning algorithm. The early decisions prune large portions of the search space and if a partition is discarded falsely, it will take great effort to resurrect it.

I'm going to write another post at some point about how I approach building and ordering the decision tree because order is crucial for efficient operation, but the point I'd like to get across today is that each hypothesis needs to be tested in realtime. Know that it is true or false and you will save a lot of time and confusion in the long run. When the issue at hand is a loss of several thousand dollars of corporate income per minute it can make the difference between being fired and getting the biggest bonus of your life. If you approach every problem as thought it were that important you will become a better programmer/sys admin/manager/mechanic/whatever.

Obsession of the Moment: Rubiks Cube

The new product that I've been working on for longer than I really care to admit is getting ready for it's big public debut at a trade show in a few weeks. As a part of the standard marketing hype, tptb have put the product's logo on a lot of random plastic things to hand out to people who have been sent to walk the show floor like mindless zombies. Most of the swag is bog standard, but one k3wl thing was created: a real live Rubik's Cube with the product logo on it.

Like most everyone alive in Amerika during the 80's, I had one or two of these before. I was more mechanical than mathematical as a kid, so instead of learning to solve the cube I learned to take it apart. I was wicked good at taking it apart too. I felt like I'd lost serious geek cred though when people at the office found out that I didn't know how to solve the cube. I was sort of redeemed when no one else seemed to be able to solve it either, but I think that some of them were faking.

It actually turns out that Joey can solve the thing and can be tricked into doing it in Rain Man fashion by merely leaving a scrambled one on his desk. This goaded me into action. A real geek must be able to show mastery over cheap plastic puzzles. Several google searches later I had read at least twenty "how to solve the cube" tutorials that ranged from memorize these transforms to seriously hardcore number theory.

I'm going with the "memorize these transforms" method. I can do two layers plus orienting the edges on the last face without thinking much so far.

Postscript: I don't know how long it will stick in my head, but I'm solving the cube repeatably in 5 to 7 minutes now without using my cheat sheet. The patterns are starting to make sense too. -- 2007-01-12

Inception

Joey told me I should start a blog, so I'm going to give it a try. Chances are pretty good that I will not keep up with it after a few weeks (or maybe even days), but I suppose I can't be a "real" computer guy unless I have the ego to think that everyone in teh interwebz should be forced to read my random rants.