Chili Champ No More

I only managed to place 2^nd in the 3^rd annual Capricorn Chili Cookoff last night. Thanks to Terri and Patrick for hosting and everyone who showed up to compete and judge. As always it was a lot of fun.

Up Front Chili

2010 Capricorn Chili Cookoff runner-up
(c) 2010 Bryan Davis some rights reserved

• 3lbs Beef chuck/shoulder cut in 1/2" cubes
• 1.5lbs Beef brisket (trimmed)
• kosher salt

• 4 red bell peppers (seeded, cored and halved)
• 6 Anaheim peppers (seeded, cored and halved)
• Kosher salt
• Fresh cracked black pepper
• Vegetable or peanut oil

• 3T Chili powder (preferably homemade)
• 1t Cumin
• 1t Coriander
• 1.5t Garlic powder

• 3 Sweet onions
• 10oz Mexican chorizo (beef or pork)

• 12oz Beer (light ale or good quality lager)
• 56oz Whole peeled tomatoes (drained)
• 4oz Tomato paste
• 3T Dark brown sugar
• 3 Bay leaves
• 10 Sprigs fresh thyme

• 2oz Baker's chocolate (100% coco)
• Beef stock

Salt meat liberally and let set at room temperature for a half and hour or so to draw out juices.

Heat oven to 350°. Place red and anaheim peppers cut side down on a foil lined baking sheet. Drizzle with oil and salt lightly. Roast peppers in oven for 20-30 minutes until softened. Remove from oven and puree in food processor.

Toast chili powder and other spices in a small dry pan to activate oils and bring out flavors.

Blot meat with a paper towel to remove excess moisture that the salt has drawn out.

Coat one side of brisket with half of pureed peppers. Roll and tie with kitchen twine. Season outside with salt and pepper. Bake in 350° oven for 1 hour on foil lined baking sheet. Remove from oven, wrap in foil and return to 300° oven for another 1 to 2 hours. Remove from oven and let rest in foil until called for later.

Season chuck with fresh cracked black pepper and 1T toasted spice mixture. Heat enough oil for sauteing in large cast iron casserole or dutch oven until shimmering. Brown meat on all sides in batches removing browned meat to a bowl to be used later.

Puree onions in food processor until coarse paste. Drain oil from casserole and render chorizo. Add onion and remaining peppers to pan. Season with salt, black pepper and 1T spice mixture. Brown until they are very dark and a crud has formed on the bottom of the pan, approximately 5 to 7 minutes. Scrape the crud and let it reform. Brown and scrape for a third time.

Add tomato paste, brown sugar, beer and browned chuck. Hand crush half of the tomatoes into the pot. Add thyme and bay leaves tied in cheese cloth for easy removal. Simmer for 2 hours.

Chop chocolate and add to chili. Crush remaining tomatoes into pot. Cube cooled brisket and fold into mixture. Adjust to taste with remaining spice mixture and salt and use beef stock to adjust consistency. Simmer for another 30 minutes to blend all flavors.

Serve with Queso Fresco, tortillas and cold beer to a large gathering of hungry friends.

Chili Reign Continues

The second annual Capricorn Chili Cookoff was last night. I managed to win 1^st place again this year, but it was much closer than last time. The 1^st, 2^nd and 3^rd place winners were only separated by 4 votes. Thanks to everyone who voted for me and congratulations to all who participated.

Pork Rock Chili

2009 Capricorn Chili Cookoff winner
(c) 2009 Bryan Davis some rights reserved

• 4lbs Pork shoulder/butt cut into bite-size cubes
• Kosher salt
• Fresh ground black pepper
• Vegetable or peanut oil

• 2T Chili powder (preferably homemade)
• 1T Chili flake (preferably homemade)
• 1t Cumin
• 1t Coriander
• 1t Paprika
• 1.5t Garlic powder
• 1t Onion powder

• 4 Anaheim peppers, cut into 1/2-inch pieces
• 2 Poblano peppers, cut into 1/2-inch pieces
• 2 large Sweet onions, cut into 1/2-inch pieces
• 1 Carrot, peeled, cut in 1/2 lengthwise, then cut into 1/2-inch pieces
• 2 Celery ribs, cut into 1/2-inch pieces

• 12oz Beer (light ale or good quality lager)
• 56oz Crushed tomatoes
• 3 Chipotle peppers in adobo, chopped
• 2T Honey
• 1.5T Dark brown sugar

• 1 bunch Cilantro
• 45oz Cannellini beans (optional)

Salt the pork liberally and let set at room temperature for a half and hour or so.

Toast chili powder, chili flake and other spices in a small dry pan to activate oils and bring out flavors.

Blot pork with a paper towel to remove excess moisture that the salt has drawn out. Season with fresh pepper and 1T toasted spice mixture. Brown 1/4 of pork with 1T oil in large cast iron casserole or dutch oven over high heat. Remove browned meat to a bowl and repeat.

Rough chop peppers, onion, carrot and celery. Puree all the vegetables in a food processor until it forms a coarse paste. Coat the bottom of same pan with fresh oil and add the pureed vegetables. Season the vegetables generously with salt and 1T spice mixture and brown until they are very dark and a crud has formed on the bottom of the pan, approximately 5 to 7 minutes. Scrape the crud and let it reform. Add the beer and half of the tomatoes and scrape the bottom of the pan. Lower the heat if things start to burn. Reduce the mixture by half.

Add chipotle, honey, brown sugar, 1T of spices and your pork to the pan. Simmer for 2 hours.

Pick the leaves from the cilantro and reserve. Chop the stems finely and add to pan along with remaining tomatoes. Taste mixture and add salt and spice mixture as desired. Simmer for another hour until pork is tender but not falling apart. If you want beans in your chili, add at this point as well.

Just before serving, chop the cilantro leaves finely and add 2T to the chili. Serve with grated monterey jack cheese, chopped cilantro, sour cream and your favorite corn bread to 8 to 10 of your hungry family and friends.

Debugging for Fun and Profit

I tend to call any troubleshooting process "debugging." A week or so ago I debugged our lawnmower when it wouldn't start. Today I spent a lot of time (unsuccessfully) debugging a mysterious apache child crash related to a php application. Tomorrow I will undoubtedly debug some software at work and some other things in "real life."

I would guess that roughly 40% of my life as a professional programmer and system administrator has been spent debugging. By that I don't mean to imply that I only write broken code (although I probably do); what I mean is that like most writing is rewriting, most programming is debugging. A first attempt at solving a problem rarely works correctly. If it was that easy to solve it must not have been much of a problem in the first place right?

For me debugging software/hardware/whatever is mostly about thinking of the most common things that could be causing the malfunction and then systematically proving that these things are or are not the cause of the current problem. When my dad taught me how to keep our crusty old lawnmower running when I was a kid he gave me the checklist "fuel, fire, air." This magic formula has pointed to the cause of 90% of the problems I have ever had with engines.

• Does it have gas in the tank?
• Is the gas getting to the cylinders?
• Is the battery/magneto/alternator working?
• Is the charge making it past the points/distributor?
• Is the spark plug fouled?
• Is the air filter clogged?
• Is the choke stuck closed?

The order here is important too. On each branch of the decision tree we start at one end of the subsystem and walk towards the other end checking the key weaknesses in between. The branches are also arranged in order of most frequent failure. All this ordering hopefully get you to the root problem earlier rather than latter. You need to tailor the tree based on prior experience. When working with the beater Datsun I drove in college, fire came before fuel because it had constant alternator and distributor problems.

Software debugging works just like the mechanical troubleshooting process for a malfunctioning engine. The only difference is the decision tree. If I'm working on a network connectivity problem the quick checklist usually goes something like "interface, firewall, routing."

• Is the physical link in place?
• Is the interface active?
• Is there a local firewall block?
• Is there an upstream firewall block?
• Is there an outbound route?
• Is there an outbound route from the next hop?

Just like the lawnmower to car differences, this list gets modified based on what I'm working on. This order worked best for border connectivity issues on the frame relay networks I used to work with. The network I'm on today calls for the firewall checking first because 50% of the time that's where the problem is.

So how do you construct this sort of list when you run into a problem you've never seen before? I tend to use first principles or cartesian doubt technique. Based on the nature of the failure you are seeing and the knowledge you have of the system, think of everything that could have gone wrong. Start with a likely cause and follow it out until you are satisfied that particular subsystem is working. Move on to the next and repeat. If the problem is in code that you (or someone you know) wrote think about the things that are typical problems in your code. Do you often make fencepost errors? Are you working in a language that is prone to API mistakes like calling a method with the arguments in the wrong order? Are you working in a threaded environment and experiencing a concurrency issue?

I wish I had something better to say at the end of all this, but maybe some part of it is non-obvious: think about what could be broken, verify that it's not broken, repeat. Sounds pretty simple when I put it like that, but there's some magic in there that I can't quite elucidate.

What I Read On My Summer (OK Spring) Vacation

• The Alibi Club, Francine Mathews
• American Gods, Neil Gaiman
• Beautiful Code: Leading Programmers Explain How They Think, Andy Oram, et al
• Beggars Ride, Nancy Kress
• The Big Knockover, Dashell Hammett
• Blue Latitudes, Tony Horwitz
• Dark Mission: The Secret History of NASA, Richard C. Hoagland & Mike Bara
• The Diamond Age, Neal Stephenson
• Friday, Robert A. Heinlein
• The Hacker and the Ants, Rudy Rucker
• Little Brother, Cory Doctorow
• Nebula Award Stories Four, Poul Anderson editor
• On Intelligence, Jeff Hawkins w/Sandra Blakeslee
• The Secret Agent, Joseph Conrad
• Snow Crash, Neal Stephenson
• White Light, Rudy Rucker

There's no central theme, but it turned out that there were a lot of books dealing with themes of computer and human intelligence. There was also a lot of just plain good reading and a handful of crazy crackpot tinfoil hat thrown in for good measure.

The Stephenson novels were re-reads of two of my all time favorite books. I'd love to work on a project to make something like the Young Lady's Illustrated Primer.

Mystery IP Address

A couple of months ago Steve noticed a lot of hits to the new company website from three IP addresses in the 38/8 block. By a lot of hits I mean something like 75% of the total traffic. The MaxMind database we use to generate region and ISP information for the traffic reports said that these addresses belonged to "Performance Systems International". Steve googled for this company and came up with a whole lot of blog and forum posts that claimed this company was home to a rogue bot that has plagued a lot of people.

I decide to dig a little deeper because most of the links that Steve sent me seemed to be a little under informed. Here's what I found:

"Performance Systems International" is the original name of PSINet which was an early tier 1 ISP. PSINet was eventually acquired by Cogent. Cogent currently owns the 38/8 ip space.

Apparently Cogent hasn't done a swip mapping for the particular addresses: 38.98.136.241, 38.98.136.242, 38.98.136.243 that the bot is coming from. Cogent does however run their own rwhois server that records the sub network assignment:

$ whois -h rwhois.cogentco.com -p 4321 38.98.136.241
%rwhois V-1.5:0010b0:00 rwhois.cogentco.com
38.98.136.241
network:ID:NET-266288E01B
network:Network-Name:NET-266288E01B
network:IP-Network:38.98.136.224/27
network:Org-Name:Ambiron, LLC
network:Street-Address:120 N LaSalle St Ste. 1250
network:City:Chicago
network:State:IL
network:Postal-Code:60602
network:Tech-Contact:ZC108-ARIN
network:Updated:2007-09-18 17:09:29
network:Updated-by:jknowles

This shows that the network block is assigned to "Ambiron, LLC". More googling leads to a press release on the Trustwave website announcing that in March of 2005 Trustwave and Ambiron merged. So the hits are coming from ips addresses owned by the company that is performing our PCI DSS security scans.

Nothing to see here folks. :)