mike@coldandhungry.org Never Say Die! - The Vancouver Diaries
What's This?

Never Say Die!

I am the best there is at debugging.

Michael David Crawford
March 12, 2007

Copyright © 2007 Michael David Crawford. All Rights Reserved.

Tonight is the night that I fix the bug I've been assigned for several days. I refuse to take "No" for an answer.

I told my project manager a while back that I was particularly good at debugging, and proved it by finding a bug in Apple's code that caused our product to fail. I used a combination of message logging, with the gdb debugger debugging a userspace demon on the test machine, and gdb debugging the OS X kernel over ethernet from a second machine.

Tonight I'll be using all those tools, plus the tcpdump protocol analyzer - that's because the product I'm working on is a networking application.

I've been working on this bug for several days (and nights). It's a little before nine PM as I write this; I'm prepared to stay the night and work through tomorrow too if I have to.

I can't tell you what the application is, but I can describe the bug in general terms: it's hard to reproduce, but if you manipulate the user interface for a while, eventually the cursor will start spinning and our UI will become non-responsive. (This is known to Mac users as "The Spinning Pizza Of Death" or "The Spinning Beach Ball Of Doom".)

The Mac isn't crashed, but our product is hung; OS X spins the cursor whenever a GUI program stops processing events.

So far all I've been able to determine is that our UI makes a call, via the kernel, to our device driver, and the device driver never returns. I'm not sure whether it's a problem with the device driver or if it's a problem with the server it's talking to over the network.

(Strictly speaking, our driver is a protocol driver, not a device driver. But both kinds of drivers are implemented in Mac OS X as I/O Kit kernel extensions, or "kexts".)

Not all of Mac OS X is Open Source, but large portions of it are, enough to make device driver debugging a lot easier than in the Classic Mac OS days when the System was completely closed. You can download what sources that are open from Apple's Open Source page. There is also the new Mac OS Forge, which I don't know much about yet.

I got to be such a wizard at debugging when I worked at Working Software, where a number of our products were implemented as operating system patches; I figured out how to patch the Classic System using the MacsBug assembly debugger.

What's This?