Richard Feynman is one of my heroes, as much for his brilliance and creativity as for his sense of humor, his matter-of-fact view of the world, and his unassuming and, well, human personality. Years ago, as I read his autobiography, Surely You're Joking, Mr. Feynman, I couldn't help thinking I would have loved to work with him. Maybe, just maybe, I might have absorbed some of the Feynman genius (or at least some of the Feynman humor) through osmosis. But I also wonder if he'd approve of my curiosity and quirky thoughts, or if he'd just shake his head and wonder if I'd ever get it right. The first chapter of his autobiography is titled "He Fixes Radios by Thinking!" But my version, one of the highlights of my career, might've been called "He Fixed a Critical Product Flaw by Accident."
I was an engineer at a tiny company, building a family of devices that still make me proud. Before most people had even heard of the Internet, our appliances let our customers connect their computers over the telephone or a network to their remote electronic equipment: the PBXes that ran their telephone systems, their network switches and routers, their alarm systems, you name it. If you could talk to it electronically, we'd figure out how to connect you to it. Better still, we could program our devices to monitor that equipment, diagnose any faults, and page a technician if we couldn't fix the problem automatically. Then the technician could access our device with his computer, connect through it to his own equipment, and fix it without leaving home!
Of course, we didn't want to open the proverbial barn door to anyone with a computer, so we built some heavy-duty security into all of our products, many of which are still considered state-of-the-art. The details were too complicated for our average customer, so we tucked them inside a fairly simple, friendly interface. So simple, in fact, that we often taught our computer-savvy customers how to write their own programs to make our devices do almost anything they could want. Seriously: anything that involved communication, monitoring, and alarming. One customer bolted them to cell phone towers. Why? You've seen the flashing red lights on top of those towers, the ones that warn pilots to keep their aircraft a safe distance away? Well, this customer wanted to monitor the lights, start a diesel generator if the power failed, and page a technician any time a light stopped working. We did that.
One particular training was held at our office during the holiday season. I sat in, partly to meet the customers, partly to see how our products were used "for real," and partly because it's an absolute treat to be in one of Mike's classes. One of the company's founders, Mike is among the smartest and most entertaining people on the planet. He's done engineering, programming, training, tech support, tech writing, sales, ... and there are only a handful of people who can do any one of those jobs as well as he did. He also has a wonderful sense of play and humor.
To start the class, Mike showed us how to write a program to turn the lights on the device on and off. First we turned them all on, then all off. Then we lit them in sequence, in pairs, and in reverse order. Since it was close to the holidays (and because we wanted to demonstrate our random number generator), he suggested we flash them randomly, like the lights on a Christmas tree.
That was when I unwittingly started channeling Feynman.
I asked innocently, "What does random mean?" Which sparked a rather animated discussion: one customer suggested that each light should turn on and off the same number of times as every other one. "If we flash the first light 3 times, then the second one 3 times, and so on, is that random?" We agreed there's something more to randomness, but after a few minutes we still hadn't satisfied ourselves with a working definition. Seizing what to this day has still been my only opportunity to use the math I'd learned in graduate school, I explained that a sequence is random if no part of it "looks like" any other part, and that we can actually measure randomness with a function called an autocorrelation. (If you follow the link, don't let the fancy math symbols fool you: it's not nearly that complicated. You just write the sequence, then write it again starting with the second number. Then you multiply the pairs of numbers together and take their average. Repeat the process starting with the third number, and then the fourth, and so on, shifting it by one more number each time. The lower the averages are, the "more random" the sequence.)
That's when Mike told us that our devices actually had two random number generators. The first is your ordinary, run-of-the-mill random number generator. The second starts with an "ordinary" random number and then encrypts it, the idea being that the encryption makes it "more random." But we couldn't verify the theory because no one knew how to test it. Until now. I ran back to my desk during an aptly timed break and returned shortly with two graphs. And, indeed, the encrypted random number generator produced an autocorrelation with averages that were far lower than the "normal" generator. Except for a single point on the graph, which spiked far too high. Hmm.... That can't be right.
What's that? Mike asked me.
I ran it several times, I answered, and the spike is always there. That's a bug in our code.
Really? Where?
I have no idea. But I'll let you know before the end of the class.
Sure enough, after a bit of digging, I found a variable in one of the encryption routines that hadn't been initialized properly. Instead of starting off at 0, it was left with whatever value happened to be in its particular location in memory when the program ran. Mike asked me what effect the bug would have on the operation of the device. I told him that, since the routine ran only when a user logged in, and the variable could hold 256 possible values, then one time in 256 you'd type the correct password and be denied access.
Mike told me that, over the course of a decade, he was absolutely certain it had happened to him twice.
I promised that it would never happen again.
