Read on Twitter
This is the story of how I fixed a long-standing bug in one of the U.S.'s systems for detecting nuclear explosions, the ARDU (Advanced Radiation detection capability Data Unit) 1/28
The ARDU system processed data from sensors on 24 GPS satellites that detected various particles (photons, protons, etc.) and decided whether they came from a nuclear explosion https://www.globalsecurity.org/space/systems/nds.htm 2/28
Note that the sun is a continuous thermonuclear explosion, so the ARDU had a lot of code to track where the Sun and its reflection off the ocean were to avoid accidentally starting WW III several times a day 3/28
The software for ARDU was written in Ada 95 by scientists in the early 90's. Some of the code looked like assembly, some like a series of logic truth tables, and much of it had zero comments 4/28
Everyone who had originally written the software had moved on, leaving the maintenance team a collection of videotaped lectures and a binder of HANDWRITTEN (!!!) design documents 5/28
The ARDU had been running for 5 years when I joined the software maintenance team in 1999, straight out of university with a double B.S. in computer science and mathematics 6/28
So what was the bug? One of the satellite sensors detected bright lights. It needed to be calibrated by being shot at by a giant laser (!!!) in the mountains near Albuquerque https://en.wikipedia.org/wiki/Starfire_Optical_Range 7/28
The sensor on the satellite would detect the laser light and send raw sensor data back to the computer on the ground, which calculated the location of the laser and compared that to its known location 8/28
Then the difference between the two locations would be used to correct all the other reports from that sensor. But there was a bug in the code that calculated the location of the giant laser 9/28
The code always returned a location that was way, way off the real location of the giant laser... but in a predictable way. No one could figure out where the bug was in the ARDU code 10/28
So one of the programmers wrote a separate FORTRAN program that calculated the location correctly from the raw sensor data. It had to be run by hand and was super annoying during Y2K testing 11/28
I couldn't do any real work until I got my security clearance and was bored out of my mind. So when I finally got my clearance 3 months after starting, I jumped on this weird calibration bug 12/28
To get the coordinates of the giant laser shooting at it, the satellite recorded where the light hit a spinning disc inside the satellite, which was turned into an [x,y] coordinate pair 13/28
The [x,y] coordinates, the position of the spinning disc, and the satellite's position defined a vector which was intersected with the surface of the earth to get the latitude and longitude of the giant laser 14/28
Transformation matrices are how you convert between different coordinate systems, so to get from the spinning disc to the earth's surface, there were, like, eleventy transformation matrices http://jsoc.stanford.edu/~jsoc/keywords/Chris_Russel/Geophysical%20Coordinate%20Transformations.htm 15/28
I noticed that the output of the buggy code seemed as if it had at some point accidentally flipped an [x,y] pair across a diagonal line instead of rotating the [x,y] pair around the origin 16/28
This kind of flip could happen if the code accidentally applied a reflection matrix instead of a rotation matrix, which look very similar https://en.wikipedia.org/wiki/Rotations_and_reflections_in_two_dimensions 17/28
So I wrote out all the transformation matrices from the ARDU code on the whiteboard in the classified vault and laboriously worked my way through what each one did by hand 18/28
Eventually I found the bug! One of the transformation matrices had sin() and cos() swapped. The result was some wonky kind of reflection instead of what it should have been, a rotation 19/28
I fixed the transformation matrix code so it did the rotation correctly, and it WORKED!!! I was so proud I fixed a bug that had baffled every other programmer working on this project for 5+ years 20/28
I don't think I was smarter than the other programmers, I think I was just (a) very bored, and (b) had learned linear algebra recently enough that I remembered how to multiply matrices by hand 21/28
I was so happy I could contribute as a baby programmer. And then I showed up to work on January 2, 2000 and my key card didn't work. Two subcontractors were having a fight and therefore I no longer had a job ¯\\_(ツ)_/¯ 22/28
Luckily for me, a week later I landed my first Linux kernel development job. But I never forgot how good it felt to fix that math bug for nuclear explosion detecting satellites and it is still on my résumé 20 years later 23/28
I wrote this from memory about something that happened 20 years ago, so several details are probably wrong, for which I apologize to the other ARDU programmers and mathematics in general 24/28
All this is totally unclassified, by the way - the only part of the ARDU that was classified was the test data simulating nuclear war scenarios, which were based on classified information about nuclear capability 25/28
You can read more about ARDU, NUDET, and Starfire Optical Range (giant lasers!!!) here: https://www.globalsecurity.org/space/systems/nds.htm https://en.wikipedia.org/wiki/Starfire_Optical_Range 26/28
I have lots more stories about being an open source programmer fresh out of university working for a military government contractor but they don't involve detecting nuclear explosions so they aren't as fun :) Thanks for reading! 27/28
And here is the Google Doc version of this thread about fixing a math bug on the system that uses satellites to detect nuclear explosions: https://docs.google.com/document/d/1pyGft-OXrSw1ZPUMZCE0mxG92_xqtzgdanHwkKLpxUQ/edit?usp=sharing 28/28
