This week I read the story about how Iran hacked and captured one of our most sophisticated military drones. It struck me that this was an excellent example of the potentially disastrous ramifications of ignoring technical debt. It appears that the potential to spoof GPS was known and ignored for many years. Acknowledging and managing technical debt is an issue that separates the whiz kids from the graybeards in the tech industry. Let me start by saying that a military boof-a-rama of this magnitude is, by necessity, going to be followed up with multiple misinformation campaigns so I doubt we’ll really know what really happened for a few decades. Nevertheless it provides a teachable moment for us so let’s explore this topic.
Technical debt is analogous to financial debt so let’s start there. Imagine you are a fresh college graduate with a new job and a newly minted credit card. Thanks to the miracle of modern financial Svengalism, you can buy things you don’t have the money to pay for. You use the card to get set up in a new apartment and at the end of the month you have a choice, you can pay off the debt or you can pay the minimal possible. Debt is not a problem if no one makes you pay attention to it. And there’s the rub, at some point you reach your limit and the party comes to an abrupt halt. Not only do you have to start paying off your debt but you have accumulated a bunch of interest that needs to be paid off as well. At this point, you are screwed. You can’t buy new stuff and you have to spend all your effort trying to pay off a mountain of debt. You pay and pay and pay but for all your effort and money going out, you have no new stuff to show for it – you are managing your financial debt.
The same thing happens with technology. This is most clearly seen in startup companies but the same thing happens in most technology companies. At a tech startup, the most important thing to do is to get the right set of functions in the market as quickly as possible so you can start bringing money in. What this means is that you are often cutting corners – skipping documentation, ignoring supportability, deferring security work, doing quick and dirty designs vs. deep architectural work. This might be a reasonable strategy for a startup because once you get the product out, you might get feedback that you need to go in an entirely different direction. If that is the case, you can “declare bankruptcy” on your existing code, walk away from those debts and start on a new path. On the other hand, if you stick with that code, you have a technical debt which is going to have to be paid. In your next development cycle, you can either begin to pay down that technical debt or you can devote 100% of your resources on the next round of quick and dirty functions. Deciding when to start paying off your technical debt is what separates the whiz kids from the graybeards in the tech industry.
I’m sure most of us at some point have found ourselves on a project that was led by whiz kids that had completely ignored their technical debt and was now in shambles. I was hired at DEC to lead the turnaround of just such a project. The product was reasonably successful but the implementation “had some issues”. When I got onboard and did an assessment of where we were I was shocked – the product was a cobbled together hack using 18 (yes 18!) different languages. There was even some ADA code in there. I joked with one of the developers on the project saying, “How did that get in here? Did someone take a night course can decide to ship his homework in the product?” Imagine my horror when then answer came back, “Yes – that is exactly what happened”. When we explained where the code base stood and what was required to move forward, the management team cancelled the project and decided to just release it with a few new bug fixes and features. The interesting aspect of this story is that the whiz kids responsible for this mess were viewed very positively – they got things done and delivered on time and they got promoted and moved on to go screw up bigger and better projects. This is an example of where ignoring technical debt lead to the death of a project. Often the results are less dramatic, you’ll get a new version of the product with fewer new features. Sometimes the results of ignoring technical debt are much more dramatic. Like our drone.
Drone technology is the darling of the US military/industry/political parties. They provide a tireless presence on the field of battle, they can project force without risking the political ramifications of putting US personnel in mortal danger, and they generate new military contracts for the latest and greatest technology. The original drones were relatively simple and effective monitoring devices. They were great but had a lot of well understood problems. The data they streamed back was unencrypted so anyone could grab the signals and see what we saw as well as get detailed operational information about what we looked for and how we looked for it as well as how we operated the drones. The designers were able to anticipate some problems and attempted to address them. This provides us a great example of where you have to be just as vigilant about a mitigation as you are about the problem it tries to mitigate. The designers understood that there were various hostile and non-hostile situations where communications between the drone and the controllers would be severed. They addressed this by programming the drone to detect the situation and return home using GPS and a preprogrammed set of coordinates for the home base. And here is where the technical debt problem comes in.
Apparently the military has known about the potential for GPS to be spoofed for 20-30 years. This is a technical debt. The Christian Science Monitor reports that what Iran did was to jam communications with the drone so it engaged it’s “fly home” program and then spoofed the GPS signal to make it land in Iran. The Iranians now have some of our most sophisticated military technology because we failed to address our technical debt.
When we decided to go-big on drones and transform them from a marginalized science project to the sharp point of the US military spear we faced a choice: we could either address our technical debt or we could add bells and missiles. It appears that we decided not to address our technical debt (or not to address it in time to avoid this mess). After all, where is the excitement in making things work? I’m sure some argued that, “They do work, they just don’t work as securely we’d like and we don’t know that others can actually spoof it”. Given a fixed budget, any money spent on addressing technical debt is money not spend adding new bells and missiles. To ship is to choose. And that is exactly the point – it is what separates the whiz kids from the graybeards.
Whiz kids will ignore technical debts, putting 100% of the budget on new features. They build houses of cards on foundations of sand and then move on before the waves come. They come off looking like superstars and then move on to screw up something bigger. As graybeards (and architects) we need to be the guardians of the future. We need to understand our debts and be vigilant in addressing them even though there is no clear and immediate benefit for doing so. This is easier said than done. It is easy when we are in charge and just set the rules. It gets harder when we don’t own those decisions and need to advocate for them and fight for budget. Few people are interested in hearing it so it takes courage to stand up and be a butthead to ensure the right things happen. Some deal with it “off the books” by padding schedules with a fixed amount of overhead to address the issues. One way of the other, it needs to happen or you end up with a disaster. You can only avoid problems for so long and then you have to address them.