February 2, 2004
(Computerworld)
How well would you do in a negotiation with a foreign vendor if the representative didn't speak English and your translator was secretly working for a competitor? How would you ever find out? And how would you investigate, even if you did suspect it? We faced this terrible problem recently, but in our case, the translators were programs, not people.
Our software translates stored data into the business information that we use to make decisions and execute trades. If we can't trust that software, then we can't trust what it's telling us about our data. And if we mistakenly trust that data, we'll lose a lot of money very quickly.
We have to place a lot of trust in the staff that writes our applications. We also do a lot of code testing, but that's mostly aimed at finding errors, not deliberately hidden malicious code. Such embedded code might be leaking data to competitors or adjusting financial reports so that the perpetrator can rip us off without being detected.
Luckily, we have sharp audit teams, we force our development teams to use development tool kits, and we conduct code reviews to make sure there are no surprises in our code's quality.
But what if we can't trust our tool kits? What if, after all the manual review and the checking of our custom code, the tool kit goes ahead and installs malicious code in our applications? That was the question I faced when our antivirus software started shouting about us having a "backdoor" program on key production servers.
Specifically, the virus checker found a Dynamic Link Library (DLL) file on most of our servers that contained the signature of a relatively new backdoor tool.
Could some development staffer have sneaked a back door into our code? We immediately shipped copies of our code to antivirus vendors for analysis. Perhaps this was a false alarm. Or perhaps some code combination was setting off the antivirus alert without actually being malicious.
But our hopes were dashed when the vendors confirmed that the code was a match. The code itself consists of a series of hooks that an attacker could use to hide files from the operating system or to collect passwords by intercepting the calls that legitimately ask for them from users.
At first, the finger of suspicion pointed squarely at the development group. Using the new antivirus signatures, we began sweeping across programmers' desktops. The infected files were everywhere -- even on the release CDs from our tool kit provider. Oh.
If the backdoor files were on the CD, then our staff was not the cause of the problem. By using the tool kit, our programmers had inadvertently created infected files with each new application.
But my relief that our own staff wasn't to blame gave way to a couple of more horrible thoughts: Had an attacker figured out a way around the defenses of not just my company but every other financial services company that uses this tool kit? Indeed, why target a well-protected financial services company when you can attack a tool kit distribution they all use?
The New Dilemma
After some frantic calls to the tool kit vendor, we tracked down the senior technical staffers. The code in question wasn't a back door, they explained, but a series of tools used for debugging so that application events could be caught and modified to let testing happen without changing the operating system.
This sounded reasonable. But why did the antivirus vendors claim it was a threat if this code was used only for harmless testing? A more junior technical chap replied in a blase manner that he was friendly with a hacker group and that they had used the same tool kit we had bought. But instead of hooking into the calls to debug and test code, they had produced a famous backdoor program.
I was at a loss. Should I trust the vendor's assurances that the tool kit wasn't malicious, even though the code was produced by a staffer who exchanged tools with a black-hat hacker group? How could we be sure that the hackers wouldn't target the vendor of the tool kit to get into the legitimate companies that had bought it?
On the other hand, we had no evidence that anything malicious had happened -- but how would we know until we lost a lot of money?
Tough Choices
I'm still trying to decide what to do. We might consider getting our staff or an independent group to review the tool kit source code, in the unlikely event that the vendor would agree to this. But that would be very expensive, and with every new release, we'd have to check it all again.
Even then, how would we know that the compiled DLLs and executable files that we received were produced from the source code that was checked?
If we don't do something, then the very tool kits we require our programmers to use to reduce security risks might provide a hacker with a direct route into the core of our company. For now, we're just running antivirus scans on all our software.
This issue has opened my eyes to the level of trust I can extend to a vendor. What would you do if you were in my situation? I welcome your suggestions.
What do you think?
This week's journal is written by a real security manager, "Vince Tuesday," whose name and employer have been disguised for obvious reasons. Contact him at vince.tuesday@hushmail.com, or join the discussion in our forum: QuickLink a1590.
To find a complete archive of our Security Manager's Journals, go to computerworld.com/secjournal.