When trusted IT pros go bad

Computerworld - It's a CIO's worst nightmare: a call from the Business Software Alliance, saying that some of the software your company uses might be pirated.

You investigate and find that not only is your software illegal, it was sold to you by a company secretly owned and operated by none other than your own IT systems administrator, who's been a trusted employee for seven years. When you start digging into the admin's activities, you find a for-pay porn website he's been running on one of your corporate servers. Then you find that he's downloaded 400 customer credit card numbers from your e-commerce server.

And here's the worst part: He's the only one with the administrative passwords.

Think it can't happen? It did, according to a security consultant who was called in to help the victim, a $250 million retailer in Pennsylvania. You never heard about it because the company kept it quiet.

Despite the occasional headlines about IT folks gone rogue, most companies sweep such situations under the rug as quickly and as quietly as possible.

An annual survey by CSO magazine, the U.S. Secret Service and CERT (a program of the Software Engineering Institute at Carnegie Mellon University) routinely finds that three quarters of companies that are victimized by insiders handle the incidents internally, says Dawn Cappelli, technical manager of CERT's Insider Threat Center. "So we know that [what's made public] is only the tip of the iceberg," she says.

By keeping things quiet, however, victimized companies deny others the opportunity to learn from their experiences. CERT has tried to fill that void. It has studied insider threats since 2001, collecting information on more than 400 cases. In its most recent report, which analyzes more than 250 cases, CERT says the most common mistakes include failing to vet job applicants thoroughly, neglecting to adequately monitor the process of granting access privileges, and overlooking red flags in behavior.

But the threats posed by privilege-laden IT employees are especially hard to recognize. For one thing, staffers' nefarious activities can look the same as their regular duties. IT employees routinely "edit and write scripts, edit code and write programs, so it doesn't look like anomalous activity," Cappelli says. They know where your security is weakest and how to cover their tracks.

Victimized companies typically won't talk, but security consultants who help clean up the messes sometimes do. We talked to three security pros who shared these stunning tales of rogue IT employees.

Pirating Software -- and Worse

The Pennsylvania retailer's tale of woe began in early 2008, when the BSA notified it that Microsoft had uncovered licensing discrepancies, according to John Linkous. Today, Linkous is chief security and compliance officer at eIQ Networks, a security consultancy. His experience with the incident involving the retailer is from his previous job, when he was vice president of operations at Sabera, a now-defunct security consultancy.

Microsoft had traced the sale of the suspect software to a sysadmin at a company that was a Sabera client. For the purposes of this story, we'll call that sysadmin "Ed." When Linkous and other members of the Sabera team were secretly called in to investigate, they found that Ed had sold more than a half-million dollars in pirated Microsoft, Adobe and SAP software to his employer.

The investigators also noticed that network bandwidth use was abnormally high. "We thought there was some kind of network-based attack going on," says Linkous. They traced the activity to a server with more than 50,000 pornographic still images and more than 2,500 videos, according to Linkous.

In addition, a forensic search of Ed's workstation uncovered a spreadsheet with hundreds of credit card numbers from the company's e-commerce site. While there was no indication that the numbers had been used, the fact that the information was in a spreadsheet implied that Ed was contemplating using the card data himself or selling it to a third party, according to Linkous.

The retailer's chief financial officer, who had originally received the call from the BSA, and others on the senior management team feared what Ed might do when confronted. He was the only one who had certain administrative passwords -- including passwords for the core network router/firewall, network switches, the corporate VPN, the HR system, email server administration, Windows Active Directory administration, and Windows desktop administration.

That meant that Ed could have held hostage nearly all the company's major business processes, including the corporate website, email, financial reporting system and payroll. "This guy had keys to the kingdom," says Linkous.

So the company and Linkous' firm launched an operation right out of Mission: Impossible. They invented a ruse that required Ed to fly overnight to California. The long flight gave Linkous' team a window of about five and a half hours during which Ed couldn't possibly access the system. Working as fast as they could, the team mapped out the network and reset all the passwords. When Ed landed in California, "the COO was there to meet him. He was fired on the spot."

The cost: Linkous estimates that the incident cost the company a total of $250,000 to $300,000, which includes Sabera's fee, the cost of flying Ed to the West Coast on short notice, the cost of litigation against Ed, the costs associated with hiring a temporary network administrator and a new CIO, and the cost of making all of the company's software licenses legitimate.

Preventive measures: What could have prevented this disaster? Obviously, at least one other person should have known the passwords. But more significant was the lack of separation of duties. The retailer had a small IT staff (just six employees), so Ed was entrusted with both administrative and security responsibilities. That meant he was monitoring himself.