Agency destroys $170K worth of IT gear over non-existent malware threat
Another $3 million worth of equipment at the Economic Development Administration would have met same fate but for lack of funds
Computerworld - The U.S. Department of Commerce's Economic Development Administration destroyed about $170,000 worth of IT equipment including computers, printers, keyboards and computer mice last year on the mistaken belief that the systems were irreparably compromised by malware.
The bureau was poised to destroy an additional $3 million worth of IT equipment but was prevented from doing so by a lack of funding for the effort, a report released by the Commerce Department's Inspector General says.
The EDA's startling overreaction to an imagined threat to its networks appears to have stemmed from an almost comical series of miscommunications between computer security incident handlers at the Department of Commerce and at the EDA.
The problems started with an alert issued by the Department of Homeland Security (DHS) in December 2011, warning the Commerce Department of a potential malware infection within its networks. Security administrators at the Commerce Department identified the potentially infected computers as belonging to the EDA and alerted the bureau of the compromise.
However, the department's initial notification to the EDA incorrectly listed a total of 146 systems as being potentially infected, when in fact just two of them were infected.
A day later, the computer incident response team at the Commerce Department sent a second e-mailed incident notification to the EDA containing new analysis that identified only two systems as being infected with malware. However, the second notification was vague and did not clearly call out the fact that the first alert had been inaccurate, according to the inspector general's report.
Instead, the second alert began by stating that the first notification had been accurate and made no mention of any mistake in the previously provided information. Subsequently, incident handlers at the EDA assumed that the second notification was merely a confirmation of the analysis in the first alert and proceeded to assume that a major portion of their network had been compromised.
Over the next several weeks, incident response teams at the Commerce Department and EDA continued to work with a completely different understanding of the scope of the problem. The incident response team at Commerce assumed that their counterparts at the EDA had read and understood that the second notification superseded the initial incorrect alert while the EDA continued laboring under the belief that 146 of its systems had been compromised.
The EDA's impressions of a widespread compromise appeared to be confirmed when a forensic analysis of two systems showed them to be infected with malware. So, when the Commerce Department eventually asked the EDA to reimage its systems in order to get rid of the malware, the EDA responded by saying that there were too many systems involved for such reimaging to be feasible.
Rather than follow-up with the EDA to see what was going on, incident handlers at the Commerce Department wrongly assumed that the EDA had done an independent analysis of its systems and had identified many more systems that had been compromised.
"Unfortunately, both organizations continued to propagate the inaccurate information ... during the incident response activities," the IG's report noted.
In January 2012, EDA's CIO, Chuck Benjamin, decided to isolate the bureau's systems from the network on the mistaken belief that the infection was rampant and could spread to other networks. The CIO's decision to disconnect the system from the network also stemmed from, what turned out to be unfounded, fears that nation-state actors were behind the network infections.
A timeline of events provided in the IG's report does not indicate when the EDA began destroying its IT systems in its effort to contain the imagined network infection. It does note however that Benjamin "concluded that the risk, or potential risk, of extremely persistent malware and nation-state activity [which did not exist] was great enough to necessitate the physical destruction of all of EDA's IT components," the report said.
"By August 1, 2012, EDA had exhausted funds for this effort and therefore halted the destruction of its remaining IT components, valued at over $3 million," the IG said.
The report slammed Benjamin and incident responders at both the Department of Commerce and the EDA for the snafu. It faulted the Commerce Department's incident response team for sending the initial incorrect notification, not properly documenting its communications, putting an inexperienced incident responder in charge of communicating with the EDA and then for not coordinating a proper response with the bureau.
The IG blamed Benjamin for not putting enough effort to properly validate the scope and seriousness of the reported infection before embarking on a needless and costly recovery effort. Even after an external security contractor hired by the EDA had identified only minor, easily remediated malware infections on the bureau's systems, Benjamin proceeded with his drastic recovery measures.
"In the end, nothing identified on EDA's components posed a significant risk to EDA's operations," the report noted. "Despite only finding common malware infections, EDA's management and CIO remained convinced that there could be extremely persistent malware somewhere in EDA's IT systems."
In total, the EDA spent $2.7 million -- or half its FY 2012 IT budget -- responding to the non-existent threat to its network. Despite fairly straightforward recovery recommendations from the National Security Agency and the DHS, the EDA focused on building out a new and improved IT infrastructure instead.
After disconnecting its systems last January, the EDA signed up for a shared service from the U.S. Census Bureau to maintain a Web presence and for email services. Last March, the bureau issued new laptops to all users and April set-up a standalone implementation of its core business applications.
In September 2012, the bureau submitted a request to the Commerce Department's IT Review Board for $26 million over the next three year to fund its recovery efforts, the IG's report noted. The request was denied.
In February this year, the Commerce Department's IT began a full-scale recovery effort and restored EDA's full operational capabilities in five weeks.
In response to the IG's findings, Matt Erskine, the deputy assistant secretary of Commerce for Economic Development noted that EDA had acted "out of an abundance of caution" throughout the incident. The response noted that the EDA had also continued to conduct and complete important work despite the disruptions.
In a separate repose, Simon Szykman, the CIO for the Department of Commerce noted that the department has launched a comprehensive incident response improvement project. As part of that effort, the Department has already completed a third-party assessment of its incident response capabilities, hired three experienced incident handlers and put a new security incident tracking system in place, Szykman noted.
Jaikumar Vijayan covers data security and privacy issues, financial services security and e-voting for Computerworld. Follow Jaikumar on Twitter at @jaivijayan or subscribe to Jaikumar's RSS feed . His e-mail address is firstname.lastname@example.org.
Read more about Government IT in Computerworld's Government IT Topic Center.
- 18 Hot IT Certifications for 2014
- CIOs Opting for IT Contractors Over Hiring Full-Time Staff
- 12 Best Free iOS 7 Holiday Shopping Apps
- For CMOs Big Data Can Lead to Big Profits
- Slideshow: 5 ways to lock down your mobile device
- Slideshow: 10 mistakes companies make after a data breach
- How to rob a bank: A social engineering walk through
- Which smartphone is the most secure?
If you think getting it right from day one is always what matters, you probably haven't been following technology too closely.
- IT Certification Study Tips
- Register for this Computerworld Insider Study Tip guide and gain access to hundreds of premium content articles, cheat sheets, product reviews and more.
- Mitigating DDoS Attacks with F5 Technology
- This document examines various DDoS attack methods and the application of specific ADC technologies to block attacks in the DDoS threat spectrum while...
- The DDoS Threat Spectrum
- Bolstered by favorable economics, today's global botnets are using distributed denial-of-service (DDoS) attacks to target firewalls, web services, and applications, often simultaneously.
- Defending Against Denial of Service Attacks
- By utilizing end-user interviews, this whitepaper explores a deeper understanding of DDoS defense plans and reveals the knowledge gaps around the Denial of...
- Strategic Solutions for Government IT
- This paper outlines why F5 is the optimum partner to help achieve the levels of security, performance and availability that are vital to...
- Armed and Dangerous: Help your IT Organization Embrace Enterprise Mobility
- Becoming a mobile enterprise means new opportunities for your organization yet letting employees choose their own devices and then access corporate resources, apps,... All Government IT White Papers
- Modernizing SAP environments with minimum risk - a path to Big Data Hear from top IDC analyst, Richard Villars, about the path you can start taking now to enable your organization to get the benefits...
- Fighting Fraud Videos: IBM Intelligent Investigation Manager Short videos about IBM Intelligent Investigation Manager (IIM) for Fraud. IIM optimizes the investigation of fraud for customers across many industries in both...
- IBM Intelligent Investigation Manager: Online Product Demo Intelligent Investigation Manager optimizes fraud investigation and analysis and it dynamically coordinates and reports on cases, provides analysis and visualization, and enables more...
- Webinar: IBM IIM for Fraud, Abuse and Waste in Government View this IBM webinar to learn about the challenges and opportunities in fraud reduction, waste, and abuse in government programs and agencies. You...
- Pre-Engineered solutions from VCE Simplify Core Infrastructure Implementation In this video, the CTO of Purdue Pharma, a privately held pharmaceutical company explains how Purdue transformed their data center infrastructure with VCE.
- All Government IT Webcasts
Does your organization offer extensive benefits, cool perks, competitive salaries, opportunities for training and advancement? Then get it recognized!
Nominate your company or another deserving organization for Computerworld's 2014 Best Places to Work in IT list now through Dec. 12, 2013.