Agency destroys $170K worth of IT gear over non-existent malware threat
Another $3 million worth of equipment at the Economic Development Administration would have met same fate but for lack of funds
Computerworld - The U.S. Department of Commerce's Economic Development Administration destroyed about $170,000 worth of IT equipment including computers, printers, keyboards and computer mice last year on the mistaken belief that the systems were irreparably compromised by malware.
The bureau was poised to destroy an additional $3 million worth of IT equipment but was prevented from doing so by a lack of funding for the effort, a report released by the Commerce Department's Inspector General says.
The EDA's startling overreaction to an imagined threat to its networks appears to have stemmed from an almost comical series of miscommunications between computer security incident handlers at the Department of Commerce and at the EDA.
The problems started with an alert issued by the Department of Homeland Security (DHS) in December 2011, warning the Commerce Department of a potential malware infection within its networks. Security administrators at the Commerce Department identified the potentially infected computers as belonging to the EDA and alerted the bureau of the compromise.
However, the department's initial notification to the EDA incorrectly listed a total of 146 systems as being potentially infected, when in fact just two of them were infected.
A day later, the computer incident response team at the Commerce Department sent a second e-mailed incident notification to the EDA containing new analysis that identified only two systems as being infected with malware. However, the second notification was vague and did not clearly call out the fact that the first alert had been inaccurate, according to the inspector general's report.
Instead, the second alert began by stating that the first notification had been accurate and made no mention of any mistake in the previously provided information. Subsequently, incident handlers at the EDA assumed that the second notification was merely a confirmation of the analysis in the first alert and proceeded to assume that a major portion of their network had been compromised.
Over the next several weeks, incident response teams at the Commerce Department and EDA continued to work with a completely different understanding of the scope of the problem. The incident response team at Commerce assumed that their counterparts at the EDA had read and understood that the second notification superseded the initial incorrect alert while the EDA continued laboring under the belief that 146 of its systems had been compromised.
The EDA's impressions of a widespread compromise appeared to be confirmed when a forensic analysis of two systems showed them to be infected with malware. So, when the Commerce Department eventually asked the EDA to reimage its systems in order to get rid of the malware, the EDA responded by saying that there were too many systems involved for such reimaging to be feasible.
Rather than follow-up with the EDA to see what was going on, incident handlers at the Commerce Department wrongly assumed that the EDA had done an independent analysis of its systems and had identified many more systems that had been compromised.
"Unfortunately, both organizations continued to propagate the inaccurate information ... during the incident response activities," the IG's report noted.
In January 2012, EDA's CIO, Chuck Benjamin, decided to isolate the bureau's systems from the network on the mistaken belief that the infection was rampant and could spread to other networks. The CIO's decision to disconnect the system from the network also stemmed from, what turned out to be unfounded, fears that nation-state actors were behind the network infections.
A timeline of events provided in the IG's report does not indicate when the EDA began destroying its IT systems in its effort to contain the imagined network infection. It does note however that Benjamin "concluded that the risk, or potential risk, of extremely persistent malware and nation-state activity [which did not exist] was great enough to necessitate the physical destruction of all of EDA's IT components," the report said.
"By August 1, 2012, EDA had exhausted funds for this effort and therefore halted the destruction of its remaining IT components, valued at over $3 million," the IG said.
The report slammed Benjamin and incident responders at both the Department of Commerce and the EDA for the snafu. It faulted the Commerce Department's incident response team for sending the initial incorrect notification, not properly documenting its communications, putting an inexperienced incident responder in charge of communicating with the EDA and then for not coordinating a proper response with the bureau.
The IG blamed Benjamin for not putting enough effort to properly validate the scope and seriousness of the reported infection before embarking on a needless and costly recovery effort. Even after an external security contractor hired by the EDA had identified only minor, easily remediated malware infections on the bureau's systems, Benjamin proceeded with his drastic recovery measures.
"In the end, nothing identified on EDA's components posed a significant risk to EDA's operations," the report noted. "Despite only finding common malware infections, EDA's management and CIO remained convinced that there could be extremely persistent malware somewhere in EDA's IT systems."
In total, the EDA spent $2.7 million -- or half its FY 2012 IT budget -- responding to the non-existent threat to its network. Despite fairly straightforward recovery recommendations from the National Security Agency and the DHS, the EDA focused on building out a new and improved IT infrastructure instead.
After disconnecting its systems last January, the EDA signed up for a shared service from the U.S. Census Bureau to maintain a Web presence and for email services. Last March, the bureau issued new laptops to all users and April set-up a standalone implementation of its core business applications.
In September 2012, the bureau submitted a request to the Commerce Department's IT Review Board for $26 million over the next three year to fund its recovery efforts, the IG's report noted. The request was denied.
In February this year, the Commerce Department's IT began a full-scale recovery effort and restored EDA's full operational capabilities in five weeks.
In response to the IG's findings, Matt Erskine, the deputy assistant secretary of Commerce for Economic Development noted that EDA had acted "out of an abundance of caution" throughout the incident. The response noted that the EDA had also continued to conduct and complete important work despite the disruptions.
In a separate repose, Simon Szykman, the CIO for the Department of Commerce noted that the department has launched a comprehensive incident response improvement project. As part of that effort, the Department has already completed a third-party assessment of its incident response capabilities, hired three experienced incident handlers and put a new security incident tracking system in place, Szykman noted.
Jaikumar Vijayan covers data security and privacy issues, financial services security and e-voting for Computerworld. Follow Jaikumar on Twitter at @jaivijayan or subscribe to Jaikumar's RSS feed . His e-mail address is email@example.com.
Read more about Government IT in Computerworld's Government IT Topic Center.
This pilot fish is a contractor at a military base, working on some very cool fire-control systems for tanks. But when he spots something obviously wrong during a live-fire test, he can't get the firing-range commander's attention.
- IT Certification Study Tips
- Register for this Computerworld Insider Study Tip guide and gain access to hundreds of premium content articles, cheat sheets, product reviews and more.
- Reduce federal infrastructure risk with compliance management and situational awareness
- IBM continuous monitoring and management solutions deliver real-time situational awareness to help federal agencies understand vulnerabilities, and protect the infrastructure.
- Considerations For Effective Software License Management
- For many reasons, software license management has become a critical issue for many IT organizations and enterprise's alike. With many licensing options, hurdles...
- eBay uses 100% OpenSource WSO2 ESB to process more than 1Billion transactions a day
- Along with eBay's success comes a huge demand to ensure reliable, 24x7 availability of the services that enable these transactions. For eBay, that...
- A Reference Architecture for the Internet of Things
- The aim of this is to provide Architects and Developers of IoT projects with an effective starting point that covers the major requirements...
- REST easy: API Design, Evolution and Connection
- RESTful design increases API performance, reduces development effort, and minimizes operational support burden. By following a few best practices and selecting RESTful tooling,... All Government IT White Papers
- It's not too late...Get Your Mobile Questions Answered Live! How can IT provide seamless and secure mobile communications and collaboration for all? Join this live Webcast as IDG asks an expert panel...
- Why do you need an enterprise mobile platform? Today companies must offer great apps that run on a range of devices, and connect to an exploding set of backend data. Appcelerator...
- The Foundation You Need to Build a Better Storage Infrastructure Watch this webcast to hear how you can maximize the economics of your data center by modifying your storage footprint and power usage...
- Building Tomorrow's Data Center with Converged Technologies A number of forces are converging: the cloud, converged infrastructure, big data and fabric architectures to name a few.
- Technology for Everyone A Kansas school district modernizes teaching and learning and paves the way to a one-to-one program with a comprehensive upgrade of its wireless...
- All Government IT Webcasts