Agency destroys $170K worth of IT gear over non-existent malware threat
Another $3 million worth of equipment at the Economic Development Administration would have met same fate but for lack of funds
Computerworld - The U.S. Department of Commerce's Economic Development Administration destroyed about $170,000 worth of IT equipment including computers, printers, keyboards and computer mice last year on the mistaken belief that the systems were irreparably compromised by malware.
The bureau was poised to destroy an additional $3 million worth of IT equipment but was prevented from doing so by a lack of funding for the effort, a report released by the Commerce Department's Inspector General says.
The EDA's startling overreaction to an imagined threat to its networks appears to have stemmed from an almost comical series of miscommunications between computer security incident handlers at the Department of Commerce and at the EDA.
The problems started with an alert issued by the Department of Homeland Security (DHS) in December 2011, warning the Commerce Department of a potential malware infection within its networks. Security administrators at the Commerce Department identified the potentially infected computers as belonging to the EDA and alerted the bureau of the compromise.
However, the department's initial notification to the EDA incorrectly listed a total of 146 systems as being potentially infected, when in fact just two of them were infected.
A day later, the computer incident response team at the Commerce Department sent a second e-mailed incident notification to the EDA containing new analysis that identified only two systems as being infected with malware. However, the second notification was vague and did not clearly call out the fact that the first alert had been inaccurate, according to the inspector general's report.
Instead, the second alert began by stating that the first notification had been accurate and made no mention of any mistake in the previously provided information. Subsequently, incident handlers at the EDA assumed that the second notification was merely a confirmation of the analysis in the first alert and proceeded to assume that a major portion of their network had been compromised.
Over the next several weeks, incident response teams at the Commerce Department and EDA continued to work with a completely different understanding of the scope of the problem. The incident response team at Commerce assumed that their counterparts at the EDA had read and understood that the second notification superseded the initial incorrect alert while the EDA continued laboring under the belief that 146 of its systems had been compromised.
The EDA's impressions of a widespread compromise appeared to be confirmed when a forensic analysis of two systems showed them to be infected with malware. So, when the Commerce Department eventually asked the EDA to reimage its systems in order to get rid of the malware, the EDA responded by saying that there were too many systems involved for such reimaging to be feasible.
Rather than follow-up with the EDA to see what was going on, incident handlers at the Commerce Department wrongly assumed that the EDA had done an independent analysis of its systems and had identified many more systems that had been compromised.
"Unfortunately, both organizations continued to propagate the inaccurate information ... during the incident response activities," the IG's report noted.
In January 2012, EDA's CIO, Chuck Benjamin, decided to isolate the bureau's systems from the network on the mistaken belief that the infection was rampant and could spread to other networks. The CIO's decision to disconnect the system from the network also stemmed from, what turned out to be unfounded, fears that nation-state actors were behind the network infections.
A timeline of events provided in the IG's report does not indicate when the EDA began destroying its IT systems in its effort to contain the imagined network infection. It does note however that Benjamin "concluded that the risk, or potential risk, of extremely persistent malware and nation-state activity [which did not exist] was great enough to necessitate the physical destruction of all of EDA's IT components," the report said.
"By August 1, 2012, EDA had exhausted funds for this effort and therefore halted the destruction of its remaining IT components, valued at over $3 million," the IG said.
The report slammed Benjamin and incident responders at both the Department of Commerce and the EDA for the snafu. It faulted the Commerce Department's incident response team for sending the initial incorrect notification, not properly documenting its communications, putting an inexperienced incident responder in charge of communicating with the EDA and then for not coordinating a proper response with the bureau.
The IG blamed Benjamin for not putting enough effort to properly validate the scope and seriousness of the reported infection before embarking on a needless and costly recovery effort. Even after an external security contractor hired by the EDA had identified only minor, easily remediated malware infections on the bureau's systems, Benjamin proceeded with his drastic recovery measures.
"In the end, nothing identified on EDA's components posed a significant risk to EDA's operations," the report noted. "Despite only finding common malware infections, EDA's management and CIO remained convinced that there could be extremely persistent malware somewhere in EDA's IT systems."
In total, the EDA spent $2.7 million -- or half its FY 2012 IT budget -- responding to the non-existent threat to its network. Despite fairly straightforward recovery recommendations from the National Security Agency and the DHS, the EDA focused on building out a new and improved IT infrastructure instead.
After disconnecting its systems last January, the EDA signed up for a shared service from the U.S. Census Bureau to maintain a Web presence and for email services. Last March, the bureau issued new laptops to all users and April set-up a standalone implementation of its core business applications.
In September 2012, the bureau submitted a request to the Commerce Department's IT Review Board for $26 million over the next three year to fund its recovery efforts, the IG's report noted. The request was denied.
In February this year, the Commerce Department's IT began a full-scale recovery effort and restored EDA's full operational capabilities in five weeks.
In response to the IG's findings, Matt Erskine, the deputy assistant secretary of Commerce for Economic Development noted that EDA had acted "out of an abundance of caution" throughout the incident. The response noted that the EDA had also continued to conduct and complete important work despite the disruptions.
In a separate repose, Simon Szykman, the CIO for the Department of Commerce noted that the department has launched a comprehensive incident response improvement project. As part of that effort, the Department has already completed a third-party assessment of its incident response capabilities, hired three experienced incident handlers and put a new security incident tracking system in place, Szykman noted.
Jaikumar Vijayan covers data security and privacy issues, financial services security and e-voting for Computerworld. Follow Jaikumar on Twitter at @jaivijayan or subscribe to Jaikumar's RSS feed . His e-mail address is email@example.com.
Read more about Government IT in Computerworld's Government IT Topic Center.
- Silicon Valley's 19 Coolest Places to Work
- Is Windows 8 Development Worth the Trouble?
- 8 Books Every IT Leader Should Read This Year
- 10 Hot Hadoop Startups to Watch
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
This state transportation department uses computer science students from a local university as programming interns, and everyone is happy with the arrangement -- until one intern learns how to bring down the mainframe.
- IT Certification Study Tips
- Register for this Computerworld Insider Study Tip guide and gain access to hundreds of premium content articles, cheat sheets, product reviews and more.
- Changing the Way Government Works: Four Technology Trends that Drive Down Costs and Increase Productivity
- This paper discusses four technology-based approaches to improving processes and increasing
productivity while driving down department and agency costs.
- Acxiom Case Study
- This case study, which focuses on Acxiom, explores how the company was able to secure employee data, reduce migration costs and boost productivity...
- Windows® XP Migration: Protect and Secure Critical Data
- With the end of the Microsoft Windows XP operating system's lifecycle on April 8, 2014, businesses are faced with the decision to migrate...
- Exponentially Accelerate Data Protection and Recovery with Simpana 10 IntelliSnap® Snapshot Management Technology
- Are you making the best use of your storage array snapshot functionality? CommVault Simpana 10 IntelliSnap technology manages hardware-based snapshots across multiple vendor...
- Simpana IntelliSnap Technology Datasheet
- With IntelliSnap you can maximize the value of your snapshot technology while dramatically reducing management overhead and complexity. All Government IT White Papers
- Top 4 Digital Signage Fails Join RMG Networks for a look at four of the most common reasons digital signage fails in corporate businesses. Learn about strategies to...
- Building Tomorrow's Infrastructure Listen to this podcast to discover how Crider Foods worked with PC Connection to update their IT infrastructure, while maintaining compliance and control.
Enhance Your Virtualization Infrastructure With IBM and Vmware
Date: Wednesday, May 14, 2014, 1:00 PM EDT
Virtualization technology is now expanding beyond the server compute elements to encompass networking and storage...
Transforming Finance, Procurement and Supply Chain Effectiveness with Cross-Functional Analytics
Date: May 6th, 2014
Time: 1 PM EDT
Attend this Webcast to find out how Oracle's packaged analytic applications enable line-of-business managers to examine all...
- Cloud Knowledge Vault Learn how your organization can benefit from the scalability, flexibility, and performance that the cloud offers through the short videos and other resources...
- All Government IT Webcasts