Agency destroys $170K worth of IT gear over non-existent malware threat
Another $3 million worth of equipment at the Economic Development Administration would have met same fate but for lack of funds
Computerworld - The U.S. Department of Commerce's Economic Development Administration destroyed about $170,000 worth of IT equipment including computers, printers, keyboards and computer mice last year on the mistaken belief that the systems were irreparably compromised by malware.
The bureau was poised to destroy an additional $3 million worth of IT equipment but was prevented from doing so by a lack of funding for the effort, a report released by the Commerce Department's Inspector General says.
The EDA's startling overreaction to an imagined threat to its networks appears to have stemmed from an almost comical series of miscommunications between computer security incident handlers at the Department of Commerce and at the EDA.
The problems started with an alert issued by the Department of Homeland Security (DHS) in December 2011, warning the Commerce Department of a potential malware infection within its networks. Security administrators at the Commerce Department identified the potentially infected computers as belonging to the EDA and alerted the bureau of the compromise.
However, the department's initial notification to the EDA incorrectly listed a total of 146 systems as being potentially infected, when in fact just two of them were infected.
A day later, the computer incident response team at the Commerce Department sent a second e-mailed incident notification to the EDA containing new analysis that identified only two systems as being infected with malware. However, the second notification was vague and did not clearly call out the fact that the first alert had been inaccurate, according to the inspector general's report.
Instead, the second alert began by stating that the first notification had been accurate and made no mention of any mistake in the previously provided information. Subsequently, incident handlers at the EDA assumed that the second notification was merely a confirmation of the analysis in the first alert and proceeded to assume that a major portion of their network had been compromised.
Over the next several weeks, incident response teams at the Commerce Department and EDA continued to work with a completely different understanding of the scope of the problem. The incident response team at Commerce assumed that their counterparts at the EDA had read and understood that the second notification superseded the initial incorrect alert while the EDA continued laboring under the belief that 146 of its systems had been compromised.
The EDA's impressions of a widespread compromise appeared to be confirmed when a forensic analysis of two systems showed them to be infected with malware. So, when the Commerce Department eventually asked the EDA to reimage its systems in order to get rid of the malware, the EDA responded by saying that there were too many systems involved for such reimaging to be feasible.
Rather than follow-up with the EDA to see what was going on, incident handlers at the Commerce Department wrongly assumed that the EDA had done an independent analysis of its systems and had identified many more systems that had been compromised.
"Unfortunately, both organizations continued to propagate the inaccurate information ... during the incident response activities," the IG's report noted.
In January 2012, EDA's CIO, Chuck Benjamin, decided to isolate the bureau's systems from the network on the mistaken belief that the infection was rampant and could spread to other networks. The CIO's decision to disconnect the system from the network also stemmed from, what turned out to be unfounded, fears that nation-state actors were behind the network infections.
A timeline of events provided in the IG's report does not indicate when the EDA began destroying its IT systems in its effort to contain the imagined network infection. It does note however that Benjamin "concluded that the risk, or potential risk, of extremely persistent malware and nation-state activity [which did not exist] was great enough to necessitate the physical destruction of all of EDA's IT components," the report said.
"By August 1, 2012, EDA had exhausted funds for this effort and therefore halted the destruction of its remaining IT components, valued at over $3 million," the IG said.
The report slammed Benjamin and incident responders at both the Department of Commerce and the EDA for the snafu. It faulted the Commerce Department's incident response team for sending the initial incorrect notification, not properly documenting its communications, putting an inexperienced incident responder in charge of communicating with the EDA and then for not coordinating a proper response with the bureau.
The IG blamed Benjamin for not putting enough effort to properly validate the scope and seriousness of the reported infection before embarking on a needless and costly recovery effort. Even after an external security contractor hired by the EDA had identified only minor, easily remediated malware infections on the bureau's systems, Benjamin proceeded with his drastic recovery measures.
"In the end, nothing identified on EDA's components posed a significant risk to EDA's operations," the report noted. "Despite only finding common malware infections, EDA's management and CIO remained convinced that there could be extremely persistent malware somewhere in EDA's IT systems."
In total, the EDA spent $2.7 million -- or half its FY 2012 IT budget -- responding to the non-existent threat to its network. Despite fairly straightforward recovery recommendations from the National Security Agency and the DHS, the EDA focused on building out a new and improved IT infrastructure instead.
After disconnecting its systems last January, the EDA signed up for a shared service from the U.S. Census Bureau to maintain a Web presence and for email services. Last March, the bureau issued new laptops to all users and April set-up a standalone implementation of its core business applications.
In September 2012, the bureau submitted a request to the Commerce Department's IT Review Board for $26 million over the next three year to fund its recovery efforts, the IG's report noted. The request was denied.
In February this year, the Commerce Department's IT began a full-scale recovery effort and restored EDA's full operational capabilities in five weeks.
In response to the IG's findings, Matt Erskine, the deputy assistant secretary of Commerce for Economic Development noted that EDA had acted "out of an abundance of caution" throughout the incident. The response noted that the EDA had also continued to conduct and complete important work despite the disruptions.
In a separate repose, Simon Szykman, the CIO for the Department of Commerce noted that the department has launched a comprehensive incident response improvement project. As part of that effort, the Department has already completed a third-party assessment of its incident response capabilities, hired three experienced incident handlers and put a new security incident tracking system in place, Szykman noted.
Jaikumar Vijayan covers data security and privacy issues, financial services security and e-voting for Computerworld. Follow Jaikumar on Twitter at @jaivijayan or subscribe to Jaikumar's RSS feed . His e-mail address is firstname.lastname@example.org.
Read more about Government IT in Computerworld's Government IT Topic Center.
- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
If you use ‘password,’ one the worst passwords, as your password, fail to keep antivirus protection updated and don’t bother to deploy security patches to close critical vulnerabilities, then maybe you should consider working for the cybersecurity-clueless federal government; you’d fit right in, according to Senator Tom Coburn's cybersecurity and critical infrastructure report.
- IT Certification Study Tips
- Register for this Computerworld Insider Study Tip guide and gain access to hundreds of premium content articles, cheat sheets, product reviews and more.
- Changing the Way Government Works: Four Technology Trends that Drive Down Costs and Increase Productivity
- This paper discusses four technology-based approaches to improving processes and increasing
productivity while driving down department and agency costs.
- HP HAVEn: See the big picture in Big Data
- HP HAVEn is the industry's first comprehensive, scalable, open, and secure platform for Big Data. Enterprises are drowning in a sea of data...
- What Datapipe customers need to know about the new PCI DSS 3.0 compliance standard
- This handy quick reference outlines what PCI DSS 3.0 is, who needs to be compliant and how Alert Logic solutions address the new...
- The 12 PCI DSS 3.0 requirements addressed by Peer 1 Hosting
- This handy quick reference outlines the 12 PCI DSS 3.0 requirements, who needs to be compliant and how Alert Logic solutions address the...
- Defense Throughout the Vulnerability Life Cycle
- This whitepaper provides insight into how to leverage threat and log management technologies to protect your IT assets throughout their vulnerability life cycle. All Government IT White Papers
- Meg Whitman presents Unlocking IT with Big Data During this Web Event you will hear Meg Whitman, President and CEO, HP discuss HAVEn - the #1 Big Data platform, as well...
- The New Way to Work Knowledge Vault This Knowledge Vault focuses on how, in today's increasingly virtual world, it's more important than ever to engage deeply with employees, suppliers, partners,...
- Getting Ready for BlackBerry Enterprise Service 10.2 Find out how BlackBerry® Enterprise Service 10 helps organizations address the full spectrum of EMM challenges, while balancing the needs of both the...
- Containerization Options: How to Choose the Best DLP Solution for Your Organization This webcast outlines a framework for making the right choice when it comes to containerization approaches, along with the pros and cons of...
- Mobile Apps and Devices Slash Customer Cycle Time Consolidated Engineering Laboratories' field employees used to collect data on triplicate forms that were sometimes hard to read and difficult to manage. After...
- All Government IT Webcasts