Update: Virginia's IT outage continues, 3 agencies still affected
A memory card within a SAN caused the outage
Computerworld - Several Virginia state agencies continue to experience problems with data access due to an outage related to problems in a storage-area network (SAN) that began last week in a data center run by outsourcer Northrop Grumman.
An automated phone message from the Virginia Information Technologies Agency (VITA) earlier today said that as of this morning as many as seven key agencies were still having problems accessing applications, shared folders and other data stored on servers in the state's Enterprise Solutions Center in Richmond.
Late today, VITA issued a statement saying that only three of the 27 agencies initially affected by the outage were still without service. Those agencies are the Department of Motor Vehicles (DMV), the Department of Taxation and the state Board of Elections. Virginia has 89 state agencies.
"The DMV was heavily impacted by this hardware failure and has been unable to process in-person driver's licenses or ID cards at its 74 customer service centers," Virginia's Secretary of Technology Jim Duffey said in a statement.
Duffey said the problem was caused by an EMC DMX-3 storage array -- the vendor's flagship product. A root cause analysis of the failure is now under way, VITA said.
"The storage unit has been repaired and we have been in the meticulous process of carefully restoring data since the failure," Duffey said. "According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented."
VITA has offered updates about its attempts to repair the outage on its Web site.
According to published reports, computer systems came to a halt last Wednesday because a memory card failure in the EMC SAN. A backup SAN that was supposed to act as a fail-over system then also experienced problems.
"I have confidence in the teams that are working on this problem. They are aggressively executing our recovery plan and are working tirelessly to restore all the affected agencies to a fully operational status," Duffey said. "They have made significant progress and continue to do so."
Earlier this year, after a long history of problems within VITA, Virginia's governor placed its CIO under his direct authority, appointing former Virginia lawmaker Sam Nixon to the position. Nixon, who had co-authored the legislation establishing VITA in 2003, was charged with the task of fixing the IT problems.
Northrop Grumman did not immediately respond to a request for comment.
VITA's previous CIO, Lem Stewart, was fired last year. According to reports, Stewart had withheld $15 million in payments to Northrop Grumman because he believed the service provider failed to meet contractual obligations.
Under the terms of a $2.4 billion contract, Northrop Grumman runs the state's data centers, help desks and other IT operations. The contract requires Northrop Grumman to keep VITA's data centers in the state.
VITA's contract with Northrop Grumman has come under heavy fire in the past for a number of reasons, including project delays, cost overruns and performance problems that included other service outages.
After an audit by Virginia's Joint Legislative Audit and Review Commission last year, VITA's contract with Northrop Grumman was modified, resulting in more stringent performance requirements and greater conditions around accountability. The contract, however, also increased payments to Northrop Grumman by $105 million over nine years.
Along with VITA, the latest outage affected the Virginia State Board of Elections, the Department of Juvenile Justice, the Department of Motor Vehicles, the Department of Social Services, the Department of Taxation and the Department of Environmental Quality.
Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian, or subscribe to Lucas's RSS feed . His e-mail address is firstname.lastname@example.org.
Read more about Disaster Recovery in Computerworld's Disaster Recovery Topic Center.
- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- Addressing the Broken State of Backup with a New Category of Disk-Based Backup Solutions Today, IT organizations are faced with a number of challenges when managing backup processes, including the need for faster backup, restore, tape copy,...
- Optimizing Approaches to Enterprise Backup and Recovery IT organizations are faced with ensuring that backups occur in the shortest amount of time and are not operationally disruptive as well as...
- How Backup Disk Architecture Impacts the Backup Window This paper compares disk based backup architectures, the impact that data deduplication has on backup performance, and how well the solution scales as...
- How Data Deduplication Impacts Recovery Data deduplication has clear benefits when it comes to efficiently retaining backup data on disk and replicating data offsite for disaster recovery --...
- Pre-Engineered solutions from VCE Simplify Core Infrastructure Implementation In this video, the CTO of Purdue Pharma, a privately held pharmaceutical company explains how Purdue transformed their data center infrastructure with VCE.
- Data Protection and Disaster Recovery with iSCSI and VMware Get this on demand webcast now All Disaster Recovery White Papers | Webcasts