Update: Virginia's IT outage continues, 3 agencies still affected
A memory card within a SAN caused the outage
Computerworld - Several Virginia state agencies continue to experience problems with data access due to an outage related to problems in a storage-area network (SAN) that began last week in a data center run by outsourcer Northrop Grumman.
An automated phone message from the Virginia Information Technologies Agency (VITA) earlier today said that as of this morning as many as seven key agencies were still having problems accessing applications, shared folders and other data stored on servers in the state's Enterprise Solutions Center in Richmond.
Late today, VITA issued a statement saying that only three of the 27 agencies initially affected by the outage were still without service. Those agencies are the Department of Motor Vehicles (DMV), the Department of Taxation and the state Board of Elections. Virginia has 89 state agencies.
"The DMV was heavily impacted by this hardware failure and has been unable to process in-person driver's licenses or ID cards at its 74 customer service centers," Virginia's Secretary of Technology Jim Duffey said in a statement.
Duffey said the problem was caused by an EMC DMX-3 storage array -- the vendor's flagship product. A root cause analysis of the failure is now under way, VITA said.
"The storage unit has been repaired and we have been in the meticulous process of carefully restoring data since the failure," Duffey said. "According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented."
VITA has offered updates about its attempts to repair the outage on its Web site.
According to published reports, computer systems came to a halt last Wednesday because a memory card failure in the EMC SAN. A backup SAN that was supposed to act as a fail-over system then also experienced problems.
"I have confidence in the teams that are working on this problem. They are aggressively executing our recovery plan and are working tirelessly to restore all the affected agencies to a fully operational status," Duffey said. "They have made significant progress and continue to do so."
Earlier this year, after a long history of problems within VITA, Virginia's governor placed its CIO under his direct authority, appointing former Virginia lawmaker Sam Nixon to the position. Nixon, who had co-authored the legislation establishing VITA in 2003, was charged with the task of fixing the IT problems.
Northrop Grumman did not immediately respond to a request for comment.
VITA's previous CIO, Lem Stewart, was fired last year. According to reports, Stewart had withheld $15 million in payments to Northrop Grumman because he believed the service provider failed to meet contractual obligations.
Under the terms of a $2.4 billion contract, Northrop Grumman runs the state's data centers, help desks and other IT operations. The contract requires Northrop Grumman to keep VITA's data centers in the state.
VITA's contract with Northrop Grumman has come under heavy fire in the past for a number of reasons, including project delays, cost overruns and performance problems that included other service outages.
After an audit by Virginia's Joint Legislative Audit and Review Commission last year, VITA's contract with Northrop Grumman was modified, resulting in more stringent performance requirements and greater conditions around accountability. The contract, however, also increased payments to Northrop Grumman by $105 million over nine years.
Along with VITA, the latest outage affected the Virginia State Board of Elections, the Department of Juvenile Justice, the Department of Motor Vehicles, the Department of Social Services, the Department of Taxation and the Department of Environmental Quality.
Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian, or subscribe to Lucas's RSS feed . His e-mail address is email@example.com.
Read more about Disaster Recovery in Computerworld's Disaster Recovery Topic Center.
- Best iPhone, iPad Business Apps for 2014
- 14 Tech Conventions You Should Attend in 2014
- 10 Desktop Apps to Power Your Windows PC
- How to Add New Job Skills Without Going Back to School
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- Review: Box beats Dropbox - and all the rest - for business Box trumps Dropbox, Engyte, Citrix ShareFile, EMC Syncplicity, and OwnCloud with rich mix of file sync, file sharing, user management, deep reporting and...
- Analyst Report-Mixed All Flash Arrays Delivers Safer Higher Performance What is the impact of an all-flash array with enterprise features and reliability on the mainstream data center? In the mainstream environment, storage...
- Embracing Flash Storage Exec Brief Flash storage can deliver impressive performance, especially for random I/O, by eliminating rotational and seek latencies that are common in all hard disk...
- Embracing Tiered Storage Exec Brief All data is not created equal and thus all data need not be treated the same by the storage system. IT executives must...
- Four Myths of High-Productivity App Dev Debunked Debunk the main myths surrounding high-productivity application development and how both platforms have overcome them.
On-Demand Webcast: 7 Reasons to Choose VoIP
Thinking about a new phone system for your business?
Be sure to watch this informative webcast. Steve Strauss, small business columnist for USA...
All Networked Storage White Papers |