Update: Virginia's IT outage continues, 3 agencies still affected

Computerworld - Several Virginia state agencies continue to experience problems with data access due to an outage related to problems in a storage-area network (SAN) that began last week in a data center run by outsourcer Northrop Grumman.

An automated phone message from the Virginia Information Technologies Agency (VITA) earlier today said that as of this morning as many as seven key agencies were still having problems accessing applications, shared folders and other data stored on servers in the state's Enterprise Solutions Center in Richmond.

Late today, VITA issued a statement saying that only three of the 27 agencies initially affected by the outage were still without service. Those agencies are the Department of Motor Vehicles (DMV), the Department of Taxation and the state Board of Elections. Virginia has 89 state agencies.

"The DMV was heavily impacted by this hardware failure and has been unable to process in-person driver's licenses or ID cards at its 74 customer service centers," Virginia's Secretary of Technology Jim Duffey said in a statement.

Duffey said the problem was caused by an EMC DMX-3 storage array -- the vendor's flagship product. A root cause analysis of the failure is now under way, VITA said.

"The storage unit has been repaired and we have been in the meticulous process of carefully restoring data since the failure," Duffey said. "According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented."

VITA has offered updates about its attempts to repair the outage on its Web site.

According to published reports, computer systems came to a halt last Wednesday because a memory card failure in the EMC SAN. A backup SAN that was supposed to act as a fail-over system then also experienced problems.

"I have confidence in the teams that are working on this problem. They are aggressively executing our recovery plan and are working tirelessly to restore all the affected agencies to a fully operational status," Duffey said. "They have made significant progress and continue to do so."

Earlier this year, after a long history of problems within VITA, Virginia's governor placed its CIO under his direct authority, appointing former Virginia lawmaker Sam Nixon to the position. Nixon, who had co-authored the legislation establishing VITA in 2003, was charged with the task of fixing the IT problems.

Northrop Grumman did not immediately respond to a request for comment.

VITA's previous CIO, Lem Stewart, was fired last year. According to reports, Stewart had withheld $15 million in payments to Northrop Grumman because he believed the service provider failed to meet contractual obligations.

Under the terms of a $2.4 billion contract, Northrop Grumman runs the state's data centers, help desks and other IT operations. The contract requires Northrop Grumman to keep VITA's data centers in the state.

VITA's contract with Northrop Grumman has come under heavy fire in the past for a number of reasons, including project delays, cost overruns and performance problems that included other service outages.

After an audit by Virginia's Joint Legislative Audit and Review Commission last year, VITA's contract with Northrop Grumman was modified, resulting in more stringent performance requirements and greater conditions around accountability. The contract, however, also increased payments to Northrop Grumman by $105 million over nine years.

Along with VITA, the latest outage affected the Virginia State Board of Elections, the Department of Juvenile Justice, the Department of Motor Vehicles, the Department of Social Services, the Department of Taxation and the Department of Environmental Quality.

Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian, or subscribe to Lucas's RSS feed . His e-mail address is lmearian@computerworld.com.

Read more about Business Continuity in Computerworld's Business Continuity Topic Center.