The VA's computer systems meltdown: What happened and why
Not following best practices can render the best technology useless
November 20, 2007 12:00 PM ETComputerworld - At times, the bad news coming from the U.S. Department of Veterans Affairs seems unstoppable: D-grade medical facilities, ongoing security and privacy breaches, and a revolving door of departing leadership. In September, during a hearing by the House Committee on Veterans' Affairs, lawmakers learned about an unscheduled system failure that took down key applications in 17 VA medical facilities for a day.
Characterized by Dr. Ben Davoren, the director of clinical informatics for the San Francisco VA Medical Center, as "the most significant technological threat to patient safety the VA has ever had," the outage has moved some observers to call into question the VA's direction in consolidating its IT operations. Yet the shutdown grew from a simple change management procedure that wasn't properly followed.
The small, undocumented change ended up bringing down the primary patient applications at 17 VA medical centers in Northern California. As a result, the schedule to centralize IT operations across more than 150 medical facilities into four regional data processing centers has been pulled back while VA IT leaders establish what the right approach is for its regionalization efforts.
The Region 1 Field Operations breakdown of Aug. 31 exposed just how challenging effecting substantial change is in a complex organization the size of the VA Office of Information & Technology (OI&T). Begun in October 2005 and originally scheduled to be completed by October 2008, the "reforming" of the IT organization at the VA involved several substantial goals: the creation of major departments along functional areas such as enterprise development, quality and performance, and IT oversight and compliance; the reassignment of 6,000 technical professionals to a more centralized management; and the adoption of 36 management processes defined in the Information Technology Infrastructure Library (ITIL).
As part of the reform effort, the VA was to shift local control of IT infrastructure operations to regional data-processing centers. Historically, each of the 150 or so medical centers run by the VA had its own IT service, its own budget authority and its own staff, as well as independence with regard to how the IT infrastructure evolved. All of the decisions regarding IT were made between a local IT leadership official and the director of that particular medical center. While that made on-site IT staff responsive to local needs, it made standardization across sites nearly impossible in areas such as security, infrastructure administration and maintenance, and disaster recovery.
The operations of its 150 medical facilities would relocate to four regional data processing centers, two in the east and two in the west. The latter, Regions 1 and 2, are located in Sacramento, Calif., and Denver respectively, run as part of the Enterprise Operations & Infrastructure (OPS) office.
Veterans Affairs
Additional Resources



White Papers & Webcasts
Southern Company
Download Now
Disaster Recovery 2008: Reduced Costs and Improved Performance
How long can your Enterprise afford to be without your data? With an accelerated disaster recovery program, you never have to answer this...
Defending Against the Storm
Download Now
HP StorageWorks EVA4400 & Microsoft
Download this video, free, compliments of HP.
Top 10 Things to Know about Data Protection
Download Now
Data Protection and Disaster Recovery with iSCSI and VMware
Get this on demand webcast now
Extending Client Refresh - 11 Steps to Maximize Savings
Register Now!
From Trust to Process: Closing the Risk Gap in Privileged Access Control
Download this Complimentary White Paper! Provided by BeyondTrust.
Lower the Cost and Complexity of a Mobile Workforce through Automation
Download This Resource Now!

Forrester Analyst Report: X86 Server Virtualization For High Availability and Disaster Recovery
Yankee Group. "Disaster Strikes! Is Your Business Ready? Disaster Preparedness for Mid-Sized Firms"
VMware White Paper: Transforming Disaster Recovery - VMware Infrastructure for rapid, reliable and cost-effective Disaster Recovery