The VA's computer systems meltdown: What happened and why
Not following best practices can render the best technology useless
Computerworld - At times, the bad news coming from the U.S. Department of Veterans Affairs seems unstoppable: D-grade medical facilities, ongoing security and privacy breaches, and a revolving door of departing leadership. In September, during a hearing by the House Committee on Veterans' Affairs, lawmakers learned about an unscheduled system failure that took down key applications in 17 VA medical facilities for a day.
Characterized by Dr. Ben Davoren, the director of clinical informatics for the San Francisco VA Medical Center, as "the most significant technological threat to patient safety the VA has ever had," the outage has moved some observers to call into question the VA's direction in consolidating its IT operations. Yet the shutdown grew from a simple change management procedure that wasn't properly followed.
The small, undocumented change ended up bringing down the primary patient applications at 17 VA medical centers in Northern California. As a result, the schedule to centralize IT operations across more than 150 medical facilities into four regional data processing centers has been pulled back while VA IT leaders establish what the right approach is for its regionalization efforts.
The Region 1 Field Operations breakdown of Aug. 31 exposed just how challenging effecting substantial change is in a complex organization the size of the VA Office of Information & Technology (OI&T). Begun in October 2005 and originally scheduled to be completed by October 2008, the "reforming" of the IT organization at the VA involved several substantial goals: the creation of major departments along functional areas such as enterprise development, quality and performance, and IT oversight and compliance; the reassignment of 6,000 technical professionals to a more centralized management; and the adoption of 36 management processes defined in the Information Technology Infrastructure Library (ITIL).
As part of the reform effort, the VA was to shift local control of IT infrastructure operations to regional data-processing centers. Historically, each of the 150 or so medical centers run by the VA had its own IT service, its own budget authority and its own staff, as well as independence with regard to how the IT infrastructure evolved. All of the decisions regarding IT were made between a local IT leadership official and the director of that particular medical center. While that made on-site IT staff responsive to local needs, it made standardization across sites nearly impossible in areas such as security, infrastructure administration and maintenance, and disaster recovery.
The operations of its 150 medical facilities would relocate to four regional data processing centers, two in the east and two in the west. The latter, Regions 1 and 2, are located in Sacramento, Calif., and Denver respectively, run as part of the Enterprise Operations & Infrastructure (OPS) office.
This pilot fish is a contractor at a military base, working on some very cool fire-control systems for tanks. But when he spots something obviously wrong during a live-fire test, he can't get the firing-range commander's attention.
- IT Certification Study Tips
- Register for this Computerworld Insider Study Tip guide and gain access to hundreds of premium content articles, cheat sheets, product reviews and more.
- Reduce federal infrastructure risk with compliance management and situational awareness
- IBM continuous monitoring and management solutions deliver real-time situational awareness to help federal agencies understand vulnerabilities, and protect the infrastructure.
- Building a New Era in Enterprise IT
- As the digital revolution spawned by the explosive growth of disruptive technologies such as mobile, cloud, social, and analytic tools continues to fundamentally...
- Mobile First: Securing Information Sprawl
- Learn how the partnership between Box and MobileIron can help you execute a "mobile first" strategy that manages and secures both mobile apps...
- AIIM Trendscape: The New Mobile Reality
- This AIIM Trendscape report shares data, expert opinions, and a unique perspective on the impact of cloud and mobility in the enterprise, surfacing...
- Empowering Your Mobile Workers
- A modern mobile IT strategy is no longer an option, it is an absolute necessity. Here's how some of the nation's most progressive... All Government IT White Papers
- NSS Labs & Cisco Present: Evaluating Leading Breach Detection Systems Today's constantly evolving advanced malware and APTs can evade point-in-time defenses to penetrate networks. Security professionals must evolve their strategy in lockstep to...
- Will the Real Endpoint Threat Detection and Response Please Stand Up? This webinar explores new technologies & process for protecting endpoints from advanced attackers as well as the innovations that are pushing the envelope...
- What should I look for in a Next Generation Firewall? SANS Provides Guidance With so many vendors claiming to have a Next Generation Firewall (NGFW), it can be difficult to tell what makes each one different....
- Why Are Customers Really Deploying an NGFW? It seems every IT Security expert is talking about the NGFW, but what are people really doing? This webcast covers 5 real-world customer...
- Charting Your Analytical Future - "Making predictive analytics part of your business processes" Webinar This session will show how predictive analytics can be used throughout the organization by anyone looking for answers and how organizations can make...
- All Government IT Webcasts