The VA's computer systems meltdown: What happened and why

Not following best practices can render the best technology useless

1 2 Page 2
Page 2 of 2

Raffin said that iI was never the intent of the realignment to downgrade the level of service experienced by people in the medical facilities. "The message I send to my folks in my organization is, 'You may work ultimately for me within OI&T, but you absolutely work for the network or facility where you're stationed,'" he said. The goal, Raffin said, was to create "that bench strength we've never had."

As an example, Raffin points to a coding compliance tool, an application that exists at all 33 medical centers in his jurisdiction that all run the same version on the same system. "There was a sliver of a [full-time employee] at every medical center that was supporting this application," he said. "There was no structure [for] maintenance and upgrades, no coordination in how we handled problem management." When a problem surfaced, 33 trouble tickets would be logged, Raffin said.

As part of the reorganization, Region 1 has set up a systems team, which includes an applications group. Two people within that group are now coordinating the management of that particular application. "It's a team approach," he said.

Likewise, similar to the argument made by companies that move employees to a service provider during an outsourcing initiative, Raffin claimed that the reassignment of personnel to an organization dedicated to IT will ultimately result in greater opportunities for them and better succession planning for OI&T.

"The only competing interest I have with regards to training are other IT folks, who need other IT training," he said. "I'm not competing with nursing education or with folks who need safety education because they operate heavy machinery at a medical center."

Along the way, that training includes an education in change management process, one of the ITIL best practices being adopted by OI&T that was "new to our IT folks," said Raffin. "They may have read it, but I'm not sure they got it."

Dr. Paul Tibbits is deputy CIO for Enterprise Development -- one of the newly created functional areas within OI&T. Tibbits pointed out that under the previous management structure, "there would have been a lot of competition for mental energy on the part of a hospital director. Does he get his IT staff to read this stuff or not read this stuff?" Under a single chain of command, that education will most assuredly take place, he said.

Tibbits' organization is taking a different approach from Region 1 in how it develops staff skills in the four ITIL processes under his charge. The three-phased approach he described involves real-time coaching and mentoring for "short-term change"; classes, conferences and workshops for midterm change; and updates in recruiting practices for the long term.

"We're hiring outside contractors to stand at the elbows and shoulders of our IT managers through the development organization to watch what they do on a day-by-day basis," said Tibbits. That effort has just begun, he said, with contractors "just coming on board now."

On the other hand, Region 1 under Raffin's leadership has introduced a three-part governance process. The first part is a technical component advisory council, which meets weekly to discuss and prioritize projects. "That is where a lot of training has occurred," said Raffin. Second, a regional governance board also meets weekly to discuss issues related to IT infrastructure. In addition, Raffin is about to implement a monthly meeting of an executive partnership council that will include both IT people and "business" representatives from the medical facilities being served.

Will bringing people together for meetings suffice to meet the needs of transforming the work habits of the 4,000 people who are now part of OPS -- what Tibbits classifies as a "workforce-reshaping challenge?" And will it prevent the kind of outage that happened on Aug. 31 from happening again somewhere else?

Tibbits sweeps aside a suggestion that the centralization of IT played a role in the outage. "Had the IT reorganization never happened, this error might have happened on Aug. 31 anyway because somebody didn't follow a procedure," he said.

Forrester's Hubbert sees the value in bringing together teams within IT to look at operations more holistically. "That's what change agents need to do -- to lay IT on its side instead of keeping it in silos ... to have that end-to-end picture," she said. Plus, that's an effective way to address shortfalls in process and bring staff along as part of the overall transformation effort, Hubbert adds. "Usually, if you take IT people into the boat and ask them what to fix, if you say, 'Hey, this is the whiteboard. Let's figure it out from there all the way back to the root cause,' they have a real willingness to cooperate," she said. From there, they can develop a process to prevent the same type of problem from surfacing again.

Region 1 Fallout

When an event takes place that impairs the operations of 17 federally funded medical centers, investigations and reviews tend to follow. In the case of Region 1, that includes an internal review of the regional data processing initiative by both the IT and Oversight & Compliance and Information Protection and Risk Management organizations, which report to Gen. Bob Howard, assistant secretary for OI&T, as well as a review coordinated by an unnamed outside firm. Raffin said he expects those reviews to be concluded early in 2008. And although that review was actually scheduled as part of the OI&T's spending plan, he acknowledged that "it's happening a little earlier than we wanted it to."

Until those results are in, the OI&T has put a "soft hold" on migrating additional medical centers into the regional data center concept, said Raffin. "From Region 1's perspective, we were almost 90% complete and should have been 100% complete by Nov. 9. Our project schedule is going to be a little delayed," he said.

Also, Howard has directed the OI&T development organization to work with the infrastructure engineering organization to design a series of system topologies that would provide varying degrees of reliability, availability, maintainability and speed, "up to and including one option that would be 'zero downtime,'" Tibbits said. "I don't think there's any question in anyone's mind that 128 data centers is too many. One might be too few. But what exactly the optimal topology is, all of that is in play right now. Regionalization of some form is alive and well and will move forward."

Region 1 has experienced a dramatic improvement in compliance, Raffin said, "with folks documenting changes in advance of their occurrence." The next phase of that will be an automated system using tools from CA Inc., which are already in use in the VA's Austin Automation Center. He expects that to be implemented within 90 days.

Region 1 has also modified procedures related to the read-only version of records maintained by Vista, the Level 2 backup plan that wasn't fully available on Aug. 31. Now, Raffin said, those systems are more consistently checked for round-the-clock availability and "any system maintenance ... is properly recorded through our change management procedure."

According to Davoren, the medical director in San Francisco, "before regionalization of IT resources -- with actual systems that contained patient information in distributed systems -- it would have been impossible to have 17 medical centers [go] down." As he told a congressional committee in September, the August system outage was "the longest unplanned downtime that we've ever had at San Francisco since we've had electronic medical records." This was proof to Davoren and others at the individual medical centers that in creating a new structure "in the name of 'standardization,'" support would "wane to a lowest common denominator for all facilities," he said.

Raffin isn't ready to give up. He recognizes that an event like the one that happened on Aug. 31 "casts a long shadow" against what he sees as a number of accomplishments. But he also maintains confidence that Region 1 -- and all of OI&T -- has the ability to pull off its transformation. "For me, it's about making sure we're listening to all of our folks and have our ears to the pavement at the medical centers to make sure we understand what our business requirements are," he said.

Change is hard, especially when it's undertaken on such a massive scale. The difficulty was foreseen early on by VA CIO Howard. "This will not be an easy or quick transformation. There will be a few difficulties along the way, and it's natural for some people to be uncomfortable with change on such a scale. But the prospect of more standardization and interoperability we can harness through this centralization is exciting," Howard said in a webcast speech to the IT workforce of the VA shortly after his confirmation hearings by the Senate Committee on Veterans Affairs.

A question remains whether the VA OI&T is moving quickly enough to keep the confidence of its numerous constituencies -- patients, medical staff, VA executives and lawmakers. As U.S. Rep. Bob Filner (D-Calif.), chair of the House Committee on Veterans' Affairs, stated during that September hearing, "We are heartened by many of the steps the VA has undertaken, but remain concerned that more should be done, and could be done ... faster."

Dian Schaffhauser is a writer who covers technology and business for a number of print and online publications. Contact her at

Copyright © 2007 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon