White House CIO Endures After Tough Start

Brook Colangelo had been on the job just six days when the email servers went down for 21 hours.

In Brook Colangelo's first 40 days as CIO of the Executive Office of the President, the White House email system was down 23% of the time while he and his staff put in 80-hour weeks, "if not more."

Colangelo began the job on Jan. 20, 2009 -- the day Barack Obama was inaugurated as president. His opening tasks were to "deliver the first presidential BlackBerry" and distribute handhelds to all top administration officials. "It was just a mind-blowing experience," he said at Computerworld's Premier 100 IT Leaders Conference in Phoenix this month.

But reality quickly set in as Colangelo found White House IT assets "in pretty bad shape."

Over 82% of the equipment had reached the end of its useful life. Many desktop computers still had floppy disk drives.

Colangelo also noted that he oversaw just one data center. "We had no redundancy," he said.

The problem became apparent after six days, when, on Jan. 26, "our email servers went down for 21 hours," said Colangelo. "In my professional career, there has not been a worse day since or ever."

Colangelo's luck changed the next morning when he was walking to a 5:30 a.m. meeting to discuss the outage with then White House Chief of Staff Rahm Emanuel. "The most amazing thing happened as [I] hit the door of the West Wing: My BlackBerry started to buzz," meaning the email was back up, he said. "I got to tell you, it was the best feeling I ever had."

The White House faced three or four more outages in the next 30 days or so, and Colangelo launched a massive review of its technology, people and processes.

Internal town halls drew some angry users. "They had floppy drives -- I knew what they were going to say," Colangelo said.

A nine-hour email and Internet access outage in February 2011 forced Colangelo to fax updates to Obama while the president was on the road -- "not a great thing to do." That incident led to the creation of a data recovery data center with redundant email servers.

The White House IT office has since developed a Web-based portal that lets staff access email and other services from home "in a secure and records-managed way," and it has increased daily data center coverage from eight hours to 24.

The office has also rolled out support for personal tablets and smartphones, increased Internet speeds, rebuilt the WhiteHouse.gov website and cut the number of assets that are nearing obsolescence by more than 50%.

"Our modernization program was very successful," said Colangelo.

Shawn McCarthy, an analyst at IDC who focuses on the government market, said that "government is famous for being both a leader and a laggard when it comes to information technology."

Some governmental groups and agencies, such as NASA, are ahead in technology adoption, while budget worries may force smaller departments to stretch PCs and other IT equipment a few extra years, said McCarthy.

Lack of backup systems and mirroring is occasionally an issue for noncritical government business systems, McCarthy added.

That problem is being addressed through the Obama administration's consolidation effort, which includes the addition of data backup and other protections, he said.The public sector generally does a lousy job supporting disaster recovery, said Gartner analyst John Kost, explaining that "the business case for some projects excludes it in order to keep costs down to get the project funded."

This version of this story was originally published in Computerworld's print edition. It was adapted from an article that appeared earlier on Computerworld.com.

Copyright © 2012 IDG Communications, Inc.

Bing’s AI chatbot came to work for me. I had to fire it.
Shop Tech Products at Amazon