Huge customer effort keeps flooded NYC data center running

How do you get hundreds of gallons of fuel up 17 stories? Your customers carry it

1 2 Page 2
Page 2 of 2

"It's kind of just very dramatic," said Casalena. But they managed thanks to a "huge group effort," he said.

"It looks like we have operationalized this to the point where we can make it work -- I can't honestly believe it," said Casalena, "It somehow worked."

They now have enough fuel on the roof to stagger deliveries.

"Morale was super high," said Joel Spolsky, the CEO and co-founder of Fog Creek Software, who runs a number of businesses out of the Peer1 data center. "People were just absolutely itching to get there and help and do something and make some kind of contribution."

"It was hard, but people were kind of excited to have a mission to work on," said Spolsky, who is also co-founder of StackExchange, a major site for developers. That operation was moved to a data center in Oregon, but Spolsky has other work in the Peer1 center that can't be easily moved.

Spolsky, who tweeted about the efforts, said he even had offers from customers and people on Twitter to help out. "I think people feel kind of helpless sitting at home watching the news, and they were so excited to be able to do something even if it was hard to make a substantial contribution," he said.

The efforts continued to restore the basement pumping system. Water kept finding its way into the basement, but Casalena said that as of early Wednesday evening, it appeared the basement flooding situation was getting under control.

Both Casalena and Spolsky praised the Peer1 New York operations staff for their efforts. Peer1, which has 19 data centers worldwide, has extra help arriving tonight, driving up from a Virginia because of problems getting flights to New York.

Along with the efforts to get the building's pumping system in operation, Robert Miggins, senior vice president of business development at Peer1, said the company is working on alternative methods to pump fuel up to the roof.

Miggins said the data center remains online, in part, because the company didn't keep the situation it faced in New York a secret. "We choose to tell customers all the details and that is what led them to show up and enable them to lend a hand," he said.

Without that notice, "we wouldn't have had the manpower there to actually bring the fuel up in time," Miggins said. "There's a lot of good will, and there's a lot of hard work and there's a few lucky bounces for good measure," he said.

No one is going to feel comfortable until power is fully restored. Spolsky said his staff has been marking the company's physical systems to speed reassembly at another data center if the need arises.

If there's any takeaway from this experience, it may be this: Casalena said he's glad he went to the data center when he did to help and to take an active role. Otherwise, "I don't know how this would have played out," he said.

Patrick Thibodeau covers cloud computing and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov or subscribe to Patrick's RSS feed . His e-mail address is pthibodeau@computerworld.com.

See more by Patrick Thibodeau on Computerworld.com.

Copyright © 2012 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
8 simple ways to clean data with Excel
  
Shop Tech Products at Amazon