A data center story for the ages: The fuel bucket brigade

The effort to save a NYC data center following Hurricane Sandy is remembered in a new film

1 2 Page 2
Page 2 of 2

Bloomberg's plan has broad IT and telecommunications implications.

It seeks a higher level of reliability and service for telecommunication facilities. The specific recommendations include requiring new hospitals "to increase their IT and telecommunications resiliency by installing two independent points-of-entry for telecom and communication to reduce the risk of outages from a single supplier."

For Fog Creek, and likely many other businesses, the most significant long-term impact of Hurricane Sandy may stem from the fact that it caught the company without a backup plan. Fog Creek has been working to fix that weakness in the months since.

For instance, the software company has improved its data replication capabilities and is developing an ability to switch operations to a separate facility in an emergency. One tool now runs in Amazon's cloud service, though that migration had been planned prior to the storm.

The storm made clear to Fog Creek that it hadn't paid enough attention to the likelihood that something bad, like a massive storm, could ever happen, said Pryor. That was a positive outcome of the experience, he said, "because it did kick us in the butt and get us to fix a lot of things that were broken."

In his office, Mazzei points to the part of the floor he slept on -- next to his phone.

The bucket brigade wasn't the group's first move. Initially, they tried hauling 55-gallon drums up the 17 flights of stairs, but the physical exertion proved too much.

Gradually, though, a realistic plan came into place. The team acquired pumps, hoses, buckets and other supplies from Home Depot and other places. Organization, rules and efficiency followed.

One rule: "Go slowly, work safely, always have your flashlight," said Mazzei.

For 72 hours, the effort continued and the data center was able to remain in operation until a more stable fuel supply was established. For the future, one option the building owners are considering is installing submersible pumps that could operate during a flood, said Mazzie.

There were a lot of motivations at work in manning the bucket brigade. For customers, it was about keeping their businesses up. For the data center workers, it was about not failing.

No one is really taking credit for saving the data center.

Mazzei is very matter-of-fact about the effort, and describes it more as an organic process that developed on its own as ideas and processes emerged collectively.

Burns, a film editor who works part time at the data center, said the team rallied around Mazzei. "It was just really important to not let him down," he recalled, "not because he was going to get upset, but because he never let us down."

This article, "A Data Center Story for the Ages: The Fuel Bucket Brigade," originally appeared on Computerworld.com.

Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov, or subscribe to Patrick's RSS feed . His email address is pthibodeau@computerworld.com.

Copyright © 2013 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon