A data center story for the ages: The fuel bucket brigade
The effort to save a NYC data center following Hurricane Sandy is remembered in a new film
Computerworld - The improbable story of a New York City data center saved by a fuel bucket brigade is not told with fondness, even if there is a sense of pride and humor in its retelling.
Those involved recall the stink of diesel fuel in dark stairwells illuminated by flashlights. There was the sheer physical exhaustion of carrying buckets of diesel up 17 floors to deposit them into a rooftop generator's fuel tank. There was the sleeping on floors and the lack of showers, and the unrelenting stress of knowing that their efforts could suddenly unravel and the data center could fail.
And of course all involved had to live with the broader weight brought on by concern about the impact that Hurricane Sandy was having on their friends and relatives, and on New York itself.
The effort to save the data center during and after the October 2012 storm is captured in a documentary-style film that Peer 1, the data center owner, produced and showed Tuesday to a group of its customers and employees.
The film, featuring photos of data center rescue efforts and interviews with participants, doesn't attempt to romanticize the events or add a false sense triumph to the actions of those involved. It just lays them out, leaving viewers with a sense of sheer exhaustion as they watch the efforts of the participants and share their sense of uncertainty about the outcome.
Peer 1 posted the video on YouTube Wednesday night.
The concept of carrying fuel to the roof "seemed like a ridiculous idea," said Michael Pryor, president of Fog Creek Software, a large user of the Peer 1 data center, in an interview. "It just didn't seem feasible."
Pryor's idea, retold during the film, was to bring some extra pumps from a fish tank in his office to the nearby Peer 1 data center to see if they might help get fuel to the rooftop generator. During the screening, that suggestion drew healthy laughter from the audience, and Pryor himself now chuckles at the idea.
But the fact that he actually thought of using fish tank pumps helps convey the sense of urgency that the data center workers and customers experienced in the early going as they struggled to come up with ideas for keeping a rooftop generator running as it burned 40 gallons of fuel an hour out of a 200-gallon tank.
Fog Creek makes a software management platform used by developers, and the company and its customers would have suffered if the 5,000 square-foot data center at 75 Broad St. in lower Manhattan went down.
Peer 1 is a short walk from Fog Creek's offices, and as the software company grew so did its use of the data center. The data center provides Fog Creek's network and infrastructure for customer equipment.
There's an area set aside for Peer 1 customers to work on their systems.
Data center employee Jeffery Burns recalled that he heard the sound of a waterfall in the elevator shaft as floodwaters entered the lobby and eventually flooded the basement and disabled the fuel pumping systems.
"Everybody in Manhattan has the same problem, how to relocate infrastructure that -- for the last 50 years -- has safely lived in basements," said Michael Mazzei, manager of the Peer 1 Broad Street data center.
Despite the damage caused by Hurricane Sandy, data centers won't be leaving Manhattan anytime soon. The island is one of the most networked places in the world.
The Peer 1 data center is in the former International Telephone & Telegraph headquarters building and has multiple network connections that give customers quick access and low latency. It's also a short walk to the waterfront.
New York Mayor Michael Bloomberg this month announced a multibillion-dollar, multidecade plan to protect the city from powerful storms, and from the impact of climate change. The plan includes the construction of barriers in low-lying parts of the city.
- Capabilities You Need in an IP Address Management Solution A mismanaged IP space can cripple an otherwise healthy network. Take a moment to understand what you need in an enterprise-ready IPAM solution.
- Optimize IT Performance & Availability: Four Steps to Establish Effective IT Management Baselines More than ever before, your company's ability to grow hinges on IT performance and availability. Download this how-to report on establishing IT baselines,...
- Considerations for Embracing Wireless Monitoring Employee behavior is once again driving major changes for IT departments - this time it's BYOD. This report details three critical steps to...
- Fixing Intermittent Performance Problems Intermittent performance problems are among the most frustrating and time-consuming issues IT administrators face. Read this white paper and learn how technology advances...
- Accelerate your innovation with IBM Bluemix™ Join us for a webcast introducing the new IBM BluemixTM. IBM Bluemix (www.bluemix.net) is a developer oriented Platform as a Service (PaaS) environment...
- Maximizing Availability for the Modern Data Center Check out this information-packed resource center for help in maximizing the availability of your data center - from overcoming challenges to choosing the... All Data Center White Papers | Webcasts
Our new bimonthly Internet of Things newsletter helps you keep pace with the rapidly evolving technologies, trends and developments related to the IoT. Subscribe now and stay up to date!