The improbable story of a New York City data center saved by a fuel bucket brigade is not told with fondness, even if there is a sense of pride and humor in its retelling.
Those involved recall the stink of diesel fuel in dark stairwells illuminated by flashlights. There was the sheer physical exhaustion of carrying buckets of diesel up 17 floors to deposit them into a rooftop generator's fuel tank. There was the sleeping on floors and the lack of showers, and the unrelenting stress of knowing that their efforts could suddenly unravel and the data center could fail.
And of course all involved had to live with the broader weight brought on by concern about the impact that Hurricane Sandy was having on their friends and relatives, and on New York itself.
The effort to save the data center during and after the October 2012 storm is captured in a documentary-style film that Peer 1, the data center owner, produced and showed Tuesday to a group of its customers and employees.
The film, featuring photos of data center rescue efforts and interviews with participants, doesn't attempt to romanticize the events or add a false sense triumph to the actions of those involved. It just lays them out, leaving viewers with a sense of sheer exhaustion as they watch the efforts of the participants and share their sense of uncertainty about the outcome.
Peer 1 posted the video on YouTube Wednesday night.
The concept of carrying fuel to the roof "seemed like a ridiculous idea," said Michael Pryor, president of Fog Creek Software, a large user of the Peer 1 data center, in an interview. "It just didn't seem feasible."
Pryor's idea, retold during the film, was to bring some extra pumps from a fish tank in his office to the nearby Peer 1 data center to see if they might help get fuel to the rooftop generator. During the screening, that suggestion drew healthy laughter from the audience, and Pryor himself now chuckles at the idea.
But the fact that he actually thought of using fish tank pumps helps convey the sense of urgency that the data center workers and customers experienced in the early going as they struggled to come up with ideas for keeping a rooftop generator running as it burned 40 gallons of fuel an hour out of a 200-gallon tank.
Fog Creek makes a software management platform used by developers, and the company and its customers would have suffered if the 5,000 square-foot data center at 75 Broad St. in lower Manhattan went down.
Peer 1 is a short walk from Fog Creek's offices, and as the software company grew so did its use of the data center. The data center provides Fog Creek's network and infrastructure for customer equipment.
There's an area set aside for Peer 1 customers to work on their systems.
Data center employee Jeffery Burns recalled that he heard the sound of a waterfall in the elevator shaft as floodwaters entered the lobby and eventually flooded the basement and disabled the fuel pumping systems.
"Everybody in Manhattan has the same problem, how to relocate infrastructure that -- for the last 50 years -- has safely lived in basements," said Michael Mazzei, manager of the Peer 1 Broad Street data center.
Despite the damage caused by Hurricane Sandy, data centers won't be leaving Manhattan anytime soon. The island is one of the most networked places in the world.
The Peer 1 data center is in the former International Telephone & Telegraph headquarters building and has multiple network connections that give customers quick access and low latency. It's also a short walk to the waterfront.
New York Mayor Michael Bloomberg this month announced a multibillion-dollar, multidecade plan to protect the city from powerful storms, and from the impact of climate change. The plan includes the construction of barriers in low-lying parts of the city.