Huge customer effort keeps flooded NYC data center running
How do you get hundreds of gallons of fuel up 17 stories? Your customers carry it
Computerworld - When seawater flooded streets in lower Manhattan Monday night, it filled the bottom floors of the basement of an office building at 75 Broad St. and its lobby up to four feet. This was not a good development for the data center operated by Peer1 Hosting.
Peer1's data center had smoothly switched over to generator power, but when Con Edison cut electric power to lower Manhattan, the rooftop generator couldn't access the 20,000 gallon fuel tank in the flooded basement. Its pumping system was disabled by storm waters from Hurricane Sandy.
The rooftop generator was using as much as 40 gallons of fuel an hour to run the nearly 13,000-square-foot data center, and the "day tank" had limited capacity. Time was running out.
Early Tuesday, Peer1's Broad Street customers were warned that fuel supplies would soon be exhausted. Among those who got the email was Anthony Casalena, the founder and CEO of Squarespace, a Web publishing and content management firm. He was in his powerless apartment in downtown Manhattan.
Casalena started Squarespace in 2003 with $30,000 borrowed from his father. Today, his company has 102 employees and hundreds of thousands of customers. As a private company, it doesn't give out exact numbers.
Casalena has been using the Peer1 data center since his business started. He was familiar with its operation and may now be its largest customer. He left his downtown apartment to see for himself what was happening, to offer help, and hopefully, he said, to delay any shut down "until the last possible second." His customers were warned and he continued to report on the status of the data center.
Casalena said it quickly became clear that the data center operations team didn't have a good way to measure the fuel supplies in the rooftop day tank. "They don't really know how much fuel was left," he said.
Near the time the data center was supposed to shut down, Casalena said the data center team determined that there was a half tank left. That triggered a new plan.
Casalena said there were some 55-gallon oil drums onsite. The data center manager suggested trying to get fuel supplies to the roof. They made arrangements with one of the fuel trucks on the street for supply. Fuel trucks have become a common sight since the power outage.
Casalena was among those helping to carry fuel to the roof.
The generator-fueling effort got organized. More people begin showing up from Squarespace and another large user, Fog Creek Software. They included software developers, systems engineers, sales representatives, executives and support personnel.
A network of people, stationed at every staircase landing, formed a bucket brigade passing fuel from one person to the next. Several dozen may have been involved over multiple shifts, including some day laborers who were also hired to help.
"Over the next night into the morning, we were able to successfully continue to coordinate fuel trucks coming and manually move hundreds of gallons of diesel to the roof and keep the thing online," Casalena said.
There were moments when they felt they weren't moving fast enough. At one point, a generator light came on showing low fuel.
- Network firewalls aren't dead yet
- Are CIOs losing power?
- As Unix fades away from data centers, it's unclear what's next
- Goldman Sachs, with 10,000 tech workers, embraces open computing
- IT managers are increasingly replacing servers with SaaS
- Microsoft sees huge potential in fuel cells
- Snap, crackle, pop: Electrical troubles shed light on NSA's data center
- Tornadoes and data centers are OK in Oklahoma
- It's twilight for small in-house data centers
- GM CIO calls HP hiring probe a 'fishing expedition'
- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- HP HAVEn: See the big picture in Big Data HP HAVEn is the industry's first comprehensive, scalable, open, and secure platform for Big Data. Enterprises are drowning in a sea of data...
- Piecing Together the Business Intelligence Puzzle Business intelligence (BI) technology collects and analyzes company data, delivering relevant information to corporate decision-makers in an effort to produce favorable outcomes.
- Harness IT -- An Introduction to Business Intelligence Solutions Learn the key selection criteria required to provide your organization with the capability to address structured data, unstructured data and mobile demands so...
- Business Intelligence Shows its Smarts Today's Business Intelligence (BI) tools provide a new way to think about data with self-service capabilities and user-friendly analytics that can be used...
- Cloud Knowledge Vault Learn how your organization can benefit from the scalability, flexibility, and performance that the cloud offers through the short videos and other resources...
- Testimonial: Cystic Fibrosis Trust Peter Hawkins, the Head of IT for Cystic Fibrosis Trust, discusses the role CommVault's Simpana software platform plays in improving the company's information... All Data Center White Papers | Webcasts