From near disaster to shaky recovery in NYC
Data centers suffer outages but survive as they wait for utility power
Computerworld - The classic quote from Warren Buffett about a financial crisis is also true for data centers. It's "only when the tide goes out do you discover who's been swimming naked," the multibillionaire businessman said.
Hurricane Sandy brought the tide in, literally. But the flooding in lower Manhattan had the same effect on data centers that, as Buffet observed, a recession has on financial services. It exposed the problems and shortcomings in the system.
There were data center outages. The causes range from flooded buildings that cut data centers off from fuel supplies, to backup generators that malfunctioned. The problems were extensive and the responses were often dramatic.
Zayo, which runs a data center in one of the most wired buildings in New York City, at 111 8th Ave., sent an "urgent communication" to customers Thursday. The temperature in one of its data center suites had reached 93 degrees Fahrenheit. Another communication soon followed, reporting that temps had "risen above 100 degrees."
Zayo conducted continuous conference calls for customers, detailed its multiple paths to a fix, including estimating how long it would take to get a 2MW backup generator delivered through New York traffic. The problem was a result of a failed isolation valve in the fuel system supplying a generator. It was fixed without any customer outages, and temperatures returned to 70 degrees.
In one case, at 75 Broad St., the response was just amazing.
Peer1, a data center that operates at 75 Broad St., was close to shutting down its data center after water flooded a basement, cutting off access to generator fuel supplies. That's when the employees of two customers of Peer1, Squarespace and Fog Creek Software, stepped in to carry fuel via a bucket brigade, up 17 flights.
There were many other reports of data center problems, particularly after the initial storm impact. Early in the week, Internap reported a data center outage at 111 8th Ave. due to a building-fed fuel system problem. Equinix reported a failed generator that "impacted service to several customers." The problems were eventually fixed at both sites.
New York's problems affected customers far and wide. Web sites were inaccessible and some networks were down.
TDS Telecommunications said it had experienced a "double failure" on two diverse routes of its network that go through New York. One source of trouble stemmed from a third-party data center outage in New York, and a second problem concerned an AT&T cable issue. About 9,000 data customers in New Hampshire were affected by the problem, since fixed.
"It's interesting," said DeAnne Boegli, TDS spokeswoman, "even when you have two options going into opposite sides of the city, they both get impacted."
Earlier in the week, monitoring firm Renesys, said the hurricane had caused outages in 10% of the networks in New York.
"Silencing ten percent of the networks in the New York area is like taking out an entire country the size of Austria, in terms of impact on the global routing table," wrote Renesys CTO James Cowie, on a blog.
It's too early to write a post mortem. The lower Manhattan data centers aren't expecting utility power until Sunday or Monday. Until then, the anxiety level will remain above normal, as data centers rely on generators, fuel trucks and ingenuity.
Patrick Thibodeau covers cloud computing and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov or subscribe to Patrick's RSS feed . His e-mail address is email@example.com.
- Network firewalls aren't dead yet
- Are CIOs losing power?
- As Unix fades away from data centers, it's unclear what's next
- Goldman Sachs, with 10,000 tech workers, embraces open computing
- IT managers are increasingly replacing servers with SaaS
- Microsoft sees huge potential in fuel cells
- Snap, crackle, pop: Electrical troubles shed light on NSA's data center
- Tornadoes and data centers are OK in Oklahoma
- It's twilight for small in-house data centers
- GM CIO calls HP hiring probe a 'fishing expedition'
Read more about Data Center in Computerworld's Data Center Topic Center.
- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- HP HAVEn: See the big picture in Big Data HP HAVEn is the industry's first comprehensive, scalable, open, and secure platform for Big Data. Enterprises are drowning in a sea of data...
- Piecing Together the Business Intelligence Puzzle Business intelligence (BI) technology collects and analyzes company data, delivering relevant information to corporate decision-makers in an effort to produce favorable outcomes.
- Harness IT -- An Introduction to Business Intelligence Solutions Learn the key selection criteria required to provide your organization with the capability to address structured data, unstructured data and mobile demands so...
- Business Intelligence Shows its Smarts Today's Business Intelligence (BI) tools provide a new way to think about data with self-service capabilities and user-friendly analytics that can be used...
- Cloud Knowledge Vault Learn how your organization can benefit from the scalability, flexibility, and performance that the cloud offers through the short videos and other resources...
- Testimonial: Cystic Fibrosis Trust Peter Hawkins, the Head of IT for Cystic Fibrosis Trust, discusses the role CommVault's Simpana software platform plays in improving the company's information... All Data Center White Papers | Webcasts