From near disaster to shaky recovery in NYC
Data centers suffer outages but survive as they wait for utility power
Computerworld - The classic quote from Warren Buffett about a financial crisis is also true for data centers. It's "only when the tide goes out do you discover who's been swimming naked," the multibillionaire businessman said.
Hurricane Sandy brought the tide in, literally. But the flooding in lower Manhattan had the same effect on data centers that, as Buffet observed, a recession has on financial services. It exposed the problems and shortcomings in the system.
There were data center outages. The causes range from flooded buildings that cut data centers off from fuel supplies, to backup generators that malfunctioned. The problems were extensive and the responses were often dramatic.
Zayo, which runs a data center in one of the most wired buildings in New York City, at 111 8th Ave., sent an "urgent communication" to customers Thursday. The temperature in one of its data center suites had reached 93 degrees Fahrenheit. Another communication soon followed, reporting that temps had "risen above 100 degrees."
Zayo conducted continuous conference calls for customers, detailed its multiple paths to a fix, including estimating how long it would take to get a 2MW backup generator delivered through New York traffic. The problem was a result of a failed isolation valve in the fuel system supplying a generator. It was fixed without any customer outages, and temperatures returned to 70 degrees.
In one case, at 75 Broad St., the response was just amazing.
Peer1, a data center that operates at 75 Broad St., was close to shutting down its data center after water flooded a basement, cutting off access to generator fuel supplies. That's when the employees of two customers of Peer1, Squarespace and Fog Creek Software, stepped in to carry fuel via a bucket brigade, up 17 flights.
There were many other reports of data center problems, particularly after the initial storm impact. Early in the week, Internap reported a data center outage at 111 8th Ave. due to a building-fed fuel system problem. Equinix reported a failed generator that "impacted service to several customers." The problems were eventually fixed at both sites.
New York's problems affected customers far and wide. Web sites were inaccessible and some networks were down.
TDS Telecommunications said it had experienced a "double failure" on two diverse routes of its network that go through New York. One source of trouble stemmed from a third-party data center outage in New York, and a second problem concerned an AT&T cable issue. About 9,000 data customers in New Hampshire were affected by the problem, since fixed.
"It's interesting," said DeAnne Boegli, TDS spokeswoman, "even when you have two options going into opposite sides of the city, they both get impacted."
Earlier in the week, monitoring firm Renesys, said the hurricane had caused outages in 10% of the networks in New York.
"Silencing ten percent of the networks in the New York area is like taking out an entire country the size of Austria, in terms of impact on the global routing table," wrote Renesys CTO James Cowie, on a blog.
It's too early to write a post mortem. The lower Manhattan data centers aren't expecting utility power until Sunday or Monday. Until then, the anxiety level will remain above normal, as data centers rely on generators, fuel trucks and ingenuity.
Patrick Thibodeau covers cloud computing and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov or subscribe to Patrick's RSS feed . His e-mail address is email@example.com.
- Pennsylvania, a 'Fortune 20' state, consolidates IT
- EPA urges efficiency, many data centers still far from it
- HP creates a new way to sell data centers
- The background on Apple's '103-degree data center'
- Network firewalls aren't dead yet
- Are CIOs losing power?
- As Unix fades away from data centers, it's unclear what's next
- Goldman Sachs, with 10,000 tech workers, embraces open computing
- IT managers are increasingly replacing servers with SaaS
- Microsoft sees huge potential in fuel cells
Read more about Data Center in Computerworld's Data Center Topic Center.
- The Truth About Virtual Computing for CAD If you're a user of graphics-intensive software such as 3D modeling, simulation and analysis, and visualization, you might be skeptical about moving to...
- Enable secure remote access to 3D data without sacrificing visual perfomance Design and manufacturing companies must adapt quickly to the demands of an increasingly global and competitive economy. To speed time to market for...
- Magic Quadrant for Data Masking Technology IBM is a leader in Gartner Inc's Magic Quadrant for Data Masking Technology. Read the full report to learn about IBM.
- 5 Ways Dropbox for Business Keeps Your Data Protected Protecting your data isn't a feature on a checklist, something to be tacked on as an afterthought. Download here to find out how...
- The Key to Happiness: Throw out Your Data Warehouse In this webinar, Kerry Reitnauer, Director, Solution Architect at FairPoint Communications will discuss the challenges the data warehouse brought, how they migrated to...
- Building Tomorrow's Data Center with Converged Technologies A number of forces are converging: the cloud, converged infrastructure, big data and fabric architectures to name a few. All Data Center White Papers | Webcasts
Our new bimonthly Internet of Things newsletter helps you keep pace with the rapidly evolving technologies, trends and developments related to the IoT. Subscribe now and stay up to date!