Drama in NYC as data center temp passes 100 degrees

Computerworld - There's real-time drama unfolding at one of the data centers operating at 111 8th Ave. in NYC, a Google-owned building that occupies a full city block.

There are a number of data centers housed in the nearly 3 million square foot building that functions as a carrier hotel, a major connection point for hundreds of domestic and international telecommunications providers.

The building, along with all of lower Manhattan, has been on generator power since Con Edison cut service to the area Monday night because of floods caused by Hurricane Sandy.

The loss of power has exposed generator problems at multiple locations around lower Manhattan, including at 111 8th Ave.

Data center operator Zayo uses some 26,500 square feet in the building, according to its Website. Thursday morning it published an "urgent communication regarding the status of the 111 8th Avenue facility."

The drama has unfolded, so far, in a series of three communications by Zayo to its customers with updates on the company's various efforts to address problems that have come as the data center temperature went up and down.

Zayo is posting all of its emergency network updates here.

In its first missive, Zayo told its customers that the temperature in one of its data center suites had reached 93 degrees. The company blamed "technical difficulties with the fuel pumping system to a subset of generators within the building, affecting the AC power."

In a later notice, published early this afternoon, Zayo said that temperatures in parts of the data center "have risen above 100 degrees."

It followed that with a note published after 2 p.m. Thursday reporting that the generator serving its data center suite "was taken off-line to implement a temporary fix." Workers hoped to stabilize the fuel pump and flow to the generator "in an attempt to return portion of cooling online," the note said.

In its latest posting, the data center operator said, "All Zayo network and customer equipment successfully transitioned to Zayo's DC battery plant with no interruption to customers. Zayo's DC plant provides for an estimated 5 hours run-time should the generator have to be taken off-line for maintenance or if there should be a failure.

"At approximately 2:12pm EST, the generator was reengaged and teams are assessing stability to bring some portion of cooling back on-line. With mitigation efforts temperatures have stabilized in a range of 95-100 degrees Fahrenheit and have been generally unchanged over past 2 hours at those levels," Zayo said in the post.

Zayo said it also has other contingencies in motion, including a 2MW generator that is in transit from New Jersey to the facility in New York City.