There's real-time drama unfolding at one of the data centers operating at 111 8th Ave. in NYC, a Google-owned building that occupies a full city block.
There are a number of data centers housed in the nearly 3 million square foot building that functions as a carrier hotel, a major connection point for hundreds of domestic and international telecommunications providers.
The building, along with all of lower Manhattan, has been on generator power since Con Edison cut service to the area Monday night because of floods caused by Hurricane Sandy.
The loss of power has exposed generator problems at multiple locations around lower Manhattan, including at 111 8th Ave.
Data center operator Zayo uses some 26,500 square feet in the building, according to its Website. Thursday morning it published an "urgent communication regarding the status of the 111 8th Avenue facility."
The drama has unfolded, so far, in a series of three communications by Zayo to its customers with updates on the company's various efforts to address problems that have come as the data center temperature went up and down.
Zayo is posting all of its emergency network updates here.
In its first missive, Zayo told its customers that the temperature in one of its data center suites had reached 93 degrees. The company blamed "technical difficulties with the fuel pumping system to a subset of generators within the building, affecting the AC power."
In a later notice, published early this afternoon, Zayo said that temperatures in parts of the data center "have risen above 100 degrees."
It followed that with a note published after 2 p.m. Thursday reporting that the generator serving its data center suite "was taken off-line to implement a temporary fix." Workers hoped to stabilize the fuel pump and flow to the generator "in an attempt to return portion of cooling online," the note said.
In its latest posting, the data center operator said, "All Zayo network and customer equipment successfully transitioned to Zayo's DC battery plant with no interruption to customers. Zayo's DC plant provides for an estimated 5 hours run-time should the generator have to be taken off-line for maintenance or if there should be a failure.
"At approximately 2:12pm EST, the generator was reengaged and teams are assessing stability to bring some portion of cooling back on-line. With mitigation efforts temperatures have stabilized in a range of 95-100 degrees Fahrenheit and have been generally unchanged over past 2 hours at those levels," Zayo said in the post.
Zayo said it also has other contingencies in motion, including a 2MW generator that is in transit from New Jersey to the facility in New York City.
"If building infrastructure is not restored, Zayo's intention would be to tie its 7th floor services and cooling infrastructure directly to this back-up power source and maintain this through full commercial utility restoration," the company said.
Delivery of the generator is expected to take 2.5 to 3.5 hours depending on traffic.
"We have not had any customer outages due to this," said Matthew Brouker," a Zayo spokesman, "All systems are running."
The company is providing ongoing conference calls with its customers with its CTO, Marty Snella, who was answering questions, and assuring them that it had hours of battery backup if the generator fails or is taken off-line.
Vince Renaud, vice president of of Uptime Institute Professional Services, in response to a query about high temperatures in a data center, said, an email response, that that American Society of Refrigeration and Air Conditioning Engineers (ASHRAE) recommends that temperatures in data cneters not exceed 80.6 degrees Fahrenheit.
"There is much debate about going into the allowable range of anything above 80 (degrees) as voiding warranties by equipment manufacturers," said Renaud.
He said that "93 degrees will produce catastrophic results for computer equipment or, if no immediate affects - then will result in 'wounded servers' that will fail later on when you least expect it."
Telx, which also operates a data center at 111 8th Ave., said, in a posted note, that as of 1 p.m. ET, there haven't been unusual temperature increases in its data center. Telx is asking customers of its two data centers in NYC, the other is at 60 Hudson St., to assist in reducing electrical load and curtail their systems where possible and to "power down any nonessential equipment at this time."
In an email to Computerworld, a Telx official said its data center isn't having problems similar to that of Zayo, suggesting that the latter's issue could be specific to it or to its infrastructure.
Telx is not expecting Con Edison to restore power to the building for three or four days.
Another data center company operating at 111 8th Ave., Equinix, reported Wednesday that it had a problem a failed generator "that impacted service to several customers."
Equinix said "repairs were made and service was restored within one hour and 15 minutes."
Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov, or subscribe to Patrick's RSS feed . His e-mail address is email@example.com.