Botched Data Center Move Takes Out 165k Web Sites

A cascading series of problems with a data center consolidation project at Web hosting company NaviSite Inc. left about 165,000 Web sites offline last week, with some of the sites remaining unreachable for six or more days.

The problems began on Nov. 3, when NaviSite tried to shift processing done on 850 servers at a data center in Baltimore to its headquarters in Andover, Mass. NaviSite acquired the Baltimore facility in August, when it bought Alabanza Corp., another hosting vendor.

Rathin Sinha, NaviSites chief marketing officer, said the company decided to move 200 of the 850 servers to Andover and migrate the data from the rest of the systems to new machines at the Massachusetts data center.

NaviSite told affected customers that their sites would be down for less than a day. But Sinha said that when the company attempted to transfer the data from the 650 servers still in Baltimore, it encountered synchronization failures that kept mushrooming.

Company officials then decided to transfer another 200 servers to Andover to help reduce the scope of the virtual migration and speed up the data transfers.

But NaviSite ran into more problems. As the host servers finally started coming up, URLs didnt work, and customers were able to access their Web sites only by using the underlying IP addresses, Sinha said.

Then, he added, NaviSites network became overloaded because of all the customers trying to get online, further exacerbating the situation.

On Thursday, NaviSite said that 90% of the Web sites were back online. The next morning, the company said it was down to a final set of [trouble] tickets.

But the length of the outages prompted some NaviSite clients to vow in forum postings that they would switch to other hosting vendors once they could retrieve their data.

Others threatened class-action lawsuits, and New York-based law firm Wolf Popper LLP confirmed that it was looking into possible legal action after being contacted by NaviSite clients.

Cynthia Brumfield, president of Washington-based Emerging Media Dynamics Inc., wrote in a blog posting that the data center consolidation was poorly planned and had slipped from mere incompetence to outrageous indifference to the needs of companies like hers.

Copyright © 2007 IDG Communications, Inc.

Shop Tech Products at Amazon