With recent outages, big data service providers take hits

Computerworld - When eBay Inc.'s PayPal unit suffered an outage Aug. 3, one of the companies affected was Sailrite Enterprises Inc., a sailing supply company in Churubusco, Ind. Sailrite lost its customer payment services for six hours.

The next day, PayPal's services failed again -- this time, for what seemed like an hour, according to Matt Grant, Sailrite's vice president. PayPal, in an e-mail, blamed the outage on a "back-end router" complicated by a failure in its redundancy measures. Grant was not amused. He posted a blunt message on PayPal's blog: "This is not acceptable."

It's not just PayPal. Other big providers of Internet-based services have seen outages, especially over the past two months. What's uncertain is whether this is a sign of a systemic problem, a run of bad luck or business as usual. Could it be that the reliability of core services is suffering along with the economy? Or is the increasing dependency on external cloud providers, coupled with near-instantaneous Twitter reports of outages, just making what's normal seem worse?

The answers aren't clear, but there are numerous signs of trouble.

Last month, Texas-based Rackspace Hosting Inc. said it would issue up to $3.5 million in service credits to customers after a data center in Dallas was hit by two separate power outages in June and July. Backup generators failed after one outage, and connectivity was lost in the other. Also in July, Google Inc.'s App Engine suffered an outage, with systems exhibiting "elevated latency and error rates." That incident appeared to last about four hours, according to Google's staff updates.

That wasn't Google's only outage this year. In February a "routine maintenance event" in a European data center knocked out Gmail for more than two hours.

The same week PayPal had trouble, a data center operated by a Houston-based hosted services provider known as The Planet was knocked offline for an hour. The problem, explained in a Twitter post by the company, was this: "There was a utility power drop, and the automatic cut-over to our UPS systems did not occur."

Other outages at smaller data centers got public attention, too: a Texas government system had problems; so did a data center in Seattle and hosted services provider Site5.com. More recently, Twitter and other sites were hit by distributed denial-of-service (DDoS) attacks.

Finally, Cisco Systems Inc.'s Web site was down for two hours earlier this month because of human error.

Kurt Roemer, chief security strategist at Citrix Systems Inc., said he sees more evidence of internal mistakes and wonders, for instance, whether Cisco's Web site outage "would that had happened a few years ago... when they had multiple people checking every single change?"