Skype blames buggy Windows software, swamped servers for outage
Promises to look into auto-updating its Windows client software to prevent another blackout
Computerworld - Skype today blamed last week's outage on a combination of overloaded instant messaging servers, buggy software, and the failure of its "supernode" infrastructure.
In a lengthy blog entry Wednesday, Lars Rabbe, Skype's chief information officer, provided more details on the outage that kept the instant message, Internet telephone and video chat service offline for much of Dec. 22 and parts of Dec. 23.
Previously, Skype had tapped its supernodes, the term for systems running Skype that also act as directories, for the outage.
Today, Rabbe said a bug in an older version of the Windows Skype client was at the root of the service's failure, although the flaw did not trigger the blackout.
The bug in version 5.0.0152 caused those Windows clients to crash when they received a delayed response from "a cluster of support servers responsible for offline instant messaging" that had been overloaded, Rabbe said. About 50% of all Skype users were running the buggy 5.0.0152 version of the Windows client last week.
Rabbe did not explain how or why those servers -- which triggered the Windows client crashes, and thus, the outage -- became unresponsive last Wednesday. Today, Skype said it was still investigating the cause of the server overload.
When the Windows clients began crashing -- at the peak, about four out of every 10 copies of version 5.0.0152 failed -- they also took down as many as 30% of Skype's supernodes, which were also running the problem-plagued edition.
The downfall of those supernodes eventually took all the rest offline as well, as users swamped the remaining supernodes with requests after experiencing a crash. The supernodes were designed to automatically shut down when loads reached certain limits, a measure Rabbe said was designed to preserve performance on their Windows hosts, which are not dedicated to Skype, but simply PCs run by users outside a firewall.
"This further increased the load on remaining supernodes and caused a positive feedback loop, which led to the near complete failures that occurred a few hours after the triggering event," Rabbe reported.
Other factors that contributed to the outage, said Rabbe, was the time of day when the incident began. "The initial crashes happened just before our usual daily peak hour (10 a.m. Pacific) ... which resulted in traffic to the supernodes that was about 100 times what would normally be expected at that time of day," he said.
Skype was offline for approximately 24 hours, from 8 a.m. Pacific on Dec. 22 to 8 a.m. the next day, Rabbe said, a claim that was somewhat at odds with earlier reports from the company, which said on Dec. 23 that two-thirds of its users were still unable to connect as of 3 a.m. that day.
To get its peer-to-peer network back on its feet, Skype added several thousand instances of its software to the network. Those copies were dedicated supernodes, and nicknamed "mega-supernodes" by the company.
By Friday, Dec. 24, Skype was pulling most of the mega-supernodes out of service as the network was restored and the usual supernodes stabilized under their loads.
Rabbe said Skype would revisit its policies on when it automatically pushes updates to clients, and investigate ways to recover faster from a failure. The former seemed to be the way Skype was leaning to prevent another outage in the future.
"We believe these measures will reduce the possibility of this type of failure occurring again," Rabbe said of client software automatic updating.
Last week, Skype CEO Tony Bates apologized for the outage, and promised paying customers that the company would e-mail vouchers for 30 minutes of free calling or extend their subscriptions by one week.
Gregg Keizer covers Microsoft, security issues, Apple, Web browsers and general technology breaking news for Computerworld. Follow Gregg on Twitter at @gkeizer or subscribe to Gregg's RSS feed . His e-mail address is firstname.lastname@example.org.
Read more about Mobile Apps in Computerworld's Mobile Apps Topic Center.
- Mission Critical: Managing Mobile Applications & Content Smartphones, tablets and other mobile devices have become embedded in enterprise processes, thanks to the consumerization of IT and a new generation of...
- Use the Mobile App Mix to Choose an Enterprise App Store Strategy In this research report Gartner outlines how organizations can optimally secure, distribute, and manage mobile applications for employees and contracted workers.
- The Case for Mobile Apps Today's mobile apps turn handheld devices into e-book readers, portable navigation systems, digital wallets and more. And for organizations with mobile workers, they...
- SANS: Next-Generation Datacenters = Next-Generation Security This whitepaper takes a look at some new technology that may allow security teams to implement more flexible and capable protection models in...
- Is SQL Server AlwaysOn really as powerful? Tips and Tricks from the field With the introduction of AlwaysOn, Windows Clustering Services is now more critical than ever.
- What Does it Take to Deliver a Superior Customer Experience? The Two Top-Rated Online Retailers, B&H Photo and Crutchfield Electronics, Share Their Secrets Discuss practical CX tools and service methods such as contact center agents and the use of realtime speech analytics to help contact center... All Mobile Apps White Papers | Webcasts