Skype blames buggy Windows software, swamped servers for outage

Computerworld - Skype today blamed last week's outage on a combination of overloaded instant messaging servers, buggy software, and the failure of its "supernode" infrastructure.

In a lengthy blog entry Wednesday, Lars Rabbe, Skype's chief information officer, provided more details on the outage that kept the instant message, Internet telephone and video chat service offline for much of Dec. 22 and parts of Dec. 23.

Previously, Skype had tapped its supernodes, the term for systems running Skype that also act as directories, for the outage.

Today, Rabbe said a bug in an older version of the Windows Skype client was at the root of the service's failure, although the flaw did not trigger the blackout.

The bug in version 5.0.0152 caused those Windows clients to crash when they received a delayed response from "a cluster of support servers responsible for offline instant messaging" that had been overloaded, Rabbe said. About 50% of all Skype users were running the buggy 5.0.0152 version of the Windows client last week.

Rabbe did not explain how or why those servers -- which triggered the Windows client crashes, and thus, the outage -- became unresponsive last Wednesday. Today, Skype said it was still investigating the cause of the server overload.

When the Windows clients began crashing -- at the peak, about four out of every 10 copies of version 5.0.0152 failed -- they also took down as many as 30% of Skype's supernodes, which were also running the problem-plagued edition.

The downfall of those supernodes eventually took all the rest offline as well, as users swamped the remaining supernodes with requests after experiencing a crash. The supernodes were designed to automatically shut down when loads reached certain limits, a measure Rabbe said was designed to preserve performance on their Windows hosts, which are not dedicated to Skype, but simply PCs run by users outside a firewall.

"This further increased the load on remaining supernodes and caused a positive feedback loop, which led to the near complete failures that occurred a few hours after the triggering event," Rabbe reported.

Other factors that contributed to the outage, said Rabbe, was the time of day when the incident began. "The initial crashes happened just before our usual daily peak hour (10 a.m. Pacific) ... which resulted in traffic to the supernodes that was about 100 times what would normally be expected at that time of day," he said.

Skype was offline for approximately 24 hours, from 8 a.m. Pacific on Dec. 22 to 8 a.m. the next day, Rabbe said, a claim that was somewhat at odds with earlier reports from the company, which said on Dec. 23 that two-thirds of its users were still unable to connect as of 3 a.m. that day.

To get its peer-to-peer network back on its feet, Skype added several thousand instances of its software to the network. Those copies were dedicated supernodes, and nicknamed "mega-supernodes" by the company.

By Friday, Dec. 24, Skype was pulling most of the mega-supernodes out of service as the network was restored and the usual supernodes stabilized under their loads.

Rabbe said Skype would revisit its policies on when it automatically pushes updates to clients, and investigate ways to recover faster from a failure. The former seemed to be the way Skype was leaning to prevent another outage in the future.

"We believe these measures will reduce the possibility of this type of failure occurring again," Rabbe said of client software automatic updating.

Last week, Skype CEO Tony Bates apologized for the outage, and promised paying customers that the company would e-mail vouchers for 30 minutes of free calling or extend their subscriptions by one week.

Gregg Keizer covers Microsoft, security issues, Apple, Web browsers and general technology breaking news for Computerworld. Follow Gregg on Twitter at @gkeizer or subscribe to Gregg's RSS feed . His e-mail address is gkeizer@computerworld.com.

Read more about Mobile Apps in Computerworld's Mobile Apps Topic Center.