When networking hardware failed at PayPal on Friday morning and its recovery systems lagged in responding, things got hectic at Shop By System, an online seller of computer accessories and components based in South Dakota.
With PayPal's online payment service down for hours globally on Friday, Shop By System staff had to scramble to deal with a sudden spike in customer service calls from buyers inquiring about orders that had timed out or been outright rejected.
While the employees tried to complete as many orders as possible via phone, Shop By System estimates that PayPal's outage resulted in a loss of between 25 percent and 30 percent of the day's potential sales.
Shop By System, a PayPal merchant for more than two years, is now considering adding other e-payment options to its online store.
"I would give them good marks overall, however when they go down it would seem they really go down, and it's typically for several hours," said Tony Maibaum, Shop By System manager, via e-mail.
In his company's experience, calling the PayPal merchant support line is of little to no help. He thinks PayPal could do a much better job reaching out to merchants when it has technical problems.
"We've never received an e-mail stating their service is down nor have we ever received an e-mail apologizing that their service was down," he said.
On Friday, he found PayPal to be slow and vague in its blog updates about the outage, which made it more difficult for Shop By System to manage the situation with its customers.
"We had previously featured Google Checkout as an option for our customers and Checkout by Amazon has been getting a lot of support lately, too. Our current thinking is to explore adding back in at least an additional checkout option for our customers as both a courtesy and a backup should another outage occur," he said.
On the PayPal blog and discussion forums, other merchants expressed similar concerns during the outage, complaining about lost sales, upset customers, unhelpful PayPal help desk staffers and spotty updates.
"Are you going to provide any more details, or is it just a case of telling all your merchants that 'something broke,' which we already knew," wrote one merchant identified as Edward.
"Triple redundancy is the only solution to this unacceptable outage," wrote another named Tom.
With the holiday shopping season approaching, online retailers can take steps to mitigate problems with third-party providers of online payments and other critical functionality for their e-stores.
It's a good idea for e-tailers to have alternative payment options in case the primary one malfunctions, said Ben Rushlo, director of performance management at Keynote Systems, a mobile and website testing and monitoring company in San Mateo.
Another tip is to implement a queueing mechanism in their e-commerce systems that would hold incoming orders and complete their processing once the affected payment system comes back online, he said.
"Outages in payment systems definitely have the potential to have a very large impact," Rushlo said.
Online retailers should routinely do load testing, measuring and monitoring of all the third-party Web services that are essential to their websites, he said.
PayPal blamed the problem on a network hardware failure and on a subsequent malfunction of the recovery and failover systems, which didn't act with the speed that they were supposed to. After that initial outage of about 90 minutes, PayPal suffered another one hours later that lasted about one hour.
That scenario sounds to Stuart Madnick, a professor at the Massachusetts Institute of Technology, like it could have been caused by a chain of unlikely mishaps.
If that's the case, the lesson for PayPal and for anyone in charge of an IT system is that the biggest catastrophes often are caused by a lack of foresight and preparation for problems whose occurrence is rare but nonetheless possible.
"Generally, we don't do a good job of protecting ourselves from events that occur fairly rarely," Madnick said.