In December, Computerworld interviewed three companies about how they scaled their sites to -- hopefully -- survive the onslaught of online holiday shoppers ("Surviving E-Christmas," Technology, Dec. 20). Well, the news is in: They and their Web sites survived, with minimal or no damage.
Luck was only part of it. Each company worked furiously before the holiday season began in October to upgrade its infrastructures and ensure that it could deliver products to customers. The companies learned four lessons: Choose tools that will scale with the site, argue for spending on infrastructure upgrades rather than advertising, always monitor network suppliers and handle key processes off-line.
There's already more work to be done before next Christmas. Consumers spent $7 billion online this past season, according to Jupiter Communications Inc. in New York. And though 90% were happy with the experience, consumers still said the process could go better. Their complaints were the same as in 1998: inventory shortfalls, high shipping and handling costs and slow site performance.
At the same time, sites must insulate themselves against denial-of-service attacks, such as those that hit Yahoo Inc., Amazon.com Inc., CNN.com, Buy.com Inc. and eBay Inc. earlier this month. Here's what these pioneers learned this holiday season, and how they're using it to meet the challenges of next year.
Lesson 1: Choose Tools That Scale
Conventional wisdom says you shouldn't make major changes to your e-commerce Web site during the holiday shopping season because changes increase the likelihood that something could go wrong. But for KBkids.com Inc. in Denver, which was established only last July and subsequently saw huge traffic increases, upgrades were a necessity.
So amend that wisdom: You can make major changes if you choose your tools carefully.
As a first priority, your site must be able to handle sudden traffic surges. KBkids.com reviewed all its systems for scalability and found that its databases were vulnerable. "A key technology that was instrumental in our scalability was database connection-pooling software that allows us to handle many user connections for every database connection we have," says Shawn Davison, vice president of technology at KBkids.com. Most database implementations -- such as the Apache servers KBkids.com uses -- require a dedicated connection between each server and database. With high levels of traffic, these one-to-one connections can become bottlenecks. So the toy vendor developed its own Java Database Connectivity pooling software, though such software is also available off-the-shelf. By pooling connections, the site can handle more traffic without slowing.
Site-monitoring tools must be carefully chosen as well because they can buckle under too much traffic and leave a company unable to gauge its site performance. KBkids.com tried various analysis packages, but none could process the site logs in less than 24 hours, and none could effectively monitor the site in real time. Had KBkids.com simply integrated these tools without testing them first, the results could have been disastrous.
One product, WebTrends from Portland, Ore.-based WebTrends Corp., "had to load all of the log files into memory, and it would get to a point where it would crash," says Davison. Though he says new WebTrends software remedies such problems, the company has since built a hybrid system with homespun analysis tools and an Oracle Corp. data warehouse.
By and large, KBkids.com's seat-of-the-pants infrastructure upgrades were successful. A server's inability to process two simultaneous coupon offers crashed the site for a few hours. But overall, PC Data Processing Inc., a Web traffic measurement firm in Port Washington, N.Y., ranked it the No. 3 online toy retailer in the past Christmas season, with more than 2.6 million unique visitors in December.
Lesson 2: Argue for Infrastructure, Not Ads
With investors demanding that online retailers produce profit growth, the nuts and bolts of sale processing and order fulfillment become as important, if not more so, than expensive, image-building advertising campaigns.
The drive for profitability helped Randy Jo Wilcox, president and chief operating officer of BuyItNow.com LLC in Tulsa, Okla., make the following choice: After October sales numbers looked especially good, she slashed the $12 million holiday advertising budget to $2 million. Then she invested in tools to make the site run faster. They included Texis, a search engine from Thunderstone Software-EPI Inc. in Lakewood, Ohio.
Unlike BuyItNow.com's previous search technology, Texis could return results for words that users misspelled. Also, users could drill deeper into specific categories by further refining searches, which is useful given the site's 13 online stores and breadth of products. Users are now finding what they want more quickly.
That meant less load on BuyItNow.com's servers, according to net.Analysis site-monitoring tools from Cambridge, Mass.-based net.Genesis Corp. "According to net.Analysis reports, 33% of people were on our site for 10 to 30 minutes, and that's a lot," says Wilcox. In addition, 12% of people were on the site for more than 30 minutes. E-commerce sites are like restaurants: Business is good, but turnover is even better.
Despite slashing advertising, BuyItNow.com met its 1999 holiday earnings projections.
Lesson 3: Monitor Your Network Suppliers
In an ideal world, phone companies, Internet service providers and network backbone providers would contact online retailers whenever their service went down. What really happens is that it's up to the retailers, with their livelihood at stake, to raise the red flag. BuyItNow.com vigilantly watches its networks with the help of a tool called VisualRoute from Datametrics Systems Corp. in Florham Park, N.J. VisualRoute lets BuyItNow.com monitor its servers, some of which are hosted by Exodus Communications Inc. in Santa Clara, Calif., to track which servers are serving which geographical areas, and to keep an eye on its network host providers. The Christmas season demanded special vigilance: Although most network providers had occasional outages, all three companies interviewed for this article saw definite congestion on the Internet as a whole this past season.
So when Los Angeles visitors to the BuyItNow.com site were getting served by a server in Washington this past Christmas -- even though BuyItNow.com has a server bank in California hosted by Exodus -- the company knew there was trouble. It turned out to be a problem with network provider Cable & Wireless USA in Vienna, Va. "We saw a lot of trouble with Cable & Wireless," says Wilcox. Furthermore, Cable & Wireless was unaware that there was a problem with the connection it provided to BuyItNow.com. "The (Cable & Wireless) primary pipe went down, and their secondary has the capability of an ISDN. The pipe they had us on was trying to take the water coming out of a firehouse and force it down a straw," says Wilcox. Integrated Services Digital Network (ISDN) lines can transfer about 128K bit/sec. After a few more hours, Cable & Wireless increased the primary pipe to a burstable OC3, the largest available network connection, which can handle 155M bit/sec. A Cable & Wireless spokesman said he couldn't discuss this specific case but added that the company's overall performance during the 1999 shopping season was excellent, as evidenced by Internet service ratings from Austin, Texas-based Matrix Information and Directory Services Inc.
BuyItNow.com also suffered a six-hour outage due to failed routers at an Exodus site in San Jose. "We lost a router in San Jose, and it was like a spiraling effect -- traffic was rerouted, it took our MCI network in Dallas (and) Chicago out," says Wilcox.
Then there was the incident with BuyItNow.com's service provider, Oklahoma City-based ioNet Inc., a division of Herndon, Va.-based PSINet Inc., as documented in "Surviving E-Christmas." IoNet, which connected BuyItNow.com's 200-person home office in Tulsa to the Internet, experienced several outages during the busy shopping season, which made updating the site and processing order fulfillment difficult at times. E-commerce functions are handled in different sites.
A spokesman for PSINet couldn't comment on those incidents.
After BuyItNow.com switched to AT&T Corp., with MCI WorldCom Inc. as a backup, reliability has gone up and average Web page load times for BuyItNow.com's employees are down from more than a minute to just three seconds.
By contrast, Rich Secor, vice president and CIO at SmarterKids.com Inc., an online educational toy retailer in Needham, Mass., says his site had absolutely no interruption of service. But only some of that was due to careful planning. "Certainly there's an element of luck in that I can't control the Internet all the way to the customer's browser. However, we did do everything we could to build redundancies into the system," he says. For example, it measured the network from many locations and checked application performance every 15 minutes. The redundancies paid off: The site handled 2.1 million unique users in December. PC Data ranked it the top educational site on the Web and the No. 4 online toy retailer this past Christmas.
Lesson 4: Handle Key Processes Off-line
During the 1998 holiday shopping season, SmarterKids.com ran into problems when CyberCash Inc., the Reston, Va.-based third-party authentication bureau it uses to check consumers' credit cards, went off-line. When things were functioning normally, consumers would enter their orders and personal information, then wait while the SmarterKids.com site verified the credit card. But with that third-party service unavailable, the verification screen timed out, and users saw a screen with only an error message. The reason: SmarterKids.com's Web site didn't know how to recover from that type of error. While SmarterKids.com claims that it was able to directly contact customers whose checkout process had suddenly aborted, subsequent outages underscored the need to avoid such incidents.
For the 1999 holiday season, SmarterKids.com succeeded in making the credit-card authorization process asynchronous, meaning it functions independently of the rest of the transaction process. Instead of making customers wait for their credit cards to be verified in real time, they would get an e-mail once the verification had taken place, confirming the order. "If the credit authorization has to wait two hours from the time you placed your order, it's not a big deal -- it's not going to affect when the order ships," says Secor.
It worked. "There were brief periods of time when our credit authorization bureau wasn't available, but customers never knew," reports Secor. By simply decoupling the authorization process, SmarterKids.com gave itself room to breathe.