Web Data: Cached or Current?

Computerworld - For all of its virtual connotations, the Internet depends entirely upon its physical infrastructure to move information around. And the physical distance from server to end user leaves plenty of time for information, in the form of packets, to get lost, resulting in e-mails that never arrive, Web pages that load incompletely and streaming audio or video that pops, flickers or just dies. So getting files closer to end users can improve performance.

One way to do that is by caching files near the edge of the network, closer to users. Barry Weber, vice president of technical infrastructure at BarnesandNoble.com Inc. in New York, says the company's BN.com site saw a 50% improvement in performance from the end users' perspective after it started using caching in February last year.

Within the past few years, more companies have embraced caching as a way to push static content out to users, frequently outsourcing the content to external content delivery networks (CDN). CDNs are groups of Web servers and caching servers, which are simpler and less expensive than Web servers but also aren't able to generate dynamic content.

Companies are increasingly turning to CDNs because they can deliver static content more reliably than the prevailing model of a few clusters of Web servers serving every request. BN.com outsources delivery of its static content to Akamai Technologies Inc. in Cambridge, Mass. After BN.com uploads new content to one of Akamai's servers, it takes two to three hours for it to become available across Akamai's CDN. The CDN intercepts all IP requests for BN.com's static content - HTML, images, streaming audio or video - and serves it to users from the available cache that's physically closest to the user.

Meanwhile, requests for dynamic content, such as book inventory levels and targeted banner advertisements, go to BN.com's servers as usual. Both find their way back to the end user, who sees only the finished Web page. Though CDNs are unnecessary on a small scale, the CDN helps keep the site running quickly when, say, a new Stephen King novel comes out and thousands of users are viewing the book's Web page on BN.com every hour.

Now, for the first time, caching is enabling companies to do things that were previously impossible or very unreliable on the Internet, such as streaming catalogs of media files. But caching still leaves something to be desired for retail companies, such as Barnesandnoble.com, that dynamically generate their Web pages with content specifically targeted at individuals.

Some companies have a financial imperative to make their video files reliably available on the Internet. And reliability has been elusive, especially as the number of simultaneous streams has increased.