A year after distributed denial-of-service attacks blasted the likes of Yahoo Inc., eBay Inc. and ETrade Group Inc. off-line, no one has found an easy way to defend against a flood of unwanted Internet Protocol packets.
In fact, everyone's still pretty much in the dark - literally, in one case - when it comes to finding a silver bullet.
A recent meeting of the DDoS Working Group, a forum organized last year to plot network defenses, was conducted solely by the light of laptops after KPMG International's Silicon Valley office was visited by one of California's rolling blackouts. In the ghostly glow could be discerned John Zent, manager of risk management at Yahoo, and Allen Yousefi, information security officer at eBay, along with representatives from security vendors eager to woo these top e-commerce firms.
The talk was no brighter than the lighting. According to several attendees, Yahoo and eBay are more than just dismayed by the slow pace of finding technical defenses to denial-of-service attacks and the even more nefarious distributed denial-of-service attacks, which let an individual launch IP attack streams from hundreds, or even thousands, of compromised computers.
Web site operators are frustrated by the apparent inability of Internet service providers and Web-hosting providers to quickly filter out denial-of-service attack traffic when it pours into their routers and servers. Whether a low-grade nuisance or the kind of multibarreled assault that upended Microsoft Corp.'s sites for three days recently, this "bad" traffic is eating up bandwidth and at times blocking legitimate traffic to the most prominent e-commerce sites.
"People are getting a little radical about it," said one attendee. For companies such as Yahoo and eBay, "it's a service-level agreement issue with the [Internet service providers] and co-location providers," he said. The attendee predicted that this year will see lawyers battling over whether distributed denial-of-service traffic should have to be filtered out to satisfy such agreements.
Efforts Under Way
Despite the gloom, there are many efforts under way to cope with all manner of denial-of-service threats - and rays of hope are visible.
Service and software providers have united to share information and forge common defenses. Promising security start-ups focusing on the problem are attracting big-name backers. Law enforcement groups - working with the network industry and its customers - are nailing the bad guys.
The DDoS Working Group is doing what it can to spur cooperation among Internet service providers. The group plans to publish recommendations for automated distributed denial-of-service defenses by the end of March.
"There are political issues and technical issues," said Tom Clare, a product manager at Check Point Software Technologies Ltd. and a DDoS Working Group member.
The document is expected to define a common intrusion detection method for collecting and measuring the percentage of bandwidth being consumed and a flow tag to identify traffic and other Layer 2 data collected from the packets. A firewall or other network device that implemented the DDoS Working Group specification would be able to report the start of an attack to the Internet service provider, and other Internet service providers, using compatible equipment, would be able to share the information.
But it's uncertain whether Internet service providers can interact smoothly even if equipment makers support a common security specification, which may leave this as yet another security proposal that never got off the drawing board.
Internet Service Providers' Role
This much is clear: Internet service providers play the critical role in the distributed denial-of-service endgame against attackers, who are heavily armed with denial-of-service "malware," software posted at hacker sites for free use. And most of the intrusion detection analysis and filtering that Internet service providers do is manual and difficult.
"We can't be held responsible for attacks on our customers," said Amir Moujtahed, director of systems engineering and corporate security at Epoch Internet, a Costa Mesa, Calif., Internet service provider. "But if customers give us the IP addresses [of the source], we will block them." Epoch has intrusion-detection equipment from NFR Security on its external and internal networks, and Epoch engineers watch the logs closely for evidence of attack signatures. But it's a labor-intensive process.
Moujtahed said Internet service providers are trying to do their part by installing antispoofing filters and cooperating with competitors through informal agreements hashed out in the ISP Service Consortium, which meets monthly.
"This is all part of the lesson learned after what happened last year," Moujtahed said. "[Internet service providers] like [Genuity], UUNet and AOL compete, but we are working together on this."
It's small comfort to the high-tech industry that the 16-year-old perpetrator of last February's incidents, a Canadian hacker nicknamed Mafiaboy, last month pleaded guilty to single-handedly attacking Amazon.com, eBay, Yahoo, Charles Schwab Corp., CNN and eTrade, among others (see story).
Mafiaboy carried out his distributed denial-of-service spree using attack tools available on the Internet that let him launch a remotely coordinated blitz of IP packets from servers compromised by agent attack "zombies." Mafiaboy awaits sentencing, but it's expected he won't get much more than two years in a juvenile detention center.
Those attacks forced most of the victimized e-commerce sites off-line for about three hours. In the heat of battle to block the blitz of IP packets, Internet service providers did what they could through filtering bad traffic and claimed victory when it ended. But security experts familiar with what occurred agree that this filtering accomplished little and that relief came because Mafiaboy simply stopped his attacks after three-hour intervals.
"The attacks happened Monday through Wednesday, and those guys were still working Friday and Saturday to figure out what happened," says Frank Huerta, CEO of Recourse Technologies, which makes security gear to detect and trace denial-of-service attacks.
Like many experts, Huerta said the work Internet service providers did manually filtering bad traffic didn't stop Mafiaboy's attacks. And though law enforcement officials did extensive work bringing him to justice, one reason they succeeded was that he bragged about his exploits in an Internet chat room.
Microsoft two weeks ago became the latest high-profile victim of a distributed denial-of-service attack, though no one seems to be bragging about causing it. The software giant lost MSN.com, Carpoint.com, Expedia.com and other Web properties for a day, hours or minutes over the course of a week (see story).
Microsoft declined to explain its response to the attacks, other than to say it was working with the FBI. However, CIO Rick Devenuti acknowledged that Microsoft "accepts full responsibility" for the inconvenience to its Web users (see story). He said the company hadn't applied "sufficient self-defense" by using third-party products at the front end of its core network.
Stopgap Measures
There are stopgap measures that Web sites can take to shore up defenses, such as using as many load-balancing and high-speed pipes as they can, as well as intrusion-detection systems that can indicate suspicious activity is suddenly on the radar screen.
And that is better than nothing. Fidelity Investments and Bear, Stearns & Co. reportedly deployed Top Layer Networks' AppSwitch with its intrusion-detection features after last February's attacks on e-commerce sites.
Overall, there's a more sober-minded assessment of the problem among vendors than a year ago.
Cisco Systems Inc. last February claimed that making use of ingress filtering in routers, a technology described in IETF draft RFC 2267plus, would stop denial-of-service attacks. But the router manufacturer has abandoned that stance.
"There is no silver bullet for a [denial-of-service] attack," said Lance Hayden, a manager with Cisco's consulting services team in Austin, Texas. But Cisco and a number of venture capital firms are investing in start-ups that are promising to develop comprehensive defense systems for distributed denial-of-service attacks. Another start-up, Arbor Networks, is also striving to find a cure.
So, too, are established security vendors, including Internet Security Systems (ISS). Allen Wilson, director of emerging technologies at ISS and a DDoS Working Group member, said that tracing this type of attack remains "very manually intensive and time-consuming. For [Internet service providers], it's one hop at a time, and you need to get a hold of people and let them know that your network is attacking theirs."
ISS claims to be developing technologies that depend on what it calls "the moving-target defense." The idea is that if an attack is launched at a Web site, the victim and Internet service provider work together to identify the source and then create a "black hole for the IP address," Wilson said. "You drop the packets but don't kill the connection, which helps trace back the attackers."
At the same time, you create a temporary IP address for your site that gets broadcast out to enable legitimate traffic to still find you.
Quantifying the denial-of-service problem isn't easy. Whenever a Web outage occurs, security experts always suspect denial of service, even if the business blames internal screw-ups. Online auction vendor eBay has suffered several Web outages in recent months that many security experts suspect were denial-of-service attacks, something eBay vehemently denies. However, it was clearly a denial-of-service attack that disabled much of the Undernet, part of the Internet Relay Chat network, early last month (see story).
After last February's attacks, the Clinton administration asked the IT industry what it could do to help combat what everyone suddenly realized was a dangerous situation on the Internet.
It took 11 months to come forward with a plan, but 19 high-tech corporations recently formed an organization called IT Information Sharing and Analysis Center (IT-ISAC), which will run a so-called virtual center to share information about denial-of-service attacks and software vulnerabilities in general. Founding members are paying almost $1 million for the privilege, although general membership fees, which won't include access to all the information, drop as low as $5,000.
The organization's database of shared information, which will be managed by ISS, is intended to help solve security problems, so vendors accessing this sensitive information have agreed not to use it as a marketing weapon.
Those who expected Internet service providers to roll out new technologies or services to help stop these attacks in the past 12 months have surely been disappointed. Internet service providers are essentially using the same spot-filtering and monitoring techniques today as a year ago. Nevertheless, Internet service providers claim that heightened awareness and vigorous monitoring have helped reduce damage.
"We regularly see attacks, but nothing at the level of last year's on multiple, highly visible customers," said Kelly Cooper, security engineer at Genuity Inc. "If we were to offer filtering and monitoring services to our customers for an extra charge, that would sort of be like blackmailing them."
Genuity expects new capabilities from router and switch vendors that will integrate IP address filtering directly into the operating system of the device. One of the most common reasons Internet service providers aren't setting up IP address filtering is that it can slow the network. However, if filtering is integrated into network devices, performance shouldn't be hurt, Cooper said.
Vint Cerf, senior vice president of Internet architecture and technology at WorldCom Inc., said standard load-balancing and content-distribution techniques that many Web-hosting service providers use reduce the negative impact of these attacks.
"Load sharing across multiple servers helps reduce the impact of classic [distributed denial-of-service] attacks because there are multiple versions of a Web site operating across the Internet," Cerf said. In addition to distributing legitimate traffic, load balancing and caching distribute rogue distributed denial-of-service packets so one server isn't crumbling under the weight of an attack.
Internet service providers also see hope in specifications being developed by the Internet Engineering Task Force. I-Trace is one preliminary technology that will allow Internet service providers to quickly find where a distributed denial-of-service attack originates. Once an Internet service provider recognizes the source of an attack, it can immediately set up a filter.
But this technology is very much in the early stages of development. All in all, it certainly seems like the industry will experience at least another year of being in the dark on distributed denial of service.
Network World senior editor Tim Greene contributed to this story.
Related story:
- University Computers Remain Hacker Havens, Feb. 12, 2001
For more security coverage, head to Computerworld's Security Watch page.
This story, "No easy defense vs. denial-of-service attacks" was originally published by Network World.