Skip the navigation

Frontline Defenders

An inside look at how one of Symantec's security operations centers protects clients from cyberattacks.

By Dan Verton
March 29, 2004 12:00 PM ET

Computerworld - It's three o'clock on a Friday afternoon, and Tim Hillyard is monitoring an ongoing cyberintrusion into the network of a major financial institution in the Northeast. Hillyard, a former U.S. Army intelligence analyst, leads a team of a dozen cybersecurity analysts, all of whom are monitoring similar attacks. But neither Hillyard nor his analysts work for the financial firm that's being targeted. Hillyard is a team leader at Symantec Corp.'s Security Operations Center (SOC) in Alexandria, Va. And the financial institution does not even know that anything unusual is happening.
Hillyard stares at a row of computer screens, all of which display a software application known as the Analyst Response Console (ARC). The color-coded user interface provides alerts and data to help analysts focus on the most critical events at any given moment. Working through the interface, the team of analysts then either enters recommendations that appear immediately in a client's Web portal or advises an engineer to block specific traffic on a client network.
"We bring in the source IP address, the source port, the destination IP address, destination port, protocol and the rule option," explains Hillyard as he responds to the attack in real time. "What I'm looking for are a few key things, such as what direction is the traffic going. Just by looking at the ARC dashboard, I can tell that this attack is an inbound attack on Port 135. Then I can actually look at the rule settings for the firewall to see what it did."
Fortunately for the financial institution under attack, the traffic was dropped at the firewall. But what if the traffic had been allowed into the network? According to Hillyard, he could have then drilled into the raw data being captured by the client's intrusion-detection system.
"This is the data that is flying across the wire in real time," he says as he digs through the packet data displayed on the screen. In this case, he finds the code word meow, which indicates the presence of the Blaster or Welchia worm. "If this traffic had been accepted, I could immediately open up an event ticket and assign it to the engineers to block the traffic and alert the client."

Grant Geyer, vice president of managed security services at Symantec Corp.
Grant Geyer, vice president of managed security services at Symantec Corp.
Image Credit: Dan Verton
This process is repeated hundreds of times a day at the Symantec SOC—one of several such facilities used to manage security for customers in 65 countries.
A relatively small cadre of security experts in each center is using cutting-edge technology to bolster security and expand the capabilities of Symantec's customers around the world. Three shifts of approximately a dozen analysts, and an equal number of security engineers, monitor more than 200 million alerts and device logs for 600 companies, many of which are among the biggest names in the Fortune 500. To protect against the threat of social engineering, Symantec declines to reveal the exact number of analysts on each shift.
Mission Control
Technology is what makes it possible for a staff of a few dozen to monitor vulnerabilities and attacks and to alert so many customers simultaneously within Symantec's service-level promise of 15 minutes.
A 42TB online data repository enables a relatively small group of analysts to continuously study attack trends across the entire Internet. Data from Symantec's more than 20,000 points of presence on the Internet floods into the database, where it's crunched and analyzed for virus, worm and other attack trends before major problems can arise.
"We've created a neighborhood-watch program across the Internet so that if you do something against one of my customers, I can flag you globally across my entire client base in an automated manner as a known bad offender," says Grant Geyer, vice president of managed security services at Symantec and manager of the SOC.
This proprietary capability, which has been developed in-house by Symantec's 30 software developers, extends the expertise of client companies by providing deep correlation technologies that enable real-time analysis of the data being monitored by the security devices, Geyer says. "This offers clients a greater degree of security than they could ever do on their own," he says, adding that most companies don't have the resources to gather and analyze that much data. "And that's the value of managed security services: to add intelligence to the real-time aspect of computer security," says Geyer.
Visiting the Symantec SOC is a little like venturing aboard the starship Enterprise. Visitors pass through a secure circular outer chamber and then enter biometric identifiers and personal identification codes just to gain access to the client viewing area, located behind a large window.
The centerpiece of the mission-control area is a large, rotating digital globe that provides real-time data on every country in which Symantec sensors are recording aberrations in Internet traffic that are targeting its clients' networks. The feeds are color-coded from yellow to red, signifying progressively greater levels of standard deviation in the amount of traffic from a particular country.
A number posted to the right of each country name indicates the number of unique IP addresses from which the SOC has recorded attacks during the past 24 hours. The system then maintains a running average for 30 days. This enables Symantec to advise its clients when they should block traffic from a particular part of the world.
Security engineers positioned in the center of the mission-control area act as remote administrators, conducting patch and performance management for clients. In many cases, they have control of the client's security devices, such as firewalls and intrusion-detection systems, and they can make configuration changes in real time when necessary.
In addition to broadcasting news feeds from CNN, rotating screens to the right and left of the control room show different attack patterns that are occurring at various customer sites. Engineers sit in the middle, surrounded by analysts who are alerted every time a spike in a particular type of traffic is detected.
"The technology normalizes, strips down all of the data to its bare essentials and then goes through a process of collating, aggregating and mining it so that an analyst sees simply the results of that data mining," explains Geyer.
The Interface
Symantec has developed and deployed a variety of mechanisms for communicating threat information to customers and for ensuring that their client communications can be authenticated.
For example, automated text-to-voice alerts allow for notification in less than 15 minutes of a client company's entire global customer base and its registered points of contact for emerging threats. In addition, companies can receive e-mails about emerging threats with links to customized Web portals that provide more-detailed information. The Web portals, in turn, receive all of the current posted warnings, the Symantec Response team's analysis of the emerging threats and recommendations for action.
RSA Security Inc. ID tokens authenticate client access to the Web portals, which allow clients to further restrict and customize access based on workgroup. The same secure tokens are used through the phone system, which provides each analyst with a pop-up window on his computer that provides proof of authentication.
The Big Picture
Real-time analysis and early warning are big selling points for managed security services such as Symantec's. Just as nothing an analyst does or types is allowed to disappear into the ether—every keystroke is recorded—so it is with the seemingly harmless events around the Internet that may eventually emerge as the next major worm or virus outbreak. Even minor events in distant parts of the world are monitored and studied for their potential effects on Symantec's customers.
Corilynn Arnold is one of the global security analysts who performs such investigations. She says it's her job to "look at the forest" as opposed to the individual trees that represent customer networks and systems. And that type of analysis is used to alert companies to potential problems, such as virus and worm outbreaks, sometimes weeks in advance of vulnerability announcements by software vendors.
Today, Arnold is keeping tabs on a spyware worm that uses a freeware tool called NetDevil to remotely control systems. "Its intent is to propagate itself like any other worm, but it's only doing it in a very selective manner," explains Arnold. "It's kind of like being a bank robber and targeting only banks where you see a robbery in progress. That's a change from worms in the past. The worms that we're seeing now have auto-update features and are designed to create massive command-and-control networks."
And what about yesterday's worms, such as Code Red and Blaster? They're still alive and well and propagating, says Arnold. " It's like a human virus," she says. "Nothing ever really dies."
BLASTER, SLAMMER AND CODE RED


Our Commenting Policies