Behavioral rules vs. signatures: Which should you use?

Many security products on the market today detect malicious attacks, but few take action to prevent them. Even more confusing for IT and security professionals is the debate over the best way to detect and prevent hacking activity: signatures or behavioral rules. Each approach has advantages and disadvantages, but by combining the two, enterprises can ensure that servers and data are fully protected.

Best practices dictate that companies adopt a defense-in-depth method of protection, which includes using the best of both technologies. One technology can't take the place of the other: Behavioral rules allow the servers to be protected from new and previously unknown attacks. However, the coverage of behavioral systems is limited, many attacks aren't covered, and the system generates many more false positives. For full forensics capability, the signature is critical in identifying attacks, so security managers can know what sort of attack is being directed at their systems.

Here are the pros and cons of both technologies and an argument for why combining the two provides the best protection.

Behavioral-based methodologies

There is a lot of hype around behavior-only technologies. In a behavioral model, the focus is on user or application behavior and not on a specific attack pattern. The goal is to distinguish between malicious and nonmalicious behaviors. The promise of such systems is great: Theoretically, this type of solution can deal with all attacks, both known and unknown. Moreover, it promises to free the user from having to keep the system updated, since there is no use of attack signatures.

A behavioral solution either defines the "good" behavior and flags any other behavior as malicious or, alternatively, defines what the "bad" behaviors are and regards all the rest as nonmalicious. In both cases, the challenge of distinguishing between good and bad behaviors is significant and more difficult than in the case of attack-signature systems. As a result, the system is less accurate, leading to an increased rate of false positives.

Yet another obstacle faced by pure behavioral systems is limited security coverage. There are many cases in which it is extremely difficult to detect malicious behavior. As an example, let's consider directory traversal attacks against Web servers.

The most straightforward directory traversal attack is to send a URL with combinations of the characters ../, which could allow the attacker to access files outside the virtual tree or files within the virtual tree for which access is otherwise prevented. Vendors have prevented these simple attacks by providing patches, but malicious hackers using coding tricks (such as Unicode) have been able to bypass the security checks and achieve their goal of compromising systems.

A purely behavioral approach can deal with this problem to some extent. However, if the attacker's goal is to access files within the Web root to which access is forbidden, the behavioral approach is incapable of identifying this attempt as malicious.

Another drawback of a purely behavioral system is that even when the system correctly identifies a malicious attack, it can't provide information about the vulnerability that was exploited, even when the vulnerability is well known. This limits the system operator's ability to gather information about the attack technique, the vulnerability, the attacker identity and recommendations on steps that need to be taken (such as patching).

Signature-based methodologies

A signature is actually a fingerprint of a given attack. The signature captures the actions, which are unique to a given attack. This pragmatic approach is focused on specific attacks and is very accurate at lowering the rate of false positives.

Still, signature-based systems have a major drawback: They can deal only with known attacks. In the past few years, purely signature-based intrusion-detection systems did not perform well. Recent Internet worms, such as Code Red and Nimda, demonstrated the need for systems that can detect and prevent unknown attacks. These worms caused huge damage to many computing farms, even when signature-based systems were present. Alternative approaches to protecting servers, such as patch deployment, were also found to be ineffective and very costly.

The argument for the hybrid approach

The most effective approach for intrusions is a hybrid one that combines the best of both protection methodologies. These hybrids avoid the fundamental trade-off by providing coverage to both known and unknown attacks and at the same time keep the false-positive rate to a minimum.

At the end of the day, managers need to decide what is most important in protecting the servers, data and files in their environments. A hybrid approach implies protection at all levels to ensure that critical information is not compromised. By doing some homework and exploring options, you'll find that combining the best and broadest layers of security for a defense-in-depth strategy forms the best possible protection.

Yona Hollander is vice president of security management at Entercept Security Technologies, an intrusion-prevention software company in San Jose. He is part of Entercept's Ricochet Team, a specialized group of security researchers dedicated to identifying, assessing and evaluating intelligence related to server threats.

Related:

Copyright © 2003 IDG Communications, Inc.

  
Shop Tech Products at Amazon