Behavioral network security: Is it right for your company?

Current Job Listings

Whether it was Mydoom, Netsky or a devious insider who wreaked havoc on your network, one thing is certain -- your network remains susceptible to a wide variety of new threats, and your most important network applications are at risk. Despite big spending, a great security team and your own best efforts, the attackers are still winning.

Sound familiar? If so, it's time to look at a new approach called behavioral security. The following will help you decide whether behavioral security makes sense for your organization.

To secure it, first understand it

The underlying premise of behavioral security is that all operationally relevant threats and attacks can be discerned through visibility into network traffic behavior. Infiltrated networks exhibit host-connection patterns that deviate from historical, or normal, connection patterns. These deviations, or anomalies, support clear analysis of the new threat and help suggest the most effective mitigation response.

Because most bad network behaviors manifest themselves as anomalies, it's difficult for any attack to fly under the behavioral radar screen. A "zero-day" worm is a good example. Because the worm is new, conventional signature-based intrusion-detection systems (IDS) don't recognize it. The worm slips past the perimeter defenses and enters your network. When hosts begin to exhibit anomalous connection behavior, behavioral security systems kick in, focusing on fast and accurate detection of those anomalies.

Behavioral systems don't depend on noticing specific bit patterns in traffic entering the perimeter of your network (where most conventional IDSs are installed). Instead, they provide connection behavior in the core of the network and can thus detect threats that enter the enterprise network, for example, from an employee's laptop that was infected while at home.

Behavior-based security products deliver useful data, such as lists of infected hosts, dependency maps for those hosts, propagation paths and the ports and protocols used in an attack. Having that data available in minutes rather than hours or days can dramatically improve your ability to react, thwart and recover from attacks.

Under the hood

The input for behavioral security systems is network traffic data. Gathering this data requires your network to be equipped with data-collection devices, such as network probes or NetFlow-enabled routers. To extract the most actionable intelligence from network traffic, the data should be aggregated into a centralized model for networkwide analysis.

The aggregated traffic data forms a baseline of normal network activity. The baseline includes information such as which hosts connect to one another and which services are used over which ports. This historical analysis is broken down by time periods, such as days of the week, over which similar connection behavior would be expected.

Recent increases in computing power enable behavior profiling in real time, even in large networks, such as those of global financial services companies with more than 100,000 hosts. As corporate networks change, with new hosts or applications going online and users being added, it's important for the connection profile to be dynamic. Continuous updates enable the profile to reflect network changes and provide more accurate event detection.

With the connection profile in place, behavioral systems begin comparing it against current activity. When deviations occur that are indicative of worms, scans, unauthorized access or user-specified policy violations, these systems characterize the threat, generate an alert and make actionable mitigation recommendations. With vendor support and an experienced security team member managing the process, a behavioral system can be installed, tuned and fully functional within a few weeks.

What to look for in a good product

In addition to dynamic profiling, key attributes of a good system include scalability, simplicity and high-quality alerting. Scalability is critical because these systems work best with a single, networkwide view of activity. Separate views of different areas or single links can create coverage overlap problems. Simplicity is important to limit the burden on IT and security staffs. Implementation should be easy, and ongoing tuning and maintenance must be minimal. High-quality alerting, which requires dynamic profile data, real-time updates of current connection behavior and powerful heuristics to contrast the two, makes for minimal duplicate alerts and fewer false positives.

Challenges and potential drawbacks

Behavioral systems don't specify a threat beyond identifying its type (such as worm, scan, anomalous connection). This drawback relates mainly to human communications; it's easier for people to understand and react to a named entity. Another factor is the additional traffic these systems create as sensors and NetFlow-enabled routers send traffic data across the network. However, for most midsize to large companies, this extra bandwidth usage is more than compensated for by the increased visibility into network behavior.

One installation challenge is ensuring that bad network behavior doesn't get folded into the baseline as normal traffic. This can be avoided during the tuning phase, when actual network-usage data can be accurately compared with the expected traffic according to network plans and topologies. These comparisons make it easy to spot and deal with suspicious activities.

Is behavioral security a fit?

Most behavioral security solutions available today are designed for large, highly distributed networks run by large corporations and government agencies. Behavioral security also makes sense for companies whose revenue-generating processes are highly dependent on network services. This includes companies in the financial services, professional services, large manufacturing, health care and media and entertainment vertical industries.

Behavioral security also makes sense for organizations such as e-commerce companies that are concerned about zero-day attacks. Companies that are opening their networks to many nonemployee credentialed users and semitrusted partners should also consider this technology.

Or not a fit?

These systems aren't well-suited for organizations that are still focused on perimeter network-security technologies, such as antivirus software, firewalls and IDSs. They aren't appropriate for environments where the network is relatively small (less than 100 hosts) or in networks where access is tightly controlled.

Carty Castaldi is vice president of engineering at Mazu Networks Inc., a provider of enterprise-class security products in Cambridge, Mass.

How collaboration apps foster digital transformation
Shop Tech Products at Amazon