InfoWorld review: Meeting the network security and compliance challenge

Log management is one of those necessary tasks that every company should do, but that few companies do consistently well. Collecting and analyzing computer and device logs can pay off in many areas, including information security, operations management, application monitoring, system troubleshooting, and compliance auditing. A good log management solution can help with any -- or all -- of these efforts.

Security auditing may be the No. 1 reason why many companies first investigate log management tools. Verizon's "2008 Data Breach Investigations Report" PDF, which is quickly becoming one of the most respected sources on computer crime statistics, said it best: "Evidence of events leading up to 82 percent of data breaches was available to the organization prior to actual compromise. Regardless of the particular type of event monitoring in use, the result was the same: Information regarding the attack was neither noticed nor acted upon."

This review covers seven different hardware and software solutions for log management: ArcSight Logger 4.0, GFI EventsManager v.8.2, LogLogic MX3020 v.4.9.1, LogRhythm LR2000-XM v.5.0, NitroSecurity NitroView ESM and ELM, Splunk 4.1.2, and Trustwave SIEM.

The goal of this review is to expose readers to a general cross-section of log management features and functionality, including what features set the different solutions apart. It's important to note that while we rank each product across a common set of evaluation criteria (on a scale of 1 to 10, 10 being the highest), the products are often dissimilar to one another -- they are often different classes of products.

For example, ArcSight's single-appliance Logger is strictly a log management solution and therefore lacks a number of features found in NitroSecurity's two-appliance SIEM (security information and event management) solution. My evaluation of both products -- and all the others in this review -- focused only on log management capabilities, and the product scorecards reflect only their log management features. I did not evaluate real-time event correlation -- the hallmark of the SIEM solution -- though I do note in the reviews and the product comparison table where those features are present. It's usually a good thing when a solution offers more capabilities at a given price point.

The product features and functions I did evaluate are those related to collecting, storing, and reviewing the wide variety of event logs a company might want to watch closely. While you won't need a complete and detailed understanding of log management to follow this product review, you might keep in mind the several distinct phases of the log management lifecycle: policy definition, configuration, collection, normalization, indexing, storage, correlation, baselining, alerting, and reporting. (You'll find summaries of these phases in the sidebar, "Living the log management lifecycle," and a more thorough treatment in my downloadable report, "Log Analysis Deep Dive: Finding Gold in Log Files.") The specific product features I examined, and the most important differences among products in this category, are explored in the remainder of this article.

Testing was done in a small private lab with 15 to 20 computers (some physical, some virtual), mimicking a small-business network with Windows, Linux, BSD, routers, and wireless clients. At times, some of the functionality was viewed when the product was running on larger, real production networks or on a remote lab created by the vendor, when more clients better demonstrated a particular feature.

I did not test vendor performance or compression reports, both of which are often exaggerated. Some vendors felt this was unfortunate because one of their strongest claims of competitive advantage was in dealing quickly with huge amounts of data. We recommend testing real-life performance before buying any log management product. This author has seen many log management products perform well when handling a few hundred machines but slow to a crawl when handling a few thousand computers.

In a pleasant turn of events (excuse the pun), I felt all of the reviewed items were solid products ready to be deployed on any company's network. Not one of the products tested would fail to provide value, although of course some would provide more value than others. Every reviewed product worked as advertised, had a myriad of useful features, and was mature enough to be used in a production environment. The top goal of this review was to highlight the features that made each product competitively distinct so that readers can decide which ones might make sense for testing in their environment.

Log management evaluation guide This section will discuss the various features available in each of the log management products tested and should help provide a framework for evaluating any other log management solution. For the seven products reviewed, I've also included a handy log management product comparison spreadsheet to help in your evaluation.

One of the first decisions to be made is whether to use an all-inclusive appliance or a software-based product. Most log management products come as appliances simply because appliances typically handle the performance and storage requirements more easily than a software product running on a general-purpose operating system. Yes, it is true that administrators could configure and optimize a software product's underlying host OS to be as efficient as an appliance -- after all, an appliance is just an operating system host running log management software. With appliances, however, the hard configuration and optimization work is already done.

The downside of appliances is that they tend to be limited to a few off-the-shelf configurations and disk capacities, and their underlying operating system -- often a Linux distro or Microsoft Windows -- may be harder to patch. Although most of the appliance vendors in this review claimed to keep the underlying host patched and up-to-date as a part of their normal product upgrades (which are often automated), I found many products running older versions of code, such as the Apache Web server, with known vulnerabilities for which patches are available. If you decide to use an appliance, ask the vendor whether they update the underlying OS quickly when patches are available; if allowed under the licensing agreement, also consider testing the product for vulnerabilities before buying.

Workload distribution. Most of the products tested provided all-in-one functionality, meaning their product would act as management console, data collector, storage device, indexer (for search queries and filters), and report generator. In addition, most products could be configured to serve in just one role or multiple roles without performing all roles.

Workload distribution is incredibly important if you plan to collect log messages from more than a few hundred clients. Not that the log management tool itself poses a bottleneck -- if an appliance, it will usually have four or more Gigabit Ethernet interfaces -- but a network can sustain only so much additional traffic without causing application and operation performance issues. Sending log messages from 1,000 computers to a single log manager can bring any network to its knees.

Work with the vendor to figure out log management instance roles and distribution to maximize performance in your environment. Every product in this review can act as a store-and-forward collector for its own products, meaning you can have one instance collect all the local traffic before forwarding the data, often compressed, to a centralized, "master" log management instance. Many of the products could also forward events to other products, especially those that support syslog and SNMP. Several products, including both software and appliances, could act as collectors only or as indexers, which tend to be the two most CPU-intensive operations.

Give the vendor your network's dimensions (network bandwidth, available capacity, and number of clients that will be monitored) and your log management plans. Then let the vendor respond with their recommended distributed configurations. With appliances, this will often result in different hardware models in different locations.

Performance is not only important in avoiding network congestion issues, but also when analyzing real-time or historical data, printing reports, and doing more involved forensic analysis. When you have tens of millions to billions of event messages to work with, you don't want to be waiting 10 minutes for a simple query to return. If your solution involves multiple log management nodes, make sure that queries and reports can work across "peers," meaning that one click in a management console will execute searches and reporting across all product instances. And test the performance differences and features when crossing peers. A few of the products have more limited features when searching across peers. All of the products tested are fairly flexible regarding workload distribution, with the lone exception being GFI EventsManager.

Most vendors will claim that they can work with environments of any size. And many vendors claim to have installed solutions handling tens of billions of messages per day, without client complaint. Ask for customer references, get any performance guarantees in writing, and test thoroughly before committing to a big-dollar purchase.

Management dashboard. Every log management product has management console dashboard that displays crucial real-time and short-term summary statistics about the log management system itself and the monitored events. Most dashboards include event messaging counts, local CPU performance, and notification about any critical events.

Almost all allow customization of the dashboard and let you configure what is shown by user or role. In most cases, but not all, dashboard displays are context-sensitive. You can click on a displayed graph to get a more detailed drill-down. A few, like NitroSecurity, allow extensive modifications where almost any metric, graph, or alert can be shown.

User roles are important, as most products allow administrators (which have full privileges) to set more limited roles. For example, some products allow limited admins to be defined, in cases where administrator-level privileges are needed but only regarding a predefined set of clients: all Windows machines, all Cisco routers, and so on. Most products have a read-only role where none of the configuration settings can be modified, but the users can run reports and see predefined graphs and metrics. Most of the products allow only two to four roles to be defined, and only allow administrators to define which screens are displayed. A few others, including Splunk, NitroSecurity, and LogLogic, allow extensive role definitions where each attribute and field on a screen can be defined per role.

Log collection. Collecting the log information from all the various monitored clients is the backbone of any log management product. Most products have both agentless methods and client agents to collect logs. Not having an agent means administrators don't need to distribute, install, and configure additional software to every client. However, agentless log collection still requires planning and work. Most products collect logs using syslog forwards, WMI queries, or other remote methods (the last two usually require client administrative passwords). All require the necessary rule modifications if firewalls are involved. Whatever you do, don't think of agentless as no work or it will surprise you.

Client agents have benefits that agentless collection methods have a hard time meeting. Most agents have multiple configuration options that allow administrators to have finer-grained control over what events are collected and how. For instance, instead of sending every log message to the centralized server, an agent can just send critical events, and store the rest locally for later retrieval if needed. Client agents can often offer transmission compression, allowing more events to be sent in less time and with smaller network bandwidth utilization, although it's doubtful you'll get the superior compression statistics that each vendor advertises in real-life scenarios.

Monitored clients can be added one at a time (usually via IP address or domain name), using mass importing (to add multiple devices at once), or using some sort of initiated querying process (usually through Active Directory browsing or IP address scanning). Most products allowed "device groups" to be created, to collect one or more monitored clients under a given group name as determined by some attribute -- for example, by device type, IP address, or name. Device groups can then be monitored as a single entity making alerting and reports easier to accomplish when trying to focus on a particular device class.