Log management in the age of compliance

'Bread crumbs' are key to what's happening with your network

With each high-profile data breach (such as those at The TJX Companies and the U.S. Department of Agriculture) or new regulation, security emphasis seems to shift away from the traditional "keep bad guys out" mentality and toward a layered, in-depth, "What's going on in here?" look at IT activity. Organizations are turning to logs to provide a continuous trail of everything that happens with their IT systems and, more importantly, with their data.

Logs of different types are generated from different sources at an astounding rate, allowing for a detailed -- if sometimes cloudy -- picture of IT activity. If a disgruntled employee with an intent to steal data accesses a database containing confidential information, there would likely be a log of that activity that someone could review to determine the who, what and when. Logs provide the bread crumbs that organizations can use to follow the paths of all of their users, bad-intentioned or not.

It follows that managing these logs can benefit an organization in many ways. They offer situational awareness and help organizations pinpoint new threats as well as allow their effective investigation. Routine log reviews and in-depth analysis of stored logs are beneficial for identifying security incidents, policy violations, fraudulent activity and operational problems shortly after they have occurred, as well as for providing information useful for resolving such problems.

Given the inherent benefits of log management, it is not surprising that log data collection and analysis is generally considered a security industry "best practice." However, a number of regulations also explicitly call for the collection, storage, maintenance and review of logs, turning log management from a "should do" to a "must do." Some of these regulations rely on National Institute of Standards and Technology Computer Security Special Publications (NIST SP) to delineate the detailed logging requirements.

In my previous article, I described the way in which three regulations (FISMA, HIPAA and PCI-DSS) affect incident-response processes. This triumvirate also affects log management, since they call for enabling logging as well as for log review.

The Federal Information Security Management Act of 2002 (FISMA)

While many criticize FISMA for being all documentation and no action, the law simply emphasizes the need for each federal agency to develop, document and implement an organizationwide program to secure the information systems that support its operations and assets. NIST SP 800-53, Recommended Security Controls for Federal Information Systems, describes log management controls including the generation, review, protection and retention of audit records, plus steps to take in the event of audit failure.

NIST 800-92, Guide to Computer Security Log Management, also created to simplify FISMA compliance, is fully devoted to log management. It describes the need for log management in federal agencies and ways to establish and maintain successful and efficient log management infrastructures -- including log generation, analysis, storage and monitoring.

NIST 800-92 discusses the importance of analyzing different kinds of logs from different sources and of clearly defining specific roles and responsibilities of those teams and individuals involved in log management. Importantly, Section 4.2 highlights the need for organization to clearly define its policy requirements (based on the appropriate regulations) for performing logging and monitoring logs, including log generation, transmission, storage and disposal, as well as explicit protections for these logs.


The Health Insurance Portability and Accountability Act of 1996 (HIPAA) outlines relevant security standards for health information. NIST SP 800-66, An Introductory Resource Guide for Implementing the Health Insurance Portability and Accountability Act Security Rule, details log management requirements for the securing of electronic protected health information. Section 4.1 of NIST 800-66 describes the need for regular review of information system activity, such as audit logs, access reports and security incident-tracking reports. Also, Section 4.22 specifies that documentation of actions and activities need to be retained for at least six years.

While the debate about whether logs can be considered documents is not finished, some organizations do choose to store logs for as long as other business documents. In addition, Appendix A of this document encourages organizations to ask a variety of log-related questions, including whether or not system performance monitoring is used to analyze system-performance logs in real time in order to spot availability problems such as active attacks.


The Payment Card Industry Data Security Standard (PCI-DSS), which applies to organizations that handle credit card transactions, mandates logging specific details and log review procedures to prevent credit card fraud, hacking and other related problems in companies that store, process or transmit credit card data.

Even though logging is present in all PCI requirements, PCI DSS also contains Requirement 10, which is dedicated to logging and log management. Under this requirement, logs for all system components must be reviewed at least daily, and these log reviews must include servers that perform security functions, such as intrusion-detection systems and authentication, authorization, and accounting protocol servers.

Further, PCI-DSS states that the organization must ensure the integrity of its logs by implementing file integrity monitoring and change detection software on logs to ensure that existing log data can not be changed without generating alerts. It also prescribes that logs from in-scope systems are stored for at least one year.

And beyond

There are a variety of other regulations that call for log management capabilities, although less explicitly than the aforementioned three. California Bill 1386 and its upcoming federal equivalent, for example, require a state agency, person or business that owns or licenses computerized data that includes personal information to disclose any breach of the security of the data to any California resident whose unencrypted personal information was acquired by an unauthorized person.

Logs, which by nature allow for tracking IT infrastructure activity, are the best way to assess if, how, when and where a data breach has occurred. Management of these logs is therefore the best way to assess what data has been accessed or stolen and, thus, who needs to be notified.

The major effect the age of compliance has had on log management is to turn it into a requirement rather than just a recommendation, and this change is certainly to the advantage of any organization subject to these regulations. It is easy to see why log collection and management is important, and the explicit inclusion of log management activities in major regulations like FISMA, HIPAA and PCI-DSS highlights how key it truly is to enterprise security as well as broader risk management needs.

The next article in this series will cover intrusion detection in the age of compliance.

Anton Chuvakin, GCIA, GCIH, GCFA is a recognized security expert and author. His current role is chief logging evangelist at LogLogic Inc., a log management and intelligence company. A frequent conference speaker, he also represents the company at various security meetings and standards organizations. He is a co-author of Security Warrior (O'Reilly Media, 2004) and a contributor to Know Your Enemy II, Information Security Management Handbook, Hacker's Challenge 3 and an upcoming book on PCI. He has published numerous papers on a broad range of security subjects and has several blogs. 

Copyright © 2007 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon