Formats such SNMP and Netflow are examples of highly structured data which are easily processed by software. On the other hand, system logs such as those generated by UNIX or Windows servers were historically designed for human evaluation rather than machine review. As such, they actually bare a closer resemblance to an email message rather than that of an event record. Much like email headers, Syslogs and Windows Event Logs have basic fields which convey only metadata such as the category, severity, and time of the event. The relevant information pertaining to the event itself is actually embedded within the "message" field of the record, which is analogous to the body of an email message. Just as there is there no standardize format for composing an email message, the placement of vital event information is often sporadic and comingled within other key data that really should be logically separated into their own discrete fields.
Coming full circle, this is where SIEM-type systems prove their worth through the data normalization process. That is, the intelligent parsing and transformation of loose data into structured information. Obviously, how well a given solution performs this task should be closely scrutinized during the product analysis phase.
Once the technological hurdles of a log management system have been addressed, the final stage of deployment is to compile a proper log management policy. Policies essentially define what to log, how the logs are recorded, and for how long logs are to be retained. Logging policies are frequently requested during times of legal action such as lawsuits or public records requests. Documenting what to log serves as the record of decision with respect to what is recorded versus that which is discarded.
Therefore, having language that clearly defines what is recorded and for how long it is retained can make the difference between efficient records requests and prolonged, time-draining depositions. Be aware that if records are requested from your organization, the requesting entity will most likely be using a completely different technology tool set. As such, it's imperative that copies of your original logs are kept in their native, unaltered state. If this isn't possible, then at the very least, logs should be easily exportable to a standardized format without loss of relevant information.
From the procedural perspective, log management policies should serve as a model for how logging is actually implemented across all enterprise systems. Areas of focus should include accurate time-keeping, log rotation and archival routines, as well as testing procedures to ensure the policies are upheld at all times.
Reference:
Kent, Karen and Souppaya, Murugiah. "Special Publication 800-92". NIST. September 2006 http://csrc.nist.gov/publications/PubsSPs.html
David Torre is an experienced security professional and CTO of Atomic Fission.
Read more about network security in CSOonline's Network Security section.
This story, "Log management basics" was originally published by CSO.