Five tips for building log management infrastructures

By Jian Zhen

April 14, 2005 12:00 PM ET

Computerworld - Whether you're building your own logging tool or evaluating a log management solution, there are at least five factors you should consider.
1. The retention period
The log retention period obviously depends on your requirements. If you are building out the infrastructure for troubleshooting and short-term reporting, you may need to keep only one or two months of logs. But if you're doing it so your company can be in compliance with the Sarbanes-Oxley Act or HIPAA regulations, you will need to keep at least six to 12 months' worth for the auditors, and possibly more, depending on your corporate retention policy.
As a rule of thumb, if your requirement is regulatory compliance, make the retention period 12 months to be safe. If you can afford it or the product can support it, go even longer.
How long a retention period you select also depends on the volume of logs you receive as well as the tool's ability to manage the log storage. If you're building your own, be sure to take into consideration a log rotation process. For example, if your retention period is 12 months, your process should remove the old logs or put them on tape. If you are evaluating a product, be sure the product has the capability of rotating/purging old logs for you. Don't spend $200,000 and then have to write your own scripts.
2. Log volume
Log volume is probably one of the most critical factors in building your infrastructure. It has a direct impact on your retention policy, report/search performance, aggregation performance and correlation performance.
Vendors often talk about log volumes in many ways but it all comes down to the number of log messages per second you receive. With that number in hand, you can calculate how much storage space you will need. For example, if your log message rate is 2,000 per second (and 3,600 seconds per hour), assuming 200 bytes per message (which is fairly normal), we have:
2,000 x 3,600 x 24 = 172.8 million messages per day
172.8M x 200 bytes = approx. 34GB per day x 30 days = approx. 1,020GB per month
That's quite a bit of data. This exercise brings up a few things you should be aware of.
First, you need a product or need to develop a solution that can handle the message rate that your environment generates. In this example, get something that can handle at least 3,000 messages per second: 2,000 to meet your requirements, and another 50% for growth and

Recommend Digg Twitter ShareThis

Email Print Send Feedback Reprints

Additional Resources

POLL RESULTS

Leading IT Pros Weigh In on Top IT Issues

Accelerate your knowledge of the IT world you inhabit by viewing the results of a series of polls taken by your IT peers. These polls of 100+ IT professionals each are available for full viewing. They cover key topics such as virtualization, processor performance, green IT, cloud computing and many others. Be a part of the buzz.

WHITE PAPER

For Best Results, Think Beyond the Box

Technology is complex. Keeping it running productively shouldn't be. To that end, you want to minimize the number of solutions needed in-house to simplify operations, maintenance, and support. Kodak offers a best-practices model. One company provides support for both scanner and software, for fast problem resolution without vendor finger-pointing. Download now!

WHITE PAPER

Accelerating the Path to Demand Intelligence with a Demand Signal Repository

Utilizing demand intelligence improves the precision of pricing, product assortments, channel/store placement, and promotion, which are all essential for sustainable revenue management performance. Learn more, download this free whitepaper today.

White Papers & Webcasts

Centralized Data Backup and Your WAN
Is your organization prepared to tackle the massive challenge of protecting your data in a cost effective and timely manner? With a growing...

Why Compliance Pays
This OnDemand webcast explores the relationship that firms with best compliance records have higher revenue, greater customer retention, lower financial losses from data...

An All-in-One Approach to Web Security
Granting web access to employees poses challenges to IT administrators and introduces unique security risks. Even as companies have perfected their security techniques...

Best Practices for Managing Business Risks from the Use of IT
(Source: Symantec) Based on exhaustive benchmarks conducted by the IT Policy Compliance, this session highlights the relationship between business risks and use of...

The Hidden Dangers of Spam
Beyond the well-understood productivity drain that spam inflicts on businesses, threats posed by illicit email circulating through a network are causing many security...

Managing And Protecting Your Ever Increasing Mobile Assets
(Source: Absolute Software) Your users are becoming more mobile each day. This is great for productivity - yet challenging for IT control. Natalie...

Open Source Security Myths Dispelled
(Source: Astaro) Open Source Software is computer software whose source code is available to the general public. This openly viewable nature...

Sun OpenSSO Enterprise Webinar
(Source: Sun) This webinar replay discusses Sun OpenSSO Enterprise innovation--the single, open-source solution that helps your business solve the challenges around internal access...

Best Practices for Backing Up VMware® with Veritas NetBackup™
VMware® is used by enterprises large and small to increase the efficiency and cost-effectiveness of their IT operations. With this in mind, Symantec...

Agile Enterprise Content Management (ECM) for Rapid ROI
(Source: IBM) Content rich business processes are a core feature of daily operations at just about any organization today. Very often these essential...

All White Papers | All Webcasts