Five tips for building log management infrastructures

Computerworld - Whether you're building your own logging tool or evaluating a log management solution, there are at least five factors you should consider.
1. The retention period
The log retention period obviously depends on your requirements. If you are building out the infrastructure for troubleshooting and short-term reporting, you may need to keep only one or two months of logs. But if you're doing it so your company can be in compliance with the Sarbanes-Oxley Act or HIPAA regulations, you will need to keep at least six to 12 months' worth for the auditors, and possibly more, depending on your corporate retention policy.
As a rule of thumb, if your requirement is regulatory compliance, make the retention period 12 months to be safe. If you can afford it or the product can support it, go even longer.
How long a retention period you select also depends on the volume of logs you receive as well as the tool's ability to manage the log storage. If you're building your own, be sure to take into consideration a log rotation process. For example, if your retention period is 12 months, your process should remove the old logs or put them on tape. If you are evaluating a product, be sure the product has the capability of rotating/purging old logs for you. Don't spend $200,000 and then have to write your own scripts.
2. Log volume
Log volume is probably one of the most critical factors in building your infrastructure. It has a direct impact on your retention policy, report/search performance, aggregation performance and correlation performance.
Vendors often talk about log volumes in many ways but it all comes down to the number of log messages per second you receive. With that number in hand, you can calculate how much storage space you will need. For example, if your log message rate is 2,000 per second (and 3,600 seconds per hour), assuming 200 bytes per message (which is fairly normal), we have:
2,000 x 3,600 x 24 = 172.8 million messages per day
172.8M x 200 bytes = approx. 34GB per day x 30 days = approx. 1,020GB per month
That's quite a bit of data. This exercise brings up a few things you should be aware of.
First, you need a product or need to develop a solution that can handle the message rate that your environment generates. In this example, get something that can handle at least 3,000 messages per second: 2,000 to meet your requirements, and another 50% for growth and