The primary purpose of most IT management tools is to identify problems within your IT systems. Whether the problem is a down server or WAN link, a saturated fiber channel or trunk port, an Exchange store or SAN queue backing up or etc -- the jobs of the IT tools are to detect these issues and to help you resolve them. However, sometimes these tools are used to forensically analyze system performance for a specific time in history; and when they are, things can get pretty interesting.
I was in Washington, D.C. recently to meet with some folks whose jobs are to take IT management system data from previous points in history, and use forensic science to analyze data, answer questions about performance over time and to make recommendations for future IT systems parameters.
OK, that sounds complicated. Let me give you an example:
Let's say that a U.S. Marine Corps unit is deployed to Afghanistan for six weeks. During that time they have some issues with connectivity of their hand-held radios, some VoIP performance issues, and a security incident with one of their e-mail servers. Now remember, they're Marines, so they figure it out in real-time and the mission doesn't suffer. A few months later, the IT management system data that was captured during that time is analyzed to offer a better understanding of what went wrong, how it happened, and how it can be avoided in the future.
The same type of thing happens in the commercial world all the time. Suppose that your company has an application outage that slows down order processing for a few hours. You work like crazy, utilizing your IT management tools to get things up and running again. However, a few months later an auditor asks you to use forensics to provide some more detail on what exactly occurred.
Using IT tools for forensics is one of my favorite things to do because you can learn so much about what's going on within your network and how your systems and applications are functioning. However, because this use case isn't the primary one in mind when IT management systems are designed, built, and deployed, using them for forensics can be difficult and usually requires some expert planning.
We've all probably used NetFlow-based applications to identify application slowdowns and network bottlenecks. NetFlow typically generates a tremendous amount of data, and so most IT tools implement specialized algorithms for compressing, summarizing, and storing the data. I myself hold a couple of patents in this area. If you didn't compress the data, your databases would grow uncontrollably and your ability to access the data would be significantly slowed. However, the compression and summarization can wreak havoc on the forensics use case if you don't really watch what you're doing.
The best way to learn more about using your own IT management tools for forensics is to give it a try. Create a situation this week that is easy to see in real-time. Don't do anything stupid here and bring down one of your production systems, but do something that's easy to identify. Then, wait a week and go looking for your problem. Yes, you're going to run into issues, but it's better to know about them now than to learn about them when you're really under the gun and an auditor is breathing down your neck.
Have you had interesting experiences using IT tools for forensics analysis? If so, please share your stories here.
Flame on...
Josh
Josh Stephens is Head Geek and VP of Technology at SolarWinds, an IT management software company based in Austin, Texas. He shares network management best practices on SolarWinds GeekSpeak and thwack. Follow Josh on Twitter @sw_headgeek and SolarWinds @solarwinds_inc.