2006 Horizon Awards Winner: Splunk's Splunk
This tool indexes all types of data in real time, making the data instantly searchable by keyword.
Many data center problems are easy to solve once you know what's going on. The hard part is finding them in the gigabytes of data dutifully logged on a millisecond basis by all the hardware, databases and applications. Manually combing through all the tiers of log data to track down a transaction or problem is slow and expensive. This is where Splunk comes in, a tool that uses search technology to speed problem resolution.
"Companies have had this fire hose of data thrown at them," says Dana Gardner, an analyst at Interarbor Solutions LLC in Gilford, N.H. "Splunk whittles down this stream so they can exploit the data."
DEVELOPERS: Michael Baum, Erik Swan, Rob Das, Rory Greene, Brian Murphy, David Carasso, Stephen Sorkin, Brad Hall, Andre Stechert, Amritpal Bath, Ivan Tam, Will Hayes, Kim Wallace, Jef Bekes, Nick Mealy, Johnvey Hwang, Ben Strawbridge and Ben Scharp.
San Francisco-based Splunk Inc. was founded in 2003 by three friends — Michael Baum, Erik Swan and Rob Das — who were running large-scale infrastructures dealing with search technology. CEO Michael Baum, for example, was running Yahoo Inc.'s e-commerce applications on more than 12,000 servers. As they discussed their jobs, they found that they were spending a lot of time and resources weeding through log file data with primitive tools. That kicked off a process that eventually led to Splunk.
Initially, they planned to add something to the hardware or application layers that would help system components talk to one another. This, however, would add to the system overhead, so they decided a better approach was to use search technology to give administrators easy access to the data that was already available.
"That's when it really got hard," says Baum. Although the developers had built search technology for companies like Yahoo and Infoseek, Web pages were a lot easier to index than the wide variety of data formats used for data logs.
Then there was the matter of establishing links between the different types of unstructured data. In Web search, the hyperlinks already existed, but not in the data center. So Splunk had to be able to not only access and index all the data in real time, but also establish relevant connections.
"It took us quite a bit longer to develop the technology than we anticipated," Baum says.
The development team behind Splunk (from left to right): Chief Technology Officer Erik Swan, CEO Michael Baum and Chief Architect Rob Das.
Image Credit: Andy Freeberg
Splunk indexes events by time, terms and relationships, and discovers relationships between different kinds of events. Rather than having to go in and look at individual log files, administrators can go into the Web interface and perform a keyword search to find the relevant information in any log file.
They can also search by time or browse event relationships. The index is constantly updated so that an event will show up in a search within seconds of occurring.
Jasmine Noel, an analyst at Ptak, Noel & Associates in New York, says companies with large, complex infrastructures will get the most benefit from using Splunk.
"Today, Splunk's sweet spot is knowledgeable IT experts who have a good idea of what they are looking for but are having difficulty finding it in the haystack of error logs and application dumps from a myriad of different servers," she says.
Like Google, "it automatically indexes everything, but its true power is unleashed when an experienced searcher is looking for something specific," says Noel.
Splunk is available either as a free download, called Splunk Server, or on an annual subscription basis for the full-featured Splunk Professional edition. Pricing ranges from $2,500 for a daily data volume of 500MB to $10,000 for 10GB.
Robb is a Computerworld contributing writer.
Read more about Applications in Computerworld's Applications Topic Center.
- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- What Datapipe customers need to know about the new PCI DSS 3.0 compliance standard This handy quick reference outlines what PCI DSS 3.0 is, who needs to be compliant and how Alert Logic solutions address the new...
- The 12 PCI DSS 3.0 requirements addressed by Peer 1 Hosting This handy quick reference outlines the 12 PCI DSS 3.0 requirements, who needs to be compliant and how Alert Logic solutions address the...
- Defense Throughout the Vulnerability Life Cycle This whitepaper provides insight into how to leverage threat and log management technologies to protect your IT assets throughout their vulnerability life cycle.
- The Critical Role of Support in Your Enterprise Mobility Management Strategy Most business leaders underestimate the importance of tech support when they choose an EMM solution. Here's what to put on your checklist.
- Live Webcast Best Practices for the Hyperconverged Enterprise Network To the Age of Constant Connectivity and Information overload
- Live Webcast Unmasking the Differences between Consumer and Enterprise File Sync & Share The consumerization of IT combined with the rapid pace of the modern mobile workplace is forcing enterprise IT teams to evaluate file sync...
- Live Webcast Government Agency Webifies Outdated COBOL Applications Let this CTO tell you how his agency converted 1980s-era green screens into an e-filing portal for the 100,000 cases handled each year...
- The New Way to Work Knowledge Vault This Knowledge Vault focuses on how, in today's increasingly virtual world, it's more important than ever to engage deeply with employees, suppliers, partners,...
- Getting Ready for BlackBerry Enterprise Service 10.2 Find out how BlackBerry® Enterprise Service 10 helps organizations address the full spectrum of EMM challenges, while balancing the needs of both the... All Applications White Papers | Webcasts