2006 Horizon Awards Winner: Splunk's Splunk
This tool indexes all types of data in real time, making the data instantly searchable by keyword.
Many data center problems are easy to solve once you know what's going on. The hard part is finding them in the gigabytes of data dutifully logged on a millisecond basis by all the hardware, databases and applications. Manually combing through all the tiers of log data to track down a transaction or problem is slow and expensive. This is where Splunk comes in, a tool that uses search technology to speed problem resolution.
"Companies have had this fire hose of data thrown at them," says Dana Gardner, an analyst at Interarbor Solutions LLC in Gilford, N.H. "Splunk whittles down this stream so they can exploit the data."
DEVELOPERS: Michael Baum, Erik Swan, Rob Das, Rory Greene, Brian Murphy, David Carasso, Stephen Sorkin, Brad Hall, Andre Stechert, Amritpal Bath, Ivan Tam, Will Hayes, Kim Wallace, Jef Bekes, Nick Mealy, Johnvey Hwang, Ben Strawbridge and Ben Scharp.
San Francisco-based Splunk Inc. was founded in 2003 by three friends — Michael Baum, Erik Swan and Rob Das — who were running large-scale infrastructures dealing with search technology. CEO Michael Baum, for example, was running Yahoo Inc.'s e-commerce applications on more than 12,000 servers. As they discussed their jobs, they found that they were spending a lot of time and resources weeding through log file data with primitive tools. That kicked off a process that eventually led to Splunk.
Initially, they planned to add something to the hardware or application layers that would help system components talk to one another. This, however, would add to the system overhead, so they decided a better approach was to use search technology to give administrators easy access to the data that was already available.
"That's when it really got hard," says Baum. Although the developers had built search technology for companies like Yahoo and Infoseek, Web pages were a lot easier to index than the wide variety of data formats used for data logs.
Then there was the matter of establishing links between the different types of unstructured data. In Web search, the hyperlinks already existed, but not in the data center. So Splunk had to be able to not only access and index all the data in real time, but also establish relevant connections.
"It took us quite a bit longer to develop the technology than we anticipated," Baum says.
The development team behind Splunk (from left to right): Chief Technology Officer Erik Swan, CEO Michael Baum and Chief Architect Rob Das.
Image Credit: Andy Freeberg
Splunk indexes events by time, terms and relationships, and discovers relationships between different kinds of events. Rather than having to go in and look at individual log files, administrators can go into the Web interface and perform a keyword search to find the relevant information in any log file.
They can also search by time or browse event relationships. The index is constantly updated so that an event will show up in a search within seconds of occurring.
Jasmine Noel, an analyst at Ptak, Noel & Associates in New York, says companies with large, complex infrastructures will get the most benefit from using Splunk.
"Today, Splunk's sweet spot is knowledgeable IT experts who have a good idea of what they are looking for but are having difficulty finding it in the haystack of error logs and application dumps from a myriad of different servers," she says.
Like Google, "it automatically indexes everything, but its true power is unleashed when an experienced searcher is looking for something specific," says Noel.
Splunk is available either as a free download, called Splunk Server, or on an annual subscription basis for the full-featured Splunk Professional edition. Pricing ranges from $2,500 for a daily data volume of 500MB to $10,000 for 10GB.
Robb is a Computerworld contributing writer.
Read more about Applications in Computerworld's Applications Topic Center.
- The Shortfall of Network Load Balancing Applications running across networks encounter a wide range of performance, security, and availability challenges as IT department strive to deliver fast, secure access...
- Leave No App Behind with Software Defined Application Services F5 Software Defined Application Services (SDAS) is the next-generation model for delivering application services that enables service injection, consumption, automation, and orchestration across...
- The ADC's Role in the New Network Infrastructure Cloud computing and the BYOD trend will impact the design of future datacenters and their supporting networks. As these trends continue, application delivery...
- Three Key Principles to Accelerating Web Applications Read this article to learn more about the essential principles driving web application acceleration technologies today.
- Transform Your IT Service Management Watch this webinar, to learn how EasyVista can increase IT productivity & efficiency and deliver streamlined & integrated IT Service & Asset Mgmt.
- Top 4 Digital Signage Fails Join RMG Networks for a look at four of the most common reasons digital signage fails in corporate businesses. Learn about strategies to... All Applications White Papers | Webcasts