Spelunking with Spunky Splunk January 10, 2006
Posted by James Webster in : software , trackbackIf you have visited Sourceforge or Slashdot recently, you may have come across a banner ad for Splunk, “a search engine for your IT data.”
The day job had some challenging production issues which are now behind us, we hope! One of the key challenges we faced in diagnosing them however was the sheer mass of log data to wade through; several different app server logs, our own log files; all of which was being generated by several machines. On top of that, some of the logs have lines with different formats (no thanks to the proliferation of umpteen different logging libraries in the Java world).
It was almost providence that I happened upon Splunk. As it is *nix only (and this is a Windows shop) I was able to install it into the Browser Appliance (Ubuntu Breezy) virtual machine running under the free VMware Player. Its easy to upload log files via Splunk’s web interface, or you can modify an XML file to have Splunk Server monitor a number of directories for new log files which are transparently uploaded and indexed. The latter gives you greater opportunity for identifying which hosts are responsible for generating which logfiles. Hey Splunkers! Can we have a capability for doing all that via the web interface?
Splunk recognises a whole bunch of different file formats natively, but more importantly it can deal with multiline log messages. All log messages of a similar type are indexed under the same event type. Event types can then be tagged, a la Del.icio.us. Queries can be run across tags, time periods, hosts, etc, etc, etc. The Splunk Server is free, and they have a professional version that supports multiple users, saved queries, etc, etc, etc. Look, just go check out the demo already!
Comments»
Hi there from Splunk. Glad to know it worked well for your problem. Re your question on specifying host and other config parameters via the GUI - coming. We’re designing it right now. Look for it within the next few releases. Would love to hear from anyone with specific requests for how it should work via support@splunk.com.