The Logs Talk!

| Comments

A couple of months ago I was looking into a method to store the server logs for troubleshooting and auditing purpose. I ended up giving up because of time.

Recently I attended the MacSysAdmin in Sweden and Ed Marczak gave a presentation about the importance to keep the log files and several useful tips. You can view it here.

So with the excitement back in the table I decided to retake the task and my first choice to centralize the log files from different servers was Splunk.

For those in NY and even for those somewhere else here is a quick installation and configuration tutorial to give you the momentum (that every nerd needs!) and get you playing with it.

In the example I am adding the log files from 4 DeployStudio servers running Mac OSX 10.6.8 into one of them to mainly track usage and troubleshoot.

Download splunk-4.3.4-136012-macosx-10.5-universal.dmg and install it onto the server.
Login to the server and open a Terminal
Run the following command as administrator

bash-3.2$ sudo /Applications/splunk/bin/splunk start --accept-license

Accept the license
Open and login as admin/changeme
Change the password when requested
In my case as the server that receives the logs is also a DeployStudio server I added some logs from the local filesystem. If you want to do this click on Source From files and directories, then browse the local filesystem and add the logs to monitor.
I would recommend to monitor for now /var/log/system.log and /var/log/secure.log* (Please read the security note in red below!!)

Because the web interface does not seem to be monitoring folders properly, at least for me, let’s add the DeployStudio logs folder from the terminal. Again on the logs server do

bash-3.2$ sudo /Applications/splunk/bin/splunk add monitor /WhateverFolder/DeployStudio/Logs/

Last thing on the server is to configure the port to receive the remote logs. Go to Manager > Forwarding and receiving > Receive data and add a new port. The default is 9997. Save and done for now on the server.

Now to the other servers or log senders. Install splunkforwarder-4.3.4-136012-macosx-10.5-universal.dmg and login to them.
The following three commands will get Splunk running, add a launch item and configure Splunk to send the logs.

bash-3.2$ sudo /Applications/splunkforwarder/bin/splunk start --accept-license<
bash-3.2$ sudo /Applications/splunkforwarder/bin/splunk enable boot-start

Changing the password on the forwarder servers may be a good idea!

bash-3.2$ sudo /Applications/splunkforwarder/bin/splunk add forward-server YourCoolDNSName:9997 -auth admin:changeme

Now add the same log files to monitor

bash-3.2$ sudo /Applications/splunkforwarder/bin/splunk add monitor /WhateverFolder/DeployStudio/Logs/
bash-3.2$ sudo /Applications/splunkforwarder/bin/splunk add monitor /var/log/secure.log*
bash-3.2$ sudo /Applications/splunkforwarder/bin/splunk add monitor  /var/log/system.log

Done with the client!

Login to the serverhttp://YourCoolDNSName:8000 and confirm that the log files from the remote servers start to appear.

Some things you can now do:
Search this string to get the top 50 users authenticated to DeployStudio. Check out the chart views and you can create a dashboard if it is useful

source="/WhateverFolder/DeployStudio/Logs/*.log" "was successfully authenticated." | rex "(?i)The user '(?P<USERNAME>[^']+)" | eval USERNAME=lower(USERNAME) | top 50 USERNAME

This one for failed login attempts

source="/WhateverFolder/DeployStudio/Logs/*.log" failed | rex "(?i)Authentication failed for user '(?P<USERNAME>[^']+)" | eval USERNAME=lower(USERNAME) | top 50 USERNAME

This one for which workflow execution

source="/WhateverFolder/DeployStudio/Logs/*.log" "Running workflow:" | rex "(?i)Running workflow: '(?P<WORKFLOWNAME>[^']+)" | top 100 WORKFLOWNAME

And this one for runs over time

source="/Volumes/Data/DeployStudio/Logs/*.log" "Running workflow: " | timechart span=1d count

There are many more useful things you can do. Maybe you want to track ARD connections

source="/var/log/secure.log" ":: viewer address: " | rex "(?i):: Viewer Address: (?P<FIELDNAME>[^ ]+)" | top 50 FIELDNAME

Yes these search strings are more for monitoring than troubleshooting but that I keep to you :o Adding the logs that are meaningful to you is trivial now.

Ed Marczak talked in the MacSysAdmin about a more *nix way of doing it by redirecting Mac OSX syslog calls to an IP address. I won’t extend much on this one because the splunkforwarder method above seems to give more granularity. But if you fancy to try this UDP! method execute this in the computer you want to monitor:

bash-3.2$ sudo echo "*.*     @YourCoolDNSName:9997" >> /etc/syslog.conf

The disclaimer contained in the /etc/syslog.conf* (Please read the security note in red below!!) is worth mentioning here for security reasons so here it comes a copy and paste (from Mac OSX10.6):

The authpriv log file should be restricted access; these messages shouldn’t go to terminals or publically-readable files.;authpriv.*;remoteauth.crit /var/log/secure.log This is why you may want to do a more granular edit of /etc/syslog.conf and/or limit the access to your logs server

Add-on: Gary L. from Puppet labs mentioned that the downside of Splunk (and it is a big one) is the free license limit. Splunk will run for free up to 500MB of logs per day and anything after that you need to pay a (costly?) license.

So far with a couple of servers and a handful of logs from each one I am seeing a usage of <5% from the free limit but this may not be your case. So take this into account before you spend hours configuring and playing with the dashboards, email reports and stuff.

Also have to say that if you are a DeployStudio client?/user you know the problems from free but not open source software!
If that sentence made a spark in your brain you may consider logstash instead.

I right now don’t have that momentum I was talking before nor the time to play with logstash. But when I do my thoughts here you shall find. Not sure if that last sentence sounds more Yoda from Star Wars or some bible thing.