Thursday, October 30, 2014

Monitoring External Scheduled Jobs Using Jenkins

Recently we had a requirement to create a dashboard so we could monitor our scheduled jobs that were important for the business. The jobs ran on different server boxes. Every now and then when we had issues with a any job, we needed to get access to the system and dig the log files. Worst of all was that we would only know about a problem with the job only when someone reported it. We wanted to build a dashboard so were on top of the jobs' status and fix issues ASAP.  Some research from our team suggested that Jenkins could help us with that.

We got an instance of Jenkins running very quickly and started playing around it. Below is what we did to get a dashboard going.

Jenkins as a Monitoring System

Firstly, we created 'Monitor an External Job' item in Jenkins. Let's call it "JobMonitor1". The configuration required for such an item in Jenkins is very minimal. All you need to do is configure how long do you want the builds (job running records) for.

Now that we had a job - we needed to get the actual JOBs that are running in a different environments to let Jenkins know that it has run. This is where the Monitoring External Jobs plugin came very handy. The plugin page shows how to use this plugin. In summary, we installed the plugin in our Jenkins instance and then got our JOB to send feedback to Jenkins. If we currently ran our job in this manner - bash runMyBash.sh, we updated it to run as follows -

java -jar /path/to/jenkins-core-*.jar "JobMonitor1" bash runMyBash.sh.

We needed to make sure some of the required jar files were available in the classpath. To let the system know which Jenkins instance to post the results to, we needed to set JENKINS_HOME in the system by running the command below.

EXPORT JENKINS_HOME= /path/to/jenkins (for Linux Systems)

SET JENKINS_HOME= /path/to/jenkins (for Windows)


When our script (runMyBash.sh) completed, it sent the details of the jobs to "JobMonitor1" and we could then monitor it in Jenkins. The downside of this approach was that we could not tell from viewing Jenkins whether the job was running. Jenkins is only notified after the process completes. We do however know how long it took for the job to complete.

Monitoring the Monitor

Now that we had the visibility of whether the last ran, passed or failed, to make it a bit more real-time, we needed to know whether the JOB ran when it was expected to run. For example, if a job is expected to run every hour, did it run every hour?

This is where it gets little dirty. We decided to create another item (e.g. "SecondLevelMonitorJob") in Jenkins that would monitor the  Jobs that  were getting notified by the remote system. So we created new 'Freestyle Project' item in Jenkins that would monitor the other job. But how exactly do you monitor it? This is where we found another plugin - check_jenkins_cron. This is simply a script that runs to see if a job that you specified ran in the time you specified.

We wrote a wrapper around this plugin to make the build fail or pass depending on what the plugin returned. So the "SecondLevelMonitorJob" would run our wrapper script. We specified in this JOB how often we expected the job to run. We made  "SecondLevelMonitorJob" run every 15 minutes to monitor a job that is expected to run every 6 hour. This way we picked up the fact that something is wrong with the job ASAP.

Dashboard

And the final piece - the Dashboard. After a lot of investigation on appropriate "Views" for Jenkins, we settled for Wall Display plugin. Some of the others we looked into either couldn't display the 'External Monitor Jobs' or the the view wasn't what we were looking for. Wall Display served our purpose very well - Big Green vs Big Red to indicate how our jobs were doing.



No comments: