Thursday, October 30, 2014

Monitoring External Scheduled Jobs Using Jenkins

Recently we had a requirement to create a dashboard so we could monitor our scheduled jobs that were important for the business. The jobs ran on different server boxes. Every now and then when we had issues with a any job, we needed to get access to the system and dig the log files. Worst of all was that we would only know about a problem with the job only when someone reported it. We wanted to build a dashboard so were on top of the jobs' status and fix issues ASAP.  Some research from our team suggested that Jenkins could help us with that.

We got an instance of Jenkins running very quickly and started playing around it. Below is what we did to get a dashboard going.

Jenkins as a Monitoring System

Firstly, we created 'Monitor an External Job' item in Jenkins. Let's call it "JobMonitor1". The configuration required for such an item in Jenkins is very minimal. All you need to do is configure how long do you want the builds (job running records) for.

Now that we had a job - we needed to get the actual JOBs that are running in a different environments to let Jenkins know that it has run. This is where the Monitoring External Jobs plugin came very handy. The plugin page shows how to use this plugin. In summary, we installed the plugin in our Jenkins instance and then got our JOB to send feedback to Jenkins. If we currently ran our job in this manner - bash, we updated it to run as follows -

java -jar /path/to/jenkins-core-*.jar "JobMonitor1" bash

We needed to make sure some of the required jar files were available in the classpath. To let the system know which Jenkins instance to post the results to, we needed to set JENKINS_HOME in the system by running the command below.

EXPORT JENKINS_HOME= /path/to/jenkins (for Linux Systems)

SET JENKINS_HOME= /path/to/jenkins (for Windows)

When our script ( completed, it sent the details of the jobs to "JobMonitor1" and we could then monitor it in Jenkins. The downside of this approach was that we could not tell from viewing Jenkins whether the job was running. Jenkins is only notified after the process completes. We do however know how long it took for the job to complete.

Monitoring the Monitor

Now that we had the visibility of whether the last ran, passed or failed, to make it a bit more real-time, we needed to know whether the JOB ran when it was expected to run. For example, if a job is expected to run every hour, did it run every hour?

This is where it gets little dirty. We decided to create another item (e.g. "SecondLevelMonitorJob") in Jenkins that would monitor the  Jobs that  were getting notified by the remote system. So we created new 'Freestyle Project' item in Jenkins that would monitor the other job. But how exactly do you monitor it? This is where we found another plugin - check_jenkins_cron. This is simply a script that runs to see if a job that you specified ran in the time you specified.

We wrote a wrapper around this plugin to make the build fail or pass depending on what the plugin returned. So the "SecondLevelMonitorJob" would run our wrapper script. We specified in this JOB how often we expected the job to run. We made  "SecondLevelMonitorJob" run every 15 minutes to monitor a job that is expected to run every 6 hour. This way we picked up the fact that something is wrong with the job ASAP.


And the final piece - the Dashboard. After a lot of investigation on appropriate "Views" for Jenkins, we settled for Wall Display plugin. Some of the others we looked into either couldn't display the 'External Monitor Jobs' or the the view wasn't what we were looking for. Wall Display served our purpose very well - Big Green vs Big Red to indicate how our jobs were doing.

Tuesday, October 21, 2014

Testing Old/Legacy Code using Powermock

I am sharing some of my finding/learning during my attempts to write tests for some codes that were not quite designed for writing tests

Breaking the dependency

You will probably find plenty of dependencies that will be very difficult to satisfy from your unit test which can be very demotivating to write tests. This link has a good description on how to break such dependencies in legacy code. Is shows how to separate the DB connection layer from the service to help making the class more testable. One approach I have taken when you have codes like this  ServiceFactory.getService("MyService"), in the method you are testing is to pass in MyService to the class file via its constructor so that you can easily pass in a mocked MyService.

Static method calls

Static method calls that are in the method that you are trying to test can be very annoying. Use PowerMock's mockStatic method to mock such calls. This link has a good explanation of how to do this. In summary, declare the class with the static method at the class level with PrepareForTest annotation e.g. @PrepareForTest({JOptionPane.class, MyFactory.class}) and do the rest in your test method. An example is below -

        JOptionPane.showMessageDialog((Component) anyObject(), (String) anyObject(), (String) anyObject(), anyInt());

Capturing arguments

 At times capturing the argument and verifying the content is the only thing you can test and you can do this the following way.

        Capture capturedArgument = new Capture();
        mySerivce.updateMyListIdDb(and(capture(capturedArgument), isA(List.class)));

        //test method call
        assertEquals(capturedArgument.getValue().size(), 1);
        MyListClass listClass = (MyListClass) capturedArgument.getValue().get(0);
        assertEquals(listClass.getValue(), "myExpectedValue");

A cleaner way to do this with latest easy mock is as below:
        Capture<List> capturedArgument = new Capture<List>();

        //test method call
        List myList = capturedArgument.getValue();


I ran into this error when a static method was invoked on a class which in turn has a static field  which needed to be loaded first. I am not exactly sure why this would happen.

java.lang.VerifyError: (class: javax/swing/plaf/metal/MetalLookAndFeel, method: getLayoutStyle signature: ()Ljavax/swing/LayoutStyle;) Wrong return type in function
    at javax.swing.UIManager.setLookAndFeel(
    at javax.swing.UIManager.initializeDefaultLAF(

Decorate your test class with @PowerMockIgnore("javax.swing.*")

This defers the loading of class or classes provided in the annotation to the systems class loader. If there are multiple classess you would like to defer then try this - @PowerMockIgnore({"javax.swing.JCheckBox", "*"})