Wednesday, November 30, 2011

Polling Must Die: Triggering Jenkins Builds from a Git Hook


As I keep saying, polling a repository from Jenkins is inefficient; it adds delay on the order of minutes before a build starts after a commit is pushed, and it adds additional loads. It is much better instead to do push-notification from the repository. In this post, I'm going to explain how to do this for Git, which brings this on par with Subversion.

The previous best practice of doing this is best summarized in this blog post. While it works, it is less than ideal. A part of the problem is that this requires hard coding of job names inside the repository hooks, making it hard to keep them up-to-date. Another problem is that if your job only cares about one branch in a busy repository, you don't want a new build to be triggered. Finally, the last problem is that you need some extra work for secured Jenkins.

With the latest Git plugin 1.1.14 (that I just released now), you can do this more easily by simply executing the following command:
curl http://yourserver/jenkins/git/notifyCommit?url=<URL of the Git repository>

This will scan all the jobs that's configured to check out the specified URL, and if they are also configured with polling, it'll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered, in turn). This allows a script to remain the same when jobs come and go in Jenkins. Or if you have multiple repositories under a single repository host application (such as Gitosis), you can share a single post-receive hook script with all the repositories. Finally, this URL doesn't require authentication even for secured Jenkins, because the server doesn't directly use anything that the client is sending. It runs polling to verify that there is a change before it actually starts a build.

One more final note — in Git, unlike Subversion, a repository does not have its own identity, and a single repository sometimes has multiple URLs to access it. In such a case, simply execute the curl commands multiple times with all the different URLs.

That's it. I hope this will help reduce the use of polling and have more people switch to push notifications. It really is addictive to see the build start in a split second after you push a change.

-- Kohsuke

CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

6 comments:

  1. I forgot to put the credit where it is due. Much of this work has been done by Paul Sowden, and I just made a few brush-up changes.

    ReplyDelete
  2. Any chance of this making its way to the Mercurial plugin? We're an Hg shop and I've been dying to get rid of polling for a long while now. Unfortunately we have the same problem with having to hardcode the job names in the repository hook.

    ReplyDelete
  3. Kamil,

    It's actually not that hard to do this for Mercurial. The entire code is captured in https://github.com/jenkinsci/git-plugin/blob/master/src/main/java/hudson/plugins/git/GitStatus.java and you can easily port this to the Mercurial plugin.

    ReplyDelete
  4. Thanks Kohsuke, I'll give it a try. I haven't programmed Java in quite some time, but I don't think it should be too hard to figure it out. I'll have to get the toolchain up and running and see how it goes..

    ReplyDelete
  5. Cool! Let us know if you need any help. See https://wiki.jenkins-ci.org/display/JENKINS/Plugin+tutorial for how to get the toolchain prepared.

    ReplyDelete
  6. “a repository does not have its own identity”—you could use the SHA-1 ID of the initial commit (or one of the initial commits if the rare case that the repo has multiple roots). Thus /git/notifyCommit?id=deadbeef123456 or whatever and Jenkins could look up repositories using any URL which contain that root commit. (Might need some caching in case a Git-based job has no workspace: would suffice to set a property on the job or store the ID in a file in the job’s root directory.)

    The same would apply to Mercurial of course.

    ReplyDelete