Wednesday, November 30, 2011

Polling Must Die: Triggering Jenkins Builds from a Git Hook

As I keep saying, polling a repository from Jenkins is inefficient; it adds delay on the order of minutes before a build starts after a commit is pushed, and it adds additional loads. It is much better instead to do push-notification from the repository. In this post, I'm going to explain how to do this for Git, which brings this on par with Subversion.

The previous best practice of doing this is best summarized in this blog post. While it works, it is less than ideal. A part of the problem is that this requires hard coding of job names inside the repository hooks, making it hard to keep them up-to-date. Another problem is that if your job only cares about one branch in a busy repository, you don't want a new build to be triggered. Finally, the last problem is that you need some extra work for secured Jenkins.

With the latest Git plugin 1.1.14 (that I just released now), you can do this more easily by simply executing the following command:
curl http://yourserver/jenkins/git/notifyCommit?url=<URL of the Git repository>

This will scan all the jobs that's configured to check out the specified URL, and if they are also configured with polling, it'll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered, in turn). This allows a script to remain the same when jobs come and go in Jenkins. Or if you have multiple repositories under a single repository host application (such as Gitosis), you can share a single post-receive hook script with all the repositories. Finally, this URL doesn't require authentication even for secured Jenkins, because the server doesn't directly use anything that the client is sending. It runs polling to verify that there is a change before it actually starts a build.

One more final note — in Git, unlike Subversion, a repository does not have its own identity, and a single repository sometimes has multiple URLs to access it. In such a case, simply execute the curl commands multiple times with all the different URLs.

That's it. I hope this will help reduce the use of polling and have more people switch to push notifications. It really is addictive to see the build start in a split second after you push a change.

-- Kohsuke

Follow CloudBees:
Facebook Twitter

Releasing Jenkins Enterprise by CloudBees

CloudBees has announced the availability of Jenkins Enterprise by CloudBees version 11.11 (November release, 2011). CloudBees has aligned itself with the Jenkins Long Term Support provided by the community. For a free evaluation copy, download here.

With the introduction of Jenkins Long Term Support (LTS) release, the Jenkins community has committed itself to providing a stable Jenkins release roughly every 3 months. The community moves to a new branch after 3 months. The community model is described here. However, quite a number of organizations stay on a stable release for more than 3 months and desire that they are not blocked waiting for community resources to address the issues that impact them.

Organizations can address both issues by moving to Jenkins Enterprise by CloudBees: CloudBees provides support for issues for 9 additional months on a LTS release and customers have access to support resources that can fix issues for them. The details of the support model is described here.

In addition to the support, Jenkins Enterprise gets you access to all CloudBees proprietary plugins. You can read more information about the plugins here. You can also run all the plugins on Jenkins trunk code-base.

Note: The current release is sync-ed with current Jenkins LTS Release Candidate (RC) release. Once the community releases the GA of LTS release, CloudBees will provide an update that syncs with the GA release.

For existing customers who are using Nectar: Nectar 11.10 was re-architected to do away the CloudBees proprietary branch and moved to Jenkins LTS. Existing customers are requested to upgrade to Jenkins Enterprise 11.11. If you do not wish to upgrade, we will continue with support on Nectar 11.04; however Nectar 11.10 is effectively now subsumed by Jenkins Enterprise by CloudBees. All plugins will continue to work as is. With this release, Nectar as a product has been superseded by Jenkins Enterprise by CloudBees

More information: support model; release notes; user guides found on the CloudBees wiki site.

Download: Existing customers and folks interested in trying out can both use the same link. Note: The war file is called jenkins.war.

- Harpreet Singh, Senior Director of Product Management

Follow CloudBees:
Facebook Twitter

Tuesday, November 29, 2011

Continuous Delivery with Grails and CloudBees

Following our previous blog post on Clojure, you may wonder if your favorite alternative web stack is supported on CloudBees Platform as a Service (PaaS).

The short answer is yes, as long as it runs on the JVM and can be packaged as a WAR archive. To be more precise, each JVM-based language has its own tools and frameworks, that can nicely integrate with CloudBees as long as the adequate plugin is provided.

Taking Groovy and Grails as a sample, DEV@Cloud provides the grails Jenkins plugin and runtimes to build and test applications, and the community contributed grails-cloud-bees plugin can manage deployment to RUN@Cloud.

Skillsmatter is organizing the Grails eXchange conference in London next week, and we will demonstrate this combination in collaboration with Grails plugin author, Marco Vermeulen. Meet us there!

Follow CloudBees:
Facebook Twitter

Monday, November 21, 2011

Easy Deployment of Clojure Apps on CloudBees

Clojure, the popular lisp dialect for the JVM, recently got much easier to deploy on cloudbees:

You will probably have (or want to have) the Leiningen build system installed, once you have this, if you are new to clojure (or curious) then a great place to start is the bootstrap web app (clojure apps use the "ring" middleware for web apps - which is what this is all built on).

Clone it - run lein deps and away you go ! Develop locally (run lein ring server) - and deploy when ready (lein cloudbees deploy) - no SDK install needed.

This magic is all brought to you via the CloudBees lein plugin.

Follow CloudBees:
Facebook Twitter

Friday, November 18, 2011

SendGrid is Now Part of the CloudBees Ecosystem

This blog is a guest post from Brandon West, Developer Relations Engineer at SendGrid…

SendGrid is happy to announce that we've partnered with CloudBees, a Platform as a Service (PaaS) provider that offers cloud-based services for building, testing and deploying Java web applications. As of today, SendGrid's email service is now part of the CloudBees Ecosystem! CloudBees and SendGrid are both focused on reducing the friction that developers have to deal with when it comes to configuring and building out infrastructure so that they can spend their time building great things.

For those of you who aren't familiar with SendGrid, our service makes it simple to send email from any web application. SendGrid's cloud-based SMTP email infrastructure relieves developers of the cost and complexity of maintaining custom in-house email infrastructures. We provide reliable email delivery, scalability, and real-time analytics along with flexible APIs that make custom integration a breeze. Our service alleviates the email headaches for developers, allowing them to focus on making great apps.

If you've got a CloudBees account, you can now enable SendGrid from the CloudBees Ecosystem page with one simple click. SendGrid is a great choice for CloudBees customers looking to add a robust cloud-based SMTP email solution to their applications, and our partnership will make it easier than ever to do just that. If you're a working on a Java application and are not already using CloudBees, we encourage you to sign up for a free account and give it a try. If you build an app with SendGrid and CloudBees and want to share it with us, we may feature it in a future blog post. We’re excited to see what people build!

Follow CloudBees:
Facebook Twitter

Wednesday, November 16, 2011

Monitoring Jenkins in Eclipse

A few months ago, we released the CloudBees Toolkit for Eclipse. The Toolkit made it's way into the Eclipse Marketplace yesterday. Eclipse users can find the toolkit through the Eclipse Market client and install it from within the IDE.

One feature that we should have highlighted more than we do is the capability of the toolkit to monitor jobs in on-premise Jenkins (or for that matter any Jenkins whose URL is accessible) from within the IDE, of course. Folks who are interested in just on-premise Jenkins can use it for free (no registration required as well). Folks who wish to move to Jenkins in the cloud will have to register with CloudBees services and then monitor, deploy from within the IDE.

Developers can monitor the disasters they are wreaking on their builds from within the IDE. Fix the disasters knowing that they caught it before management could see it and then mingle around with pointy-hair nonchalance. "Broken build? What broken build? I fixed that ages ago." ;-)

- Harpreet Singh, Senior Director of Product Management

Follow CloudBees:
Facebook Twitter

Tuesday, November 15, 2011

The CI Train Has Left the Station

Photo licensed under Creative Commons by mybulldog

Red Hat's announcement today of OpenShift's support of Jenkins was inevitable and very welcome here at CloudBees. Of course, Jenkins continues to be the strongest CI offering and a very successful open source project, and RedHat is an active, supportive member of the community.

When RedHat gets OpenShift into production and figures out the business model for it, CloudBees will have some competition in a PaaS marketplace that has already validated the role of CI and cloud application lifecycle management. And you shouldn't expect RedHat to be the last of the incumbent packaged middleware vendors to figure out the value that continuous integration and continuous delivery bring to the cloud. That's because the developers and companies who are harnessing the cloud are learning that continuous integration and delivery are going to be an incredibly important part of their competitive position as they take advantage of PaaS and the flexibility of IaaS to deliver their own software as a service.

The cloud is the future of middleware, PaaS gives Java developers the platform to take advantage of it, and companies who miss the train will be waiting at the station for the old Client-Server Local to pull in. Welcome to the future, Red Hat!

For those of you wanting a ticket on the express train now, you can join thousands of others using Jenkins as a Service on the CloudBees Platform, in production today with automated production deployment on your choice of Java runtimes -- all brought to you by the Jenkins experts.
-- Steve Harris, Senior VP of Products

Follow CloudBees:
Facebook Twitter

Introducing Template Plugin

The Nectar 11.10 release that we just made has the template plugin, which I think is one of the very useful plugins that we added to Nectar that's not available in open-source Jenkins. So I wanted to talk about it a bit.

Many serious Jenkins users have had this common problem that they've got a lot of jobs that look similar. JENKINS-3157, which is currently the most voted issue in JIRA, speaks to that same pain point. The template plugin is my answer to this problem.

What this plugin does, is to allow you to define a "model." A model is an abstract concept that fits your particular problem domain. You do this through defining a set of attributes, kind of like how you define a class with a set of properties in Java, or how you define a table with a set of columns in database. You then define a transformation, which converts this model into something that Jenkins understands.

Creating a template

This is all rather abstract, so let's look at a concrete example. We are going to define a hello world builder template, which lets you enforce the standard way to greet someone. You can then use this template to greet a lot of people in many jobs.

Installing this plugin, you get the "templates" menu on the left. You can click that and create a template:

You create a builder template from the same faimilar wizard you use to create a job:

Then here is the main meat:

There are two main things to configure here: attributes and a transformer.

When you define a template, you first ask yourself, “What do I want my users to enter when they use my template?” The answer to that question becomes attributes. In this hello world builder, we want the user to configure who we are saying hello to, so we define one attribute named “target” for this purpose. The user should see the single text field for this, so we choose the type accordingly. The display name and inline help are self-explanatory. They control what the user will see when they are editing their free-style projects to use our new builder.

The second question you should ask when you define a template is, “How does it execute?” (or more generally, “How does it map to the terms Jenkins understands?”) The answer to that question becomes the transformer. In this tutorial, our builder will turn into a shell script that says hello (whereas your real template would probably turn into a shell script that gets some real work done, like building a component, running a test, etc.) So we’ll choose, “Generate a shell script to execute via Groovy.”

In the text area, we’ll fill the shell script that this build step is going to execute, but with expressions here and there of the form ${...} (because this is a Groovy template). ${target} refers to the actual value of the target attribute we defined above, and ${build.fullDisplayName} is a Groovy expression to access the getFullDisplayName() method (which returns the full name of the build) of the build object, which refers to the current build in progress. The ${build.fullDisplayName} needs to be quoted because this is going to look like test #1, and # gets interpreted by shell as a comment sign unless you quote it.

Using a template

Now that we have our first builder template, let’s create a free-style project that actually uses it. Go back to the Jenkins top page, and create a new free-style project. You’ll be taken to the configuration page. This part of Jenkins should already be familiar to you.

When you click “Add build step”, you should see the newly created “Say Hello World” builder. You click it, and you see the say Hello World builder added to your project. And you see the “target” attribute you defined in the configuration page.

I configured this to say hello to Kohsuke. If we save it and run it, lo and behold, you'll see that it works:
[workspace] $ /bin/sh -xe /tmp/
+ echo Hello to Kohsuke from test #1
Hello to Kohsuke from test #1
Finished: SUCCESS

This allows you as the build guy to create and enfroce a certain way your stuff gets built/tested, and your users get shielded from all the gory details of how the build/test actually gets done. You can create a layer between Jenkins and your users, and make Jenkins talk in the language your users understand, instead of making them talk in the language Jenkins understands. This is one of the points of the template plugin.

Changing a template

Now, let’s change the definition of the template, and see how it affects the instances.
We’ll go back to the template definition by going back to the top page, clicking the “Templates” link in the left, and clicking the configuration icon on the right to the “Say Hello World” template.
Instead of saying hello, now we make it say good evening. Click “save” to save this new definition:

Now, when you update the template definition, all the use of this template automatically reflects the change you made. So without revisiting the configuration page of the freestyle job, let’s simply schedule another build and see its console output:
[workspace] $ /bin/sh -xe /tmp/
+ echo Good evening to Kohsuke from test #2
Good evening to Kohsuke from test #2
Finished: SUCCESS

This is another point of the template plugin — when you change the definition, all the uses of this template get updated right away. So if you need to tweak how those high-level abstract concepts map to the lower level command sequences, you only need to edit it once and save it.

Try it out for yourself

If you like what you've read so far (and I hope you do), please play with this. You can either download the whole Nectar and play with it in a separate sandbox, or you can enable the secondary update center to install these plugins on your OSS Jenkins deployments.

And there's a lot more to the template plugin, which I hope to write about in a few days.
-- Kohsuke

Follow CloudBees:
Facebook Twitter

Sunday, November 13, 2011

Virtualization and Clouds

Virtualization - Who Needs It?

Many would say, and few would disagree, that machine virtualization was a key enabling technology that spawned clouds. Virtualization of machines is actually old. Really old. 1960s old (of course if you were born then you aren't old, just perfect) - and often used when the machine capacities, in some way, exceeded the software capacity to make use of it efficiently (memory, disk, CPU time, etc. - most often CPU time). So you can totally see - without even thinking too much, that the aims of cloud computing are very similar to what virtualization has to offer in terms of making use of the physical machinery.

However, if you look at what are arguably the biggest clouds in use today - and certainly the ones that have been available for some time:
  • Amazon EC2
  • Rackspace
  • Google (App Engine - notably - but even their internal usage)
  • Salesforce ( - and more)
Only one of those actually used virtualization as a core part of its offering (AWS). Rackspace does as well now (cloud servers), but traditionally was managed hosting (hardware - hence the name!). Google and Salesforce did just fine with plain old operating systems and hardware - they simply built a platform on top of it.

You could argue that for infrastructure clouds virtualization is mandatory - but even this would not be true - just very convenient.

The thing is, the higher up you go in a platform the less you care about the layers underneath. For example, a platform as a service (PaaS) doesn't require a VM based infrastructure for it to do its job (you don't care). However, for the implementers of that platform - sure - virtualization may be convenient, but not necessary.

PaaS in particular can turn the lower layers into a commodity - surprisingly, even Cloud Foundry from VMware themselves shows this! There is no dependency on any particular virtualization technology at ALL in it. In fact, it doesn't even require virtualization. It does, however, require some way to provide pools of machines - virtual or otherwise, to host application agents.

Now virtualization is here to stay - for a while at least - and most organisations that have IT are making use of it. However, expect the focus, as always, to drift ever higher in the level of abstraction - PaaS being the next layer up - and this will drive down both the cost and visibility of the virtualization software. Expect also to see it increasingly baked into the OS - it is just one technique available to slide up machines (LXC, linux containers, on Linux eventually could provide a lot of the benefits in a hyper-efficient way, for example).
-- Michael Neale, Elite Developer and Architect

Follow CloudBees:
Facebook Twitter

Wednesday, November 9, 2011

Announcing the Availability of Nectar 11.10 - New Release of CloudBees' Enterprise Version of Jenkins CI

Last week we released the new version of Nectar. The version is called Nectar 11.10. This is a short FAQ for existing Nectar/Jenkins users to let them know what's in the release.

Q. Why should a Jenkins user consider Nectar?
  • As the adoption of Jenkins grows in an organization so does the complexity of managing Jenkins. Nectar plugins helps administrators to work with large number of jobs, projects, team and deal with this increasing complexity.
  • Nectar helps secure your installation (for example, with Role-Based Access and other features)
  • Nectar helps to improve optimization of existing resources through multiple plugins so that organizations can do more with the same resources (often cutting down on costly hardware purchases)
  • Nectar is released on a 6 month cadence with numerous patch releases between releases so it offers a "throat to choke" for organizations that desire ... well a "throat to choke :-)"

Q. What is being released as part of Nectar 11.10?
  • Release has few new plugins that help manage large numbers of jobs (through the Templates plugin) and improve resource utilization (through Even Load Strategy plugin and Skip Next Build plugin)
  • Release helps optimize resource utilization with the enhancements made to the VMWare Pool Auto-scaling plugin and better manage backups with additional capabilities to the Backup plugin
  • Release introduces the capability to run Nectar plugins on Jenkins for existing customers

Q. What are the specifics of the plugins (new or updated)?
  • Skip Next Build Plugin
    • This plugin allows users to skip building a project for a short period of time. This is useful in cases where build failures are expected such as some external resources are unavailable for a certain period of time or important merges in process. By using this plugin, users can suppress the false failures. The build automatically turns on after the elapsed time period.
  • Templates Plugin
    • The plugin captures the “sameness” of configuration in multiple places. Administrators can define templates of jobs/build steps and replicate them while creating new jobs. Changes are made in one central location and are reflected in all dependent configurations.
  • Even Load Strategy Plugin
    • This plugin changes the default job-allocation algorithm of Jenkins to allocate jobs to idle machines instead of the “preferred” ones as today
  • VMWare Pool Auto-scaling Plugin
    • This plugin allows you to create slave machines that are VMWare ESXi/vCenter VMs. This offers better resource utilization of your machines. VMWare plugin now allows you to use folders in vCenter to simplify managing large number of VMs. You only have to point to a vapp folder and all machines are automatically picked up – allowing users to dynamically add/remove machines without changing configurations.
  • Backup Plugin
    • The backup plugin allows you to create Jenkins jobs that regularly back up part or the entire Jenkins configuration. This plugin has been updated to automatically manage the number of backups and throw away throw away old unneeded backups based on your configuration. One of the policy supported is the "exponential decay" policy, which keeps larger number of newer backups and fewer number of old ones.

Q. Where can I learn more? 

Q.  I am an existing customers, how can I upgrade?
  • Nectar upgrades are just like Jenkins upgrade. You can read the steps for the upgrade in the release notes.

Q. I am an existing Jenkins user, can I try this out?
You can download and try out the release for 30 days for free.

- Harpreet Singh, Senior Director of Product Management

Follow CloudBees:
Facebook Twitter

Friday, November 4, 2011

Jenkins Community - Report Bugs, Win Big!

CloudBees thinks the Jenkins community is pretty amazing! Because the community has come so far and is doing so many great things, we are committed to supporting the ongoing growth, maturity and viability of the Jenkins Continuous Integration (CI) platform. CloudBees also wants to promote engagement amongst the Jenkins community.
In support of these goals, CloudBees announces two fun contests. The goal of both is to encourage you to report bugs and/or resolutions for the Jenkins CI platform. These two activities improve the platform, which benefits the entire Jenkins Community.
The two contests, announced at the Jenkins User Conference in October, are:
1. Bring Me Some Bugs
2. Bug Bounty

For details about both contests, read more here.
You can win:

Good luck bounty hunting—and watch out, bugs!
Heidi Gilmore, Marketing
Follow CloudBees:
Facebook Twitter