Tuesday, June 30, 2015

As Jenkins Grows Up, We Invite Our Business Partners To Grow With Us.

As I am writing this post, CloudBees reached a milestone in the number of employees. I think the milestone hit many of us by surprise. “Really,” we thought. “So soon?” But if you look back over the past couple of quarters, it’s pretty apparent that our internal growth was inevitable.  

The number of Jenkins deployments is rapidly rising. At last measure, there are more than 100,000 active installations of Jenkins running. And, as enterprise companies deploy more and more Jenkins, the need for enterprise-grade solutions are accelerating at a very similar rate. A recent blog by CloudBees CEO Sacha Labourey discusses how organizations are transforming their use of Jenkins as a Continuous Integration (CI) tool to using it as a platform to bring enterprise-wide Continuous Delivery (CD). And as our customers have matured their deployments, so have the solutions and offerings from CloudBees, including the most recent launch of CloudBees Jenkins Platform.

The fact is… we are growing. And as we grow, our partners- resellers, services providers, training partners and technology partners- will all play an increasingly critical role delivering the enterprise-scale Jenkins solutions and complimentary tools and platforms our joint customers are seeking.  

Which is why we are committed to equipping our partners with the skills, resources and tools to help you get the most from the opportunity that Jenkins offers. Next month, CloudBees will announce new developments in our Partner Program to meet the needs of our growing partner ecosystem and to help all maximize the vast opportunities Jenkins presents. All current or potential partners- including global resellers, service providers and training partners are invited to attend our informational webinar on July 16 at 11 am ET. This presentation will provide an overview of the latest product developments and expanded opportunities available to partners to help grow your business through enterprise-scale Jenkins solutions.

We look forward to sharing these exciting developments with you next month and working with you to uncover new opportunities, deliver the latest in Jenkins innovations and solutions to our joint customers, and expand your business.


Durga Sammeta
Global Alliances and Channels


Durga is Senior Director of Global Alliances and Channels and is based in San Jose.


Wednesday, June 24, 2015

CloudBees Jenkins Platform: Accelerating CD in Enterprises

If you follow CloudBees and Jenkins, you must’ve heard a flurry of announcements at Jenkins User Conference in Washington DC and London.


This blog summarizes all the new goodies that CloudBees has launched at these conferences.

The Launch of the CloudBees Jenkins Platform

Organizations have matured from using Jenkins to do Continuous Integration to using it as a platform to bring enterprise-wide Continuous Delivery and they have used CloudBees products CloudBees Jenkins Enterprise and CloudBees Jenkins Operations Center to do so.


With the launch of the CloudBees Jenkins Platform, we bundle these in one easy consumable package with couple of editions (Team, Enterprise) serving small teams and enterprise administrators.


Screenshot 2015-06-09 18.41.56.png


Each edition comes with features that are outlined here (refer to the CJP documentation for details)
Screenshot 2015-06-09 18.44.07.png


Screenshot 2015-06-09 18.44.42.png


Welcome to “Solution Packs”

The CloudBees Jenkins Platform allows CloudBees to serve enterprise audiences with specific needs better. We do so through the ability to deliver specific feature sets through “solution packs”. One of the first pack that we are launching today, is the Amazon Solution Pack.


This pack lets customers share “elastic slaves” hosted on AWS with all Jenkins masters managed by CloudBees Jenkins Operation Center within an organization - these masters themselves are running on-premise or in the cloud. In addition, the CloudBees Jenkins Platform lets users directly use Amazon Web Services  within a Jenkins job using AWS CLI. Thus, developers can access any service that is accessible through the CLI as part of their build and deployment jobs.


In addition, we are providing AMIs on the Amazon market place to help Amazon customers easily bootstrap the complete CloudBees Jenkins Platform - i.e. both CloudBees Jenkins Enterprise and CloudBees Jenkins Operations Center - on AWS.


CloudBees Jenkins Platform for Pivotal CF


Last November, we announced a partnership with Pivotal and the release of CloudBees Jenkins Enterprise on Pivotal CF. After a successful launch, our customers quickly came back to us and asked support for CloudBees Jenkins Operation Center as well, so that those organizations could roll out an enterprise-wide Continuous Delivery platform fully hosted on the Pivotal CF platform.


So today, we are proud to announce the extension of this partnership with the release of CloudBees Jenkins Operation Center on Pivotal CF. And since we just announced a new packaging of our offering (see above), in a few weeks, we will be providing the complete CloudBees Jenkins Platform on Pivotal CF and remove individual references to CloudBees Jenkins Enterprise and CloudBees Jenkins Operations Center.


CloudBees Jenkins Platform for Microsoft Azure


Microsoft customers have demanded CloudBees Jenkins on the Azure platform for quite a while - and I am happy to announce that Microsoft and CloudBees have signed a partnership to make Azure a prime location for your Jenkins deployments. Full support will come in several steps.


Today, we are releasing CloudBees Jenkins Operations Center/CloudBees Jenkins Enterprise for Microsoft Azure - these are Azure images that help Microsoft customers be up and running quickly with both CloudBees products. The current images are based on our November 2014 release but we will be updating them in the next few weeks with the May CloudBees Jenkins Platform release.


My crystal ball tells me that there will be a lot of interesting announcements as we take the partnership forward.


New features in the CloudBees Jenkins Platform

At the Jenkins User Conference, we also announced a number of new CloudBees Jenkins Platform features:
  1. Stabilize production masters and eliminate downtime to teams caused by jobs that aren’t stable: The ability to promote jobs from masters used for testing to masters used for production:
    1. Some of the most sophisticated IT departments, use CloudBees Jenkins Operations Center to manage Jenkins and create new jobs on a test master and then job is stable they want to promote it to production. We have made this process easy and seamless. Features include:
      1. Validate if the job will run successfully on the target master before promoting. Examples include checks mentioned in the next bullet.
      2. Implicitly perform validation before promoting aka “pre-flight checks”. These checks include:
        1. Validate that the core versions of Jenkins on the test and production masters are compatible.
        2. Validate that plugins that are used on the test master are available on the production master.
      3. A job is re-promoted perhaps after a few fixes, then the ability to preserve the history of the job on the target master.
  1. Build cross organizational and cross master pipelines: Trigger jobs across masters
    1. This feature helps organizations build CD pipelines that span across masters. Thus, enabling scenarios such as, jobs on Dev teams master tickle jobs on QA teams master. Some of the features are:
      1. Integrated with CloudBees Role-based Access Control: thus jobs can only be triggered by employees with the right permission
      2. Ease of use features such as a quick path browser to easily navigate to the downstream jobs if it is on a different master within a cluster.
  1. Improved UX, especially regarding the getting started experience:
    1. A very common ask is to make Jenkins UI more modern, we have taken first steps to address it in our product. If you have opinions, both positive or negative, we will like to hear about it.

New features in CJP but delivered in OSS

At CloudBees, we wear two hats both: Open Source and Proprietary product :-). So some of the biggest features that we have delivered this semester actually landed in OSS (hence are available both in open source and as part of the CloudBees Jenkins Platform).


Workflow Improvements

At the end of last year, CloudBees with the Jenkins community delivered a substantial new sub-system in Jenkins: Jenkins Workflow. Jenkins Workflow helps build real world pipelines programmatically. We have been busy since since and have released multiple new versions (8 to be specific). Workflow 1.8 brings in notable new features including increased support for third-party plugins such as Copy Artifact, Credentials Binding, etc. They are now all supported as first class citizen as part of a workflow definition. You can refer to the compatibility matrix for up-to-date information.


The following table captures most plugin update changes (Refer to the release notes for details):
.
Plugin Support
Improved Workflow DSL Features
Improved DSL Features
buildStep: get downstream build properties
Load script from SCM: CI as source code
waitUntil: wait for external event before starting
Safe Restart: restart Jenkins if WF isn’t running
Mercurial
Sleep: sleep for sometime before proceeding

Rebuild: rebuild jobs with initial parameters
fileExists: check if a file exists before proceeding

Build Token Root: securely trigger builds
withEnv: attach env variables before proceeding

Parallel Test Executor


Mail: support for mail within a workflow step


Perforce




The most interesting (well at-least to me :-)) is the ability to do CI-as-code with the “load script from SCM” feature. With this feature, developers can check in their build script into the source code repository and point their Jenkins job to the repository and Jenkins uses the script as its job configuration.


The CloudBees and Jenkins community will continue to add support in Workflow for Jenkins plugins - so watch this space.

Continuous Delivery with Docker

I saved the best for the last: a big effort from CloudBees in conjunction with the community has been providing first class support for Docker to build continuous delivery pipelines. I have written a separate blog to call this feature set out.

Parting thoughts:

I am pleased to see the breadth of solutions that we bring to the market today. It isn’t often that a release includes partnerships and solutions offered in the variety of domains as we did today. I am excited that we have pushed the boundaries by enabling modern, sophisticated pipelines with Jenkins and Docker.


What gets me most excited is the potential product and open source opportunities across CloudBees and Jenkins as we go ahead.


I would like to quote Robert Frost on behalf of CloudBees and Jenkins:

The woods are lovely, dark and deep,   
But I have promises to keep,   
And miles to go before I sleep,   
And miles to go before I sleep.


  • Harpreet Singh

Links


  1. The Docker and Jenkins White Paper 
  2. Workflow Compatibility Support




Harpreet Singh
Vice President of Product Management 
CloudBees

Harpreet is the Vice President of Product Management and is based out of San Jose. Follow Harpreet on Twitter

Tuesday, June 23, 2015

Bringing Continuous Delivery to Cloud-Scale with Jenkins, Docker and "Tiger"

At JUC London I attended Bringing Continuous Delivery to Cloud-Scale with Jenkins, Docker and "Tiger" talk by Kohsuke Kwaguchi and Harpreet Singh.

"Continuous Delivery", "Cloud" and "Docker" - all buzzwords in the - this talk premises to be of high interest - or just vapor-ware! - room was packed; Here are my live notes


Kohsuke and Harpreet introduced the "Tiger" project they are working on (one of them asking for more and more features, the other implementing them when he's not doing a talk at some conference - I let you guess who's who).

CloudBees is focussing on Continuous Delivery (further noted "CD" for consistency). They took Tesla car as a sample to illustrate this, as a Tesla car can receive upgrades during the night to fix a technical issue identified on running cars one day before, and let users benefit the latest fixes/features with minimal delay.

To reconcile Dev and Ops tools within a single workflow to embrace all the continuous delivery process, workflow plugin is a key component to offer better flexibility. Docker is another major brick on the lego-puzzle team have to build to address CD challenge. With lightweight isolation it offers better reproducibility. a set of Docker-related plugins have been announced at JUC DC. Combined together, they allow to package the app and resources into container, and orchestrate their usage through the CD pipeline.

  • build and publish docker image (with credentials support for private repositories)
  • listen to dockerhub event so jenkins do trigger a build when some image is updated, to ensure everything is always up-to-date
  • workflow support to make docker images and container first class citizens in workflow DSL.


Kohsuke made a live demo of such an integration. He committed a fix to a demo project, which triggered a jenkins build to publish a fresh new Docker image. DockerHub notification then do trigger the CD workflow to upgrade the production application with this up-to-date Docker image. Docker traceability do record docker image fingerprints so we can check which Docker image was used and which jenkins build did created it.

Other demonstrated use-case is about managing build environment with Docker images. Docker plugin let you use docker containers as jenkins slaves. Docker Custom build environment let you control this directly from job configuration, or as a Dockerfile committed to your SCM side-by-side with project source code.

Docker definitively is a major component in Jenkins way to address the CD challenge. CloudBees is also working on addressing large scale installations with support for Docker-shared execution within CloudBees Jenkins Operation Center. Harpreet also announced plan to deliver Kubernetes support on next release. Operation Center is evolving to embrace multi-master installation, with "promotion" for jobs to get moved from one master to another, cross master triggers, and such multi-master interactions.

CloudBees product line is evolving into CloudBees platform : Team Edition for small team, Enterprise edition for larger installations, with "packs" for specific set of additional features (Amazon support for sample), and a fresh new "Tiger" project - here we go - aka Jenkins-as-a-Service, dedicated to big companies.


DEV@Cloud already do offer such a service with thousands jenkins masters hosted on Amazon and elastic build slave infrastructure. Tiger goal is to offer the same experience behind company firewall. Multi-tenanted masters and slaves provisioned on-demand without administration hell, is built on top of CloudBees platform so do benefit all the tooling provided by CloudBees Jenkins Enterprise (security, monitoring, visualization). 

Kohsuke made a quick demo of this new product. From CloudBees Jenkins Operation Center web UI he provisioned a fresh new client master. Tiger is managing the underlying infrastructure - based on Mesos and Docker containers - to find adequate "box" to host this new instance and storage bucket, and is sharing build resources the same way. Within the minute you get a fresh new jenkins master setup, ready to host team jobs and builds. Tiger is moving jenkins to the Cloud-scale with such a multi-tenant distributed solution.

So, Docker again. Seems this is not about the Tiger I expected but actually some Tiger Whale...



Jenkins / Docker / Continuous Delivery story is just starting, and lot's more feature and tools integration will come to offer simpler/better/faster (Daft Punk TM) Continuous Delivery.



Monday, June 22, 2015

Templating Jenkins Build Environments with Docker Containers


Builds often require that credentials or tooling be available to the slave node which runs it. For a small installation with few specialized jobs, this may be manageable using generic slaves, but when these requirements are multiplied by the thousands of jobs that many organizations running per day, managing and standardizing these slave environments becomes more challenging.

What is Docker?

Docker is an open-source project that provides a platform for building and shipping applications using containers. This platform enables developers to easily create standardized environments that ensure that a testing environment is the same as the production environment, as well as providing a lightweight solution for virtualizing applications.


Docker containers are lightweight runtime environments that consist of an application and its dependencies. These containers run “on the metal” of a machine, allowing them to avoid the 1-5% of CPU overhead and 5-10% of memory overhead associated with traditional virtualization technologies. They can also be created from a read-only template called a Docker image.  

Docker images can be created from an environment definition called a Dockerfile or from a running Docker container which has been committed as an image. Once a Docker image exists, it can be pushed to a registry like Docker Hub and a container can be created from that image, creating a runtime environment with a guaranteed set of tools and applications installed to it. Similarly, containers can be committed to images which are then committed to Docker Hub.

Docker for bootstrapping and templating slaves
Docker has established itself as a popular and convenient way to bootstrap isolated and reproducible environments, which enables Docker containers to be the most maintainable slave environments. Docker containers’ tooling and other configurations can be version controlled in an environment definition called a Dockerfile, and Dockerfiles allows multiple identical containers can be created quickly using this definition or for more customized off-shoots to be created by using that Dockerfile's image as a base.

The CloudBees Custom Builds Environment Plugin allows Docker images and files to serve as template for Jenkins slaves, reducing the administrative overhead of a slave installation to only updating a few lines in a handful of environment definitions for potentially thousands of slaves.

Building with Docker Containers
This plugin adds the option “Build inside a Docker container” in the build environment configuration of a job. To enable it, simply scroll to the “Build Environment” section of any Jenkins job and select the “Build inside a Docker container” option. You will then be able to specify whether a slave container should be created from a Dockerfile checked into the workspace (e.g. the file was in the root of the project) or whether to pull an explicit image from a Docker registry to use as the slave container.















Customized slave environments
For generic builds, you can leverage the most popular Jenkins slave image in Docker Hub called evarga/jenkins-slave or create a new image with a custom Dockerfile for any specialized builds that requires some build dependencies which should need to be available in the workspace, such as credentials.

To create a custom environment, you will need to create your own Docker slave image. This can be done by creating a new Dockerfile or running an existing slave image such as “evarga/jenkins-slave”, then installing the necessary custom tooling or credentials and committing your changes to a new image.

To create a new image from a Dockerfile, you can simply edit the below copy of the “evarga/Jenkins-slave” file using the Dockerfile guidelines and reference:

FROM ubuntu:trusty
MAINTAINER Ervin Varga <ervin.varga@gmail.com>
RUN apt-get update
RUN apt-get -y upgrade
RUN apt-get install -y openssh-server
RUN sed -i 's|session    required     pam_loginuid.so|session    optional     pam_loginuid.so|g' /etc/pam.d/sshd
RUN mkdir -p /var/run/sshd
RUN apt-get install -y openjdk-7-jdk
RUN adduser --quiet jenkins
RUN echo "jenkins:jenkins" | chpasswd
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]


Builds which are built within a Docker container will be identifiable by the Docker icon displayed inline within a job’s build history.










Where do I start?
  1. The CloudBees Docker Custom Build Environment Plugin is an open-source plugin, so it is available for download from the open-source update center or packaged as part of the CloudBees Jenkins Platform.
  2. Other plugins complement and enhance the pipelines possible with this plugin. Read more about their uses cases in these blogs:


  1. More information can be found in the newly released Jenkins Cookbook




Tracy Kennedy
Associate Product Manager
CloudBees 

Tracy Kennedy is an associate product manager for CloudBees and is based in Richmond. Read more about Tracy in her Meet the Bees blog post and follow her on Twitter.