Monday, December 26, 2011

Jenkins Now Acts as an SSH Daemon

Starting in the upcoming Jenkins 1.446, Jenkins speaks the server-side of the SSH protocol. I think this is exciting, and here is why.

For quite some time now, there is Jenkins CLI, which lets you access Jenkins from the command line by using a custom client JAR file. There are several dozen commands available (that you can check by visiting http://server/jenkins/cli), ranging from creating a job to shutting down Jenkins. These CLIs are often a lot more scriptable than HTTP interfaces, and so they are often used by other programs to interact with Jenkins.

Jenkins 1.446 exposes a large subset of these commands through SSH, making it even easier to access this functionality. You no longer need any special client JAR nor Java VM to talk to Jenkins. All you need is a stock SSH client. As I explained earlier, Jenkins lets you register your SSH public keys, and that is how you authenticate.

As always, the server side of this is extensible. Plugins can define additional commands to be exposed via SSH, which is a great building block for more sophisticated integrations. This is particularly so because SSH is a standard transport protocol for many tools, such as Git, Subversion and so on. I'm planning to write a few plugins that take advantage of this in the coming days.

This Wiki article talks more about the details, including how you can discover the endpoint, how you can verify the server key, and so on.

(And no, this is not about making Jenkins a general-purpose SSH server that you can use to login to the server!)
--Kohsuke Kawaguchi, Jenkins CI Founder, Elite Developer at CloudBees


CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Friday, December 23, 2011

DEV@cloud Meets GitHub Janky


GitHub announced Jenkins-based continuous integration using Janky and Hubot with Campfire chat integration. Basically, it means from the Campfire chatroom we can setup a Jenkins job, setup service hooks on GitHub to trigger builds, get build info and so forth.

It sounded pretty compelling so I thought I would give it a try. The setup requires running Janky and Hubot web applications, and to wire them with Campfire chatroom and Jenkins server.

Here is how I went about setting it all up to trigger Jenkins job management from chat room.

Jenkins
I got a Jenkins instance provisioned with few mouse clicks at CloudBees DEV@cloud. You can run your own Jenkins server and also then do all the administration and management, but it's much easier and cheaper to have someone else do it for you.

CloudBees offers the DEV@cloud platform which includes Jenkins in the Cloud, and unlimited Git and SVN repositories with all PaaS benefits (backups, high availability, secured, etc.). Checkout our new reduced pricing and choose the plan that fits your needs. You can start with a free subscription or choose the plan that fits your needs. Finally, if you have an Open Source project, you can join our FOSS program.

Not only are there obvious cost and PaaS-centric benefits for developers, rather than hosting Jenkins on their own, but CloudBees DEV@cloud comes ready to work with Janky and already has GitHub integration. Thus, it was an obvious choice for me.

Janky needs a Notification plugin to receive build related notifications. I installed the Notification plugin on the Jenkins Plugin management console.


Janky
Janky is a web application that talks to Jenkins, GitHub, Hubot and Campfire chat room. It's a Ruby/Sinatra application that serves the commands that are entered in the chat room and relayed via Hubot chat bot. It supports various commands meant to manage the builds of your GitHub repositories by communicating with Jenkins and GitHub. I followed Janky GitHub page to set it up. You can host it anywhere but make sure it accepts incoming connections. For our RUN@cloud platform, we are working on adding procfile support which will allow running Janky or Hubot or other non-Java languages based apps. Stay tuned, as it's being cooked!

Setting Janky was not that hard. The most critical part of this process was setting up about a dozen environment variables, most of these were credentials of GitHub, Hubot, Campfire chat and URL for Jenkins. Jenkins running on CloudBees DEV@cloud is protected, so I set JANKY_BUILDER_DEFAULT to the URL with basic auth. Please note the URL encoding applied to the basic auth username:password value.



Once Janky was setup, accessing it in browser shows the page below

Hubot
Hubot is a chat bot developed by Github. It has built in support for Campfire chat. There are other adapters for various chat and IM systems, for example, IRC, GTalk etc. It's quite extensible, you can drop your own script to add additional commands that this bot can respond to. To add support for Jenkins commands, I needed to add janky.coffee script. I also had to set the environment variable to point to Janky endpoint URL with shared basic auth. This Janky endpoint listens to communication from Hubot.

Note about configuring Campfire chat: I had to set campfire env variable for Janky and Hubot with the chat room name and id, respectively.

While testing, I noticed Janky did not support talking to Jenkins over HTTPS. I fixed it in my Janky fork. My pull request has been accepted by the Janky project owner and it is now merged to the trunk. The next version of Janky gem should have this fix. In the meantime, you can either use my fork or Janky trunk which has these fixes merged.

Show Time
Once I had Hubot and Janky configured, I logged into my Campfire chat room as a guest and noticed Hubot present in the chat room as Vivek P, since the Campfire account was set up with this name, and Hubot chat bot was set up with the same account. Real Vivek is me. Once in the chat room, I typed a command to set up Jenkins jobs for my GitHub repositories.



Almost instantly, there were jobs created on my Jenkins instance running over DEV@cloud. The job names were not what I expected, they look like some kind of unique ID, maybe a commit ID, but I don't think they are.



On checking the Jenkins job configuration, the notification plugin configuration had an entry with Janky endpoint where it can post build notifications.


The Job was configured with the default build script that goes in the Janky configuration config/jobs/default.xml.erb,



To customize the build script, update default.xml.erb with your custom script and run hubot ci setup in the Campfire chatroom and the corresponding Jenkins job will be updated with the new script.

To sum it up, Janky is a great idea that opens up the social aspects of managing Jenkins jobs along with GitHub repos! DEV@cloud works seamlessly with Janky and provides many other benefits.

DEV@cloud is already GitHub friendly. Read how you can establish post-commit-hook to trigger build on commits to your GitHub repositories or how you can setup continuous deployment of your GitHub repo on RUN@cloud platform using DEV@cloud Jenkins in the cloud.

If you don't already have a CloudBees account, sign up now. You can subscribe to free DEV@cloud and RUN@cloud services to provision your instance of Jenkins, MySQL DB and deploy apps -- all automatically providing the benefits that come with the cloud and with the additional value-add of deep GitHub integration. And of course use Janky to manage them from a chat room!

-- Vivek Pandey
Elite Developer, CloudBees

CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Thursday, December 22, 2011

Jenkins Community Survey Results: 82% Consider Jenkins Mission Critical

The results from the first Jenkins Community Survey are in! 624 people took the time to share how they use Jenkins and tell us what capabilities matter most to them.

Highlights:
  • 82% of respondents said Jenkins is mission critical to their development process
  • 71% use Jenkins as their only CI tool
  • Jenkins is useful for companies of any size, but 76% work for organizations with > 10 developers
  • 80% of respondents have 3 or fewer masters and 10 or fewer build agents
  • 65% of companies have 10 or fewer executors
  • 76% have between 2 and 50 projects
  • Additional reporting capabilities and high availability were the highest-prioritized features

Check out the full survey results here (minimal registration required). You’ll learn which features users thought were most important to develop, which factors are most influential in making Jenkins easier to adopt, and many other insights into a typical Jenkins installation.

Many, many thanks to everyone who took the survey. You have been instrumental in helping the Jenkins community focus on the most needed features! And congratulations to Chris Dolan, who won the drawing for the AppleTV.

CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Wednesday, December 21, 2011

Hello Java, Hello iPad, Hello World!


A CloudBees New Year’s Challenge

Experience the joy of Platform as a Service as we ring in the New Year!  Publish an sample “Hello Java in the Cloud” application on the CloudBees PaaS by January 16th, and we’ll enter you to win an iPad 2.  We even give you the example code – all you have to do is roll it live, which only takes minutes. And maybe spice it up a bit (you can add your own text to the photo above)!

Bonus for Charity
For every app created and tweeted under the #CloudBeesJava hashtag, we’ll donate $5 to the International Committee of the Red Cross!

Bonus X2
For everyone who ReTweets your app link with the hashtag #CloudBeesJava, we’ll donate an additional $1!

Bonus X3
Any app that is published and tweeted is eligible, but if you make your app fun, interesting, and/or festive, we’ll triple-enter your name to win the iPad.

How It Works
  1. Publish an app on CloudBees. Get started now!
  2. Using the handy button, tweet the app’s URL using the hashtag, #CloudBeesJava. Feel free to post on your Facebook page, blog, etc.
  3. We’ll monitor the hashtag and enter each app to win the iPad. Each RT will garner $1 more to charity (up to $1000 for RTs, but there's no donation limit for apps deployed). If you modify the sample app to make something cool, we’ll enter you three times. No worries if you write an app but don’t have Twitter – just email me the URL and I’ll get your name in the pot (lisa AT cloudbees.com).
  4. Win! You could win an iPad 2. The International Committee of the Red Cross wins more resources to help the world. And everyone wins the satisfaction of successfully launching a Java an app in the cloud! Read the contest details here.
Give it a try!

PS - Meme alert: If "I can haz PaaS" doesn't make any sense to you, check out I can has cheezburger and LOLcats...


CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Tuesday, December 20, 2011

Upcoming Jenkins Training in San Francisco & Tokyo



CloudBees will be hosting another training in the San Francisco Bay Area on January 26, 2012, and February 23rd in Tokyo.

This 1-day training starts with the basics and then covers more advanced techniques, especially in combination with some plugins. As we deliver this training more, we gradually adjust the material to cover more advanced topics, like promotion, build pipeline, parameterized triggers, and so on — the kind of functionality that's getting more traction lately as Jenkins starts to encompass continuous delivery.

And for me, this training will be the first to use the latest "install plugins without restart" feature, so that should help with attendees keeping the focus a bit.

In these trainings, we cap the number of attendees to a fairly small size so that I can pay attention to each attendee enough. People normally bring in questions that go beyond the training curriculum to discuss the problems they face in their day-to-day Jenkins administrations. I welcome those, too!

If you are interested, please sign up while seats are still available!

- Kohsuke

CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Monday, December 19, 2011

I've Got 99 Problems but Scala Ain't One

If you're havin' programming problems i feel bad for you son
I got 99 problems but scala ain't one - Jay Z

OK so maybe Jay Z didn't quite say the above, but there has been some hand wringing lately about the complexity of Scala - just as Scala is showing signs of becoming quite popular.

You can see some of this here, and also some great blogs addressing this bought up by the commercial stewards of Scala, Typesafe here and here.

I thought I would share something about our use and experiences with Scala at CloudBees. We support Scala in various ways, and have many enthusiastic users of it on our platform - but we also use it quite heavily ourselves.

The 2 main areas we use it:
  • Cloud control, allocation of build workloads and Jenkins masters ("providore")
  • The core admin "cloud controller" for RUN@cloud
So you can see this is pretty core stuff. Every app deployed, every build performed, every new user who joins etc... all pass through a significant amount of non trivial Scala.

In the first area (DEV@cloud) - it was chosen as I (Michael) was quite familiar with it - and time is was of the essence - we iterated quickly in working out good ways of interacting with cloud apis (through jclouds - which worked out of the box). Good automated testing tools (Mockito, Scalatest) proved invaluable - as did a really great command line for "live" testing and modifications. Keeping the code compact also helped (lots of higher order functions working over lists and maps going on).

In the second area - RUN@cloud - in parallel Spike built the cloud administration (controller) in a very similar fashion for very similar reasons (we didn't even know each other at the time) - with great results as well.

In time - others joined in - and have been able to contribute and comprehend the code - without too many issues. Scala may not be to everyones taste but it certainly proved to be no roadblock at all for us so far.

We have avoided (partly by accident, partly by design) too many dependencies on libraries that are written in Scala - avoiding most difficulty in updating versions. We also deploy these particular applications as services (that communicate via HTTP, AMQP etc) - not as libraries to be consumed by other JVM apps: perhaps it is the lucky combination of these two points that explain why we did not feel the "pain" described above.

In any case - we are excited by what Scala offers - libraries like Akka are quite an achievement, the Play! Framework, and more - all made richer by Scala. Typesafe are an exciting company and look like they are building a fantastic stack for the future too.


Hopefully there will be many ways we can help the Scala and wider community address these growing pains - for example ensuring the Play Framework v2 Keeps It Clean - for multiple versions of the JDK. There is even talk of matrix builds in Jenkins for many Scala libraries and versions.

The future seems bright - for Scala as well as all JVM languages.

CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Wednesday, December 14, 2011

How Much Does CI Development in the Cloud Cost? Less Than You Think!

Developing in the cloud is now so cost effective, you will try to justify NOT developing in the cloud! We have some new tools to help you estimate how much it would cost you to develop on the CloudBees PaaS. But first, some background...

To run Jenkins, you need to allocate a machine to be what is called a "Master"; this Master contains your build job definitions as well as the Jenkins job runtime scheduler - which runs at all times. While you can run build jobs on this Master, its capacity is limited and you will typically have to allocate additional computing capacity to the Master by attaching to it additional build machines - also known as "Slaves". A single Master can have tens of Slaves attached to it.

The efficient technology we have developed for the CloudBees Platform allows us to partition and isolate EC2 resources for improved efficiency. Our underlying unit of compute power consumed by any application, including our own services like Jenkins, is an "app cell." Since our processing units are much more fine-grained than Amazon's, we can make use of our own pool more efficiently than just the m1.* levels available to you from Amazon. Consequently, the CloudBees Platform will allocate just enough capacity to run your Jenkins Master, making it a much more efficient setup than on an m1.small setup, for example. Yet, no build jobs will ever run on this Master; your build job will execute on dynamically allocated dedicated machines of your requested size (m1.small, m1.large), and will do so only for the duration of your build, by the minute. Furthermore, if you need to perform several builds in parallel, multiple machines will be dynamically allocated to your Master - but only for the duration of those builds. For example, running five ten-minute builds sequentially or in parallel will lead to the exact same cost on DEV@cloud! Consequently, with DEV@cloud, you end-up paying for a minimal fixed monthly fee for your Jenkins Master as well as for the exact number of build-minutes you've consumed (and only those in excess of your monthly quota).

If you were to do this yourself on Amazon EC2, you would have to decide how many machines of what size to allocate of the different EC2 machine sizes -- both for your Master as well as for your Slaves -- and you'd have to account for concurrent build Slaves upfront by statically allocating the number of machines your want to your Master. Alternatively, you could handle the dynamic provisioning and de-provisioning yourself, as well as manage the Jenkins workspace state.

Thus, not even accounting for the labor associated with managing the EC2 resources directly, as well as the Jenkins setup, CloudBees is able to offer a lower price for its development services like Jenkins than you would be able to achieve yourself using Amazon directly.

How much lower? Check it out - go here and calculate!

Onward,
Sacha

Sacha Labourey, CEO
CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Tuesday, December 13, 2011

Enhanced Pricing and Packaging for the CloudBees Platform


Since we launched our DEV@cloud development platform service more than a year ago, we have experienced tremendous success! Today we have thousands of developers using our services for their builds and tests.

In the spirit of giving at this time of year, we are pleased to announce:
  • Significantly lower pricing for DEV@cloud services
  • Changes in our technical support offering
...all nicely packaged up, just in time for the holidays. Read below for more about our progress this past year, and these exciting new announcements.

2011 Platform Highlights
In the last six months, we have added lots of new features, including:
  • The ability to add integrated CloudBees Ecosystem Partner services to your DEV@cloud environment. Ecosystem Partner offerings include popular services, such as JFrog, Sauce Labs, SonarSource and XWiki

  • Complete integration between the DEV@cloud and RUN@cloud services, with no need for you to set up any credentials or install any plugins

  • Ability to select a build machine that best fits your needs
We will continue to add more features on an ongoing basis. Additionally, we are working on several performance enhancements.

Lower Pricing for DEV@cloud Services: Efficient Technology and User Adoption
Today, we are announcing some changes to our development services pricing. First of all, we are removing all restrictions to the number of Git and Subversion repositories you can create: you are now only limited by the disk space allocated to your account. This is especially handy if you want to host one application per Git repository, for example.

With regard to Jenkins, we are significantly dropping our build minute prices and extending our build minute quotas! Given the massive number of builds we are now performing, we are dropping build prices by more than 80% and moving from one cent per build minute (or 60 cents per hour) to just 10.6 cents per hour! Our m1.large build prices will benefit from the same discount and will be billed at the equivalent of 42.5 cents per hour (billing still takes place by the minute). If you want to estimate what your new bill will be, we have also added a pricing calculator to our DEV@cloud pricing page.

Also, effective immediately, our Pro and Enterprise subscription customers will benefit from a doubling of the free build quota: from 2,500 free build minutes to 5,000 for Pro subscribers, and from 5,000 to 10,000 build minutes for Enterprise subscribers.
At these prices, doing continuous integration in the cloud should be mandatory!

Changes in Our Support Offering
Finally, on the support front, we are making some immediate changes. We will now provide silver support to all of our customers and only offer premium support options on-demand. Note that if you already benefit from a premium support subscription option, this change will not apply to you; you will be able to continue getting excellent support from CloudBees based on your current support terms and conditions. 

We are working on additional improvements and will be communicating more about them very soon. Stay tuned!

P.S. Just as we were getting ready to publish this blog, InfoQ published a round-up/overview of six major PaaS offerings, including CloudBees. CloudBees was ranked very highly! Read the InfoQ article.

Onward,
Sacha 

Sacha Labourey, CEO
CloudBees
 
Follow CloudBees:
Facebook Twitter

Session Affinity (Sticky Sessions)

We are rolling out session affinity for the routing/revproxy layer for RUN@cloud applications. Session affinity means that when a user is routed through to the backend server - they stay with that server as long as possible (makes it much easier for some frameworks to scale). If you don't know what it is - you probably don't need it !

If you are interested in testing this out in your account - we would love to hear from you.

Currently this requires having your own SSL router setup - as per http://wiki.cloudbees.com/bin/view/RUN/AppSSL. If you have that and wish to try session affinity - raise a support ticket with the name of your router that you wish enabled and instructions will be provided.

Enjoy!

CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Monday, December 12, 2011

Devops in the Cloud – Meetup this Thursday at Netflix

Netflix, JFrog, and CloudBees are excited to host a Devops in the Cloud Meetup this Thursday! Taking place for the first time at the Silicon Valley, Devops in the Cloud will put the spotlight on cloud computing best practices, tricks and tips from those who have been there.

Where: Netflix campus in Los Gatos, CA

When:  Thursday, December 15th, 5:30 PM to 8:30 PM (PT)

Details: Full invitation here, but it’s sold out and the waiting list is full. So…

Live Stream:  Attend virtually via live-stream!

You’ll hear about thoughts, solutions, tools and visions of future development practices in the cloud. Even better, a Netflix case study will demonstrate how many of these powerful processes are already possible.

Featured Presentations
  • Building Cloud Tools for Netflix – Carl Quinn, Joe Sondow, Netflix
  • Where are my runtime artifacts?! Cloud Deployment Using a Smart Repository – Frederic Simon, JFrog
  • Maintaining reliability in an unreliable world – Jeremy Edberg, Netflix
  • DEV@cloud, Behind the curtain – Kohsuke Kawaguchi, Jenkins founder and CloudBees Architect

Whether you're a Software Developer, a Release Engineer or a DevOps, this event is your opportunity to take a glimpse into the future of CI in the cloud.

Don't miss the live stream!

CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Friday, December 9, 2011

What's Next for Jenkins Enterprise by CloudBees?

We are running the shortest survey in the world. Help us understand what you will like to see next in Jenkins Enterprise by CloudBees?

Click here to take survey

Thanks,
Harpreet Singh, Senior Director of Product Management

CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter

Wednesday, December 7, 2011

Jenkins Plugin Tip: Access Control and Visibility in Actions


As we discussed before, Action is one of the primary ways plugins use to add more information to the top page, project pages, build pages, and so on.

Today I'd like to talk about writing protected actions that require some kind of access control. Depending on your requirements, there are several ways to do this. There are three parts in Action that affects behaviours with this regard.

One is the getIconFileName() method. As its javadoc states, returning null from this method hides your action from the HTML page. So if you have an action and you don't want to show it for users who don't have the permission to access it, it could be something like the following:

   public String getIconFileName() {
      return Jenkins.getInstance().hasPermission(Model.LIST) ? "gear.png" : null;
  }


The next is the getUrlName() method. This method also allows null as the return value. This might sound similar to getIconFileName() but the implication is different. With getIconFileName() returning null, the rendered HTML page won't show the link but if someone knows the URL, they can still access it. But with this method returning null, it's as if no such URL is recognized.

So with the code like the following, if someone without the permission requests it, he'll get 404 not found (modulo a bug, which I just fixed toward 1.443). This is sometimes desirable, as not only can you hide the data, but you can also hide the fact that something exists. For jobs, Jenkins does this — http://jenkins/job/top-secret-project/ returns 404 not found, not 401 forbidden, so guessing the project name will not help attackers gain any information.

   public String getUrlName() {
      return Jenkins.getInstance().hasPermission(Model.LIST) ? "template" : null;
  }


But this is sometimes undesirable, as users will not be prompted for authentication. If someone hits the link with an expired session, he'll get 404 not found and he needs to be smart enough to know that this is because he hasn't logged in, then navigate manually to the login page and come back. To avoid this problem, you'll do the following and advertise the URL all the time.

   public String getUrlName() {
      return "template";
  }


And that brings me to the third part, because getUrlName() now always returning non-null means you aren't actually checking if those who are accessing has a permission to do so. To do this, you use StaplerProxy.

class MyAction implements Action, StaplerProxy {
  ...

  public Object getTarget() {
      Jenkins.getInstance().checkPermission(Model.LIST);
      return this;
  }
}


What happens is that when the URL that someone requests hits this action (or something underneath), we'll verify that the requestor has the permission. If not, this will initiate the authentication, such as redirecting the user to the login page, or initiating the OpenID protocol with the preconfigured identity provider. Normally this interface is used to defer the UI processing to another object (sort of like how symlink works), but in this case we return this to indicate that we process this request by ourselves, and it's smart enough not to cause infinitely recursion.

I guess the rule of thumb is that (1) you have to check the permission either in getUrlName() or getTarget(), or else there's no access control, (2) you use getIconFileName() control the visibility in the HTML page, and (3) you use getUrlName() to control if you want 401 or 404 in case the access is denied.

- Kohsuke

CloudBees
www.cloudbees.com
 
Follow CloudBees:
Facebook Twitter