Tuesday, July 31, 2012

Getting Started with CloudBees and Eclipse

I've created a series of videos to show how to use the Eclipse IDE with the CloudBees Java PaaS platform that you can find on CloudBeesTV.  I spent a day or so working through all this and I thought it would be a good idea to capture what I learnt in some step-by-step tutorials - the integration is really very well-done.  BTW, I really like the new Camtasia software for Mac from TechSmith: I've been using an older version for Windows for a while and always liked it, but Camtasia 2 for Mac is awesome.

Eclipse Setup

If you are fairly new to CloudBees and/or Eclipse, then I'd recommend watching Part 1, which goes through what you need to do to set up your Eclipse environment and particularly which Eclipse plug-ins to install, but for a quick reference here are the plug-ins I'm using with Eclipse Indigo/Juno:

As you can see in the screenshot below, the CloudBees Eclipse plug-in (the CloudBees view is open in the bottom left corner and here I'm using it to watch the output of a Jenkins CI build in the Eclipse console output).  You'll need the eGit plug-in for Git SCM integration with the CloudBees Forge and either Subclipse or Subversive if you plan on using SVN, plus the m2e plug-in for Maven integration: you can find all the necessary plug-ins easily on Eclipse Marketplace or just use the URLs from the picture above.  Don't forget to install eGit and Subclipse/Subversive before you install the CloudBees Eclipse plug-in or you won't get the CloudBees Forge integration plug-ins.

Getting Started with CloudBees and Eclipse

If you're all good with your Eclipse setup, then the remaining videos will walk through creating both Ant and Maven projects with CloudBees integration within Eclipse, and how to configure these to work with cloud-based PaaS services, including SCM repositories, Jenkins CI continuous build integration and RUN@cloud deployment and monitoring.  If you are pretty comfortable with all this, or if you just want to see an overview of how the CloudBees Eclipse integration works, then you might want to start with the CloudBees and Eclipse Overview video, which goes through all the scenarios in one recording; if you want a little more of a step-by-step explanation of any of the features, then here are the links to the videos in the series:

  • Part 1 introduces the CloudBees PaaS environment and walks through my Eclipse setup, which plug-ins to install and how to set up the environment.
  • Part 2 covers using the CloudBees Eclipse plug-in to create a new CloudBees project, and how to deploy locally and to the Cloud.  The CloudBees plugin lets you create a web app template that you can use as a starter project.
  • Part 3 explains how easy it is to integrate other cloud-based services with your apps, as well as a quick look at the Jenkins CI (Continuous Integration) service.
  • Part 4 shows how to import Maven projects using the CloudBees plug-in, and how to deploy applications locally and in the Cloud via the Bees SDK and Eclipse.  The video also shows how to use CloudBees environments and Maven profiles to control the deployment process.  This is a really useful feature and I'll be doing a follow-on blog to drill down a bit deeper into how this works and what you can use it to do.
  • Part 5 illustrates how to use Maven m2e Run Configurations with the CloudBees Maven plug-in to select CloudBees environment profiles when deploying from Eclipse (Part 6 shows how to do the same when deploying to the CloudBees PaaS).
  • Part 6 concludes by showing how to select CloudBees environment profiles when deploying to the CloudBees PaaS.

Try It Yourself

If you want to follow along, just sign up for a CloudBees development account (it's free!) and get busy.  I've put the sample app up on GitHub, so just clone from there with
git clone https://github.com/mqprichard/start-cloudbees-eclipse.git


By Mark Prichard, CloudBees

Mark Prichard is Java PaaS Evangelist and Senior Director of Product Management at CloudBees. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was Product Manager for the WebLogic Platform. A graduate of St John's College, Cambridge and the Cambridge University Computer Laboratory, Mark is based at the CloudBees office in Los Altos, CA.
Follow Mark on twitter and his blog Clouds, Bees and Blogs.

Follow CloudBees:


Wednesday, July 25, 2012

Continuous Information Vol. 3: Jenkins Growth Explodes

Summer 2012 - Volume 3

40K+ Jenkins Installations, 1M+ Jobs
It will come as no surprise to you that the Butler’s popularity is growing wildly. As the glue that binds today’s build and software validation systems, Jenkins has become a core part of the software development processes in both startups and enterprises alike. In March we crossed a major milestone of 40,000 Jenkins installations, and at the end of June, the world boasted more than 43,500 installations!
Other cool stats:
  • More than 1.25M jobs
  • 500+ plugins!
  • 13,400 support tickets created as of January 2012
    (up from ~0 in January 2007)
Check out the statistics on Jenkins usage for yourself.

Table of Contents:

40K+ Jenkins Installations, 1M+ Jobs
What’s new in Jenkins?
Call for JUC Speakers & Sponsors
Eliminate Downtime with HA Jenkins
Featured Blogs
Build Your GitHub Projects with Jenkins in the Cloud on BuildHive
Kohsuke’s Pick: Interesting OSS Plugins
Useful Free Enterprise Plugins

New Book
Jenkins Continuous Integration Cookbook

by Alan Mark Berg – 80+ recipes to maintain, secure, communicate, test, build and improve the software development process with Jenkins.

Jenkins Wants YOUJenkins-CIA-program
Jenkins depends on its community to expand the core, write plugins and spread the good word about Jenkins. Jenkins CIA (short for CI Ambassador) program is recruiting agents to hit the streets. Learn how you can help the community and get some cool SWAG too!
Jenkins Featured in Google I/O 2012
Jenkins was featured at Google I/O a few weeks ago in a talk about Android development tools. Here’s an awesome video showing Jenkins integration with Lint, a new Android developer tool.
Webinar RecordinG:
Eliminate Jenkins Downtime, Increase Development Uptime
Kohsuke and Harpreet Singh show High Availability and benefits delivered in CloudBees’ new Jenkins Enterprise plugins.
  Webinar recording:
9 Ways to Dominate Development
with Jenkins
Learn tips for making the most out of Jenkins, brought to you by Jenkins developer Ryan Campbell and WANdisco’s Director of Training, Mike Lester. This webinar builds on Kohsuke Kawaguchi’s white paper,
7 Ways to Optimize Jenkins.
Start a Jenkins Meetup
Can’t make it to a JUC, or just want more Jenkins? Start a Jenkins meetup in your community. Just drop us a note at continuous-information@cloudbees.com for tips on where to start. 
Are you doing something exciting with Jenkins? Send us a short overview and we’ll share it with the community. Just drop an email to continuous-information@cloudbees.com

What's new in Jenkins?
Kohsuke highlights the hottest Jenkins activities…
Jenkins user conference update
Call for Speakers and Sponsors:
JUC San Francisco 9/30 and
Antwerp 11/13
Speaking submissions are now open for both conferences. Here’s your chance to tell your story, share best practices you have developed and showcase how Jenkins technology has made your life – and your software – better! Submit your talk before July 27 for San Francisco and September 21 for Antwerp, and of course, register to attend.
If your company uses Jenkins, consider sponsoring a JUC to help give back to the community. We just introduced a new Corporate sponsorship level to make it even easier.
If you haven’t been to a JUC yet, take a peek to see what it’s about – check out Jesse Farinacci’s overview of JUC NY in May and Kohsuke’s blog summarizing JUC Paris in April. Kohsuke also wrote a great review of JUC New York, and you can also take a look at the New York videos.
  • Tokyo, on July 29, over 900 Jenkins users already registered!
  • San Francisco, on September 30, immediately preceding JavaOne
  • Antwerp, on November 13, immediately preceding Devoxx
Special thanks to our JUC sponsors for all 2012 JUCs:
Eliminate Downtime
with HA Jenkins
If you need enterprise robustness and high availability in Jenkins, check out Jenkins Enterprise by CloudBees. Your Jenkins installation will be supported and enhanced by Jenkins experts from CloudBees, plus you get plugins specifically designed to address enterprise requirements. Jenkins Enterprise by CloudBees helps you automatically recover from failures, secure Jenkins, prevent failures altogether, provision faster, better govern Jenkins, and accelerate the build process.
Some of the newest features include… Take the first step towards eliminating Jenkins downtime – get details and check out the full list of Jenkins Enterprise by CloudBees plugins
Featured Blogs
Build Your GitHub Projects with Jenkins in the Cloud on BuildHive
Talk about build automation! Combine cloud-powered Jenkins with your GitHub development projects using the new BuildHive service. Added bonus: show your build status on GitHub and easily verify the quality of incoming pull requests.
BuildHive and Jenkins automatically detect your build platforms so your builds and tests run seamlessly. As soon as a repository is changed, the system automatically initiates the build. If Jenkins says your builds are blue, they are good-to-go. Try it out – it’s free and super-simple.
kohsuke's pick
Interesting OSS Plugins
Useful Free Enterprise Plugins
  • Folders plugin – organize jobs using folders
  • Backup to Cloud plugin – lets you backup Jenkins through Jenkins into the cloud
  • And more, including the Wasted Minutes plugin, CloudBees Deployer plugin, and CloudBees Status plugin
… brought to you by CloudBees. Learn more

Hopefully see you at a JUC this Fall! In the meantime, if you have other suggestions or thoughts on how to make Continuous Information more useful to you, please send the newsletter team a note. 

Lisa Wells
Managing Editor, Continuous Information

Kohsuke Kawaguchi
Technical Editor, Continuous Information
If you no longer wish to receive emails from CloudBees, click on the following link: Unsubscribe

Monday, July 23, 2012

Denial of Service Attack on DNS Event

Recently our DNS (domain name service) provider (Zerigo) came under a distributed denial of service attack (DDOS). DNS is a critical service and this caused outages of applications and services for some users - DNS is distributed - so this effect was not observed evenly (i.e. what you experienced was probably different to what someone else experienced). DDOS are hard to combat as they are conducted from many compromised desktop machines (usually unknown to the owner of the machine) - you can read more about them here.

We use a service like Zerigo as they have servers in many continents, allowing for high availability of DNS. However, this attack was targeted and sustained- and clearly this global distribution was not able to handle it. We were forced to make changes which take time to propagate to your local DNS service (DNS updates always take some amount of time due to TTL - even in that case, some DNS servers can cache entries for longer than they are meant to).

Following is a sequence of events and what is being planned to mitigate this issue:

On Monday July 23rd, at 3:30AM UTC Cloudbees engineers in the APAC region were alerted by our infrastructure monitoring to a higher than normal number of alerts of accessibility within the infrastructure.  While these types of alerts are not uncommon due to the large number of moving pieces within the Cloudbees system, at certain thresholds they typically indicate larger scale problems such as network or connectivity issues.

Over the next 90 minutes, specific accessibility checks performed by the engineering team indicated that systems were still reachable, and that the problem seemed more likely related to networking issues from the monitoring system itself.

Around 5:00AM UTC engineers began to realize the problem seemed more likely related to DNS issues.  At this time our engineers were able to confirm that our DNS provider, Zerigo, was having an outage related to a DDoS attack.

Over the next few hours engineers began moving pieces of infrastructure around, and reconfiguring DNS entries to work around the attack.  DNS availability was intermittent, but working.  Engineers in different parts of the APAC region were experiencing different levels of availability of Cloudbees services based on where they were.

At 10:00AM UTC, as engineers in the US-EAST region began coming available, it became evident that the DNS outage experienced by Zerigo would be long term.  At this point we began our plans for migrating DNS to a new provider.  Over the next 60 minutes we manually copied over all production facing DNS entries in order to bring systems back online as quickly as possible.  At around 11:00AM UTC we switched over to the new DNS provider and watched as propagation began.

Since 11:00AM UTC, DNS entries for cloudbees.com and cloudbees.net related domains should have returned to normal.

There were a number of factors of this event we want to address:

1) It took a long time for our engineering team to recognize the issue.  This issue stems from the fact that our monitoring alerted it was unable to reach certain services, but when we accessed those services it was clear they were running okay.  However, in many cases we reach services through a Virtual Private Network (VPN) which uses a separate name resolution system that our production facing one.  Thus, when monitoring said services were down, and we verified they were in fact, up, the initial inclination was that monitoring was reporting faulty status data.

Because we have not experienced a DNS outage before, we were not initially looking at the DNS service as a potential vector for an outage.  To better handle this scenario, we will be adding monitoring of our DNS system health specifically to our internal and external monitoring systems.

2) Our choice of DNS provider, Zerigo, was made a few years ago during a time when our infrastructure needs were different than today.  Zerigo was a great choice at the time, and gave us easy scaled DNS with global capacity.  However, today demonstrated some of the limitation to their infrastructure, and also to their public updates of status reports.  While we have no doubts today's events will make Zerigo's DNS infrastructure even stronger, we feel it's time to move our DNS to a provider that has greater resources and capacity to handle large scale DDoS attacks.  We will be making assessments of these providers over the next few weeks.

3) Our cloudbees.com and cloudbees.net top level domain TTL settings are set for very long expiry times, and are currently not configurable with our domain registrar.  This created problems today in that after we switched our DNS to a new provider, some customers still retained cached DNS data and were not able to retrieve the records from the new provider for extended periods of time.  We also intend to fix this over the next few weeks.

4) Our status site, status.cloudbees.com, was unavailable in most cases due to the DNS outage.  We have gone to great lengths to decouple this site from our running infrastructure so it's available even if other pieces are not.  However, today showed another weakness we need to address.

5) Certain aspects of our RUN@Cloud infrastructure remained available during the DNS outage, because DNS for certain critical pieces are still served by the stax.net domain, which is hosted by a different DNS provider.  Some customers may not have been impacted as a result, and we will use the knowledge we gained today with this separation of services and DNS providers as we move forward with hardening our setup.

We are very sorry for the inconvenience this caused - thankfully this is a problem that can be addressed and will be.

-- Caleb Tennis, Elite Developer

Follow CloudBees: