Thursday, January 16, 2014

Real Devices, Real Tests - Using a Device Cloud with Jenkins

Mobile application developers: I'd like to introduce our newest verified partner,AppThwack and their device cloud service for mobile application testing. I tried this out over the Christmas holidays, really liked the service and am very glad to have AppThwack on board with DEV@cloud. If you're doing continuous integration or automated testing of iOS/Android mobile applications, then I would strongly recommend you check out AppThwack. You'll find the AppThwack partner listing on the CloudBees website at http://www.cloudbees.com/platform-service-appthwack.cb.

I have a pet Android application project that I use to try out various build/test features of the Jenkins CI platform, and recently I decided to experiment with running those automated test suites on real devices using AppThwack's device cloud. I've written a lot in the past about how to run tests using software emulators to simulate real devices and there's definitely a place for that: with DEV@cloud, it's easy to configure and I have a job set up to build whenever I push a commit to the GitHub repository where the project is hosted, in classic CI style. You can see that job online here; it's a pretty good example of how to set up and run builds with automated tests using the Android Gradle Plugin with an emulator.

However, for real-life continuous integration and testing of mobile applications, that's really just the starting point.  Emulator-based testing alone isn't enough - you need to see how your application behaves on real devices.  When I develop, I normally have at least one device permanently connected to my laptop and I run gradle clean connectedCheck regularly to check that I haven't broken anything: I want something similar as part of my continuous integration process, and that's where the service from AppThwack is really handy. There are quite a few vendors out there offering device clouds for testing and, to be honest, I tried AppThwack more or less on a whim - however, I have been very impressed with their service (and their API/plugin support, more on that later) and I think that most people who use Jenkins to build and test mobile applications would find them a great fit.  As always, I've put some detailed examples online, so that you can see exactly how it works.

In this blog, I'll go over the basics of how to use a device cloud with CloudBees' DEV@cloud Jenkins CI service (actually, it works exactly the same if you are using Jenkins Enterprise or your own local Jenkins installation).  In a follow-up blog, I'll dig a bit deeper into automated Android/iOS testing and some useful lessons learned from this exercise.  Being able to do multiple tests in parallel really forces you to think carefully about your testing strategy and definitely helps to drive out any issues with the tests themselves, but more on that next time.

Getting started with AppThwack is easy: just go to https://appthwack.com and sign up.  There's a free trial option, so you can get started straightaway.  Go to the Profile page (drop-down menu under your username, top-right on the home page) and make a note of your AppThwack API key, as you'll need that when configuring the Jenkins plugin.  You'll also want to create an iOS or Android project (make a note of the name, you'll need that too) and define one or more device pools for your tests.  Create your project and then upload your .apk or .ipa app build by clicking on the green button: that will take you to the screen shown below, where you can either select from a pre-defined set of devices, or define your own ("select from list").  In my example, I've created a custom device pool: again, make a note of the device pool name.


If you have an iOS or Android test project handy, then you can go ahead and upload that to run your tests directly from the web page, but I'm going to focus here on how to use the service with automated Jenkins CI build jobs.  Make sure you have the AppThwack Jenkins Plugin installed via the update center; then go to Jenkins -> Configure Jenkins  and scroll down until you see the AppThwack section. Enter your API key and save.  Now go to the Jenkins configuration for your iOS or Android project and add a Post-Build Action ("Run Tests on AppThwack").  If you don't see that option in the drop-down list, it means that the plugin hasn't been installed: did you remember to restart Jenkins?  You can see example builds online for Android and iOS projects.  Here's what the Android configuration looks like:



In this example, my project is "gasp" and I'm using a single-device pool ("nexus72" - an Asus Nexus 7.2). This is a Gradle build using the Android Gradle plugin, so the debug and test .apk files are located in the build/apk folder, but if you're building your project with ant you'll find those in the <project>/bin and <test-project>/bin folders.  I'm running JUnit and Robotium tests, but AppThwack supports a variety of iOS/Android test frameworks.  Save the configuration and you're ready to run the build. Assuming that your project builds without errors, your application and tests will be uploaded and and the test runs scheduled.  You'll see a link in the console output that will take you to the details for the test run: here's what it looks like for a completed project:


There's a wealth of information about the test run by device, with performance data plus of course the full test results.  You can publish that data (click "Share" for the link) directly from AppThwack, but we want to get that information back into Jenkins, so that we can configure other, post-build actions based on the success or failure of the build.  As I write this, the current version of the AppThwack plugin (1.4) doesn't support that, but hey, this is Jenkins and the plugin is open-source, so I added some code to poll the AppThwack server to get the job status and to download the results to the Jenkins workspace when the test run completes.  Give the credit to the AppThwack folks: they have done a nice job exposing their APIs (check out the AppThwack github page), so extending the plugin was easy enough. By the time you read this, there should be a new version with this functionality but just in case, here's the link to my repo with the change, so you can cut your own SNAPSHOT build.

 Now the console output looks like this:


Browsing the workspace for the project, you'll see a folder named device-results-<id> for each test run, with sub-folders for each device tested containing test results in raw and JUnit XML format, like this:


Bottom line: using a device cloud service like AppThwack is definitely the way to go for continuous integration of mobile applications.  You can test your app on literally hundreds of devices: all using non-rooted, commercially-available models.  You can structure your CI setup to build up from simple compilation, through emulator-based tests, to single-device and multi-device test runs, all the way to beta testing using services like HockeyApp, TestFlight or Zubhium.  You can do it all in the cloud with DEV@cloud, or from your local Jenkins CI installation.  There's no excuse for submitting a buggy app to the App/Play Store! 


Mark Prichard, Senior Director of Product Management
CloudBees


Mark Prichard is Java PaaS Evangelist for CloudBees. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was Product Manager for the WebLogic Platform. A graduate of St John's College, Cambridge and the Cambridge University Computer Laboratory, Mark works for CloudBees in Los Altos, CA.  Follow Mark on Twitter and via his blog Clouds, Bees and Blogs.

1 comment:

  1. Thanks for the write-up Mark. Your compilation-to-test CI example tells a great story, and we're excited to have AppThwack play a part!

    ReplyDelete