Release management, part 3: Supporting and scaling idempotent, isolated, and parallel jobs in Jenkins build pipeline with Docker

This blog post aims to illustrate how Docker is introduced into an existing Jenkins build pipeline for the purpose of performing test automation in parallel whilst guaranteeing isolated and idempotent environments that are scalable.

tldr;

In short, this is a truly exciting example of real world devops tool chaining with Docker as part of a build pipeline. Needlessly to say, by incorporating Docker into our build pipeline, it has has trimmed at least 70% of the build pipeline completion time by enabling lockless and parallel job execution. Since each jobs can run in parallel with Docker guaranteeing an isolated, idempotent environment for test automation, scaling out the build pipeline would be as simple as adding more Jenkins slaves. With clever use of Docker, Vagrant, Git and Jenkins together, this kind of tool chaining provides Platform as a Service (PaaS) that empowers both developers and operations alike, a shining example of devops.

In the first part of the Release Management blog series, the design of the build pipeline was conceived and outlined. Verification of the build pipeline design was covered in part two of the series. Permeating throughout this series is testability and automation. In addressing the correctness of test results and outcomes, the test execution environment must be isolated and test execution idempotent.

Guaranteeing an idempotent and isolated job execution

Ensuring a clean slate and running on a controlled environment when executing tests is paramount to the correctness of the test results. The first blog post discussed Vagrant as the tool for creating VMs, then running said tests on these VMs. Each job would perform vagrant up with a number of vagrant ssh command with specific -c parameter, then vagrant destroy for tear down. These commands guaranteed an idempotent and an isolated environment for each test execution in a Jenkins job. However there was a catch. The command, vagrant up, cannot be executed in parallel. See this discussion. This meant jobs that utilised Vagrant must be specified as a protected resource, accessible by one job, one slave at a time. This locking mechanism is supported by the Concurrent Throttle Builds Jenkins plugin, where jobs will wait and queue for certain resource to be freed before execution. The kicker is that this plugin is broken since October 2013 for Jenkins 1.536 and Concurrent Throttle 1.8. See Jenkins issue #19986 and #20211. Suffice to say, as of October last year, the build pipeline ran on a single Jenkins executor, taking a full hour to complete one pipeline build.

Speeding up and scaling out with parallel job execution

Whilst Vagrant definitely offers isolated and idempotent environments, at the time of writing, firing up parallel Vagrant boxes with VirtualBox was buggy at best. It is later confirmed that VirtualBox as a provider does not support parallel provisioning in Vagrant. Naturally if AWS was utilised as a Vagrant provider, this would be non-issue. Alas, it was not the case.

Secondly, provisioning each box takes a long time. Vagrant startup time are in minutes and increases significantly when compounded by copious amounts of apt-get or yum installs during provisioning. Although this can be mitigated by having pre-built Vagrant boxes with all dependencies installed, but it does become an added maintenance task.

The solution here is Docker. Unlike Vagrant, Docker creates lightweight containers, akin to LXC. Each containers can be pulled and pushed. This is similar to the Git model, with docker.io offering a centralised repository of all Docker images. With the recent Docker 1.0 release, it is ready for prime time and production usage. Indeed, using Docker is quick and easy. Create the necessary Dockerfile with the same provisioning steps as Vagrantfile is a breeze. When the same user as the Jenkins slave is included in the Dockerfile, upon successfully building the image, the Jenkins slave can then mount any directory onto this container and have the same user as Jenkins slave writing to that mounted directory from the container. This is central for reporting back test results.

Docker images, which act as a baseline for a container, provides an idempotent and isolated environment with the major benefit of being able to run containers in parallel, even if they are based on the same image.

Vagrant vs Docker – a moot comparison

The biggest question is does Docker completely replace Vagrant? The answer is a very clear no. Vagrant is extremely useful during development, essentially giving developers the ability to run as many varied platforms as possible. Docker is a container, an isolated kernel namespace running on top of the a shared Linux kernel. The difference is minute, but depending on the testing requirements, when VMs are required, Vagrant is still a valid choice over Docker.

Platform as a service with Git, Jenkins, Docker and Vagrant! Devops to the rescue

Having Vagrantfile and Dockerfile in their own repo is clever. With Git and Jenkins, provisioning updated sets of Vagrant boxes or Docker containers from Vagrantfile and Dockerfile respectively is easy and will give you the ability to track changes in these VMs and containers. When changes are detected, Vagrant boxes can be rebuilt and list of available boxes updated. Likewise, Docker containers can be rebuilt and pushed as the latest tag. This will both enable and empower the development team to maintain their own production VMs and containers for development, testing and releases. This is PaaS with liberal sprinkling of devops goodness.

Advertisements

11 thoughts on “Release management, part 3: Supporting and scaling idempotent, isolated, and parallel jobs in Jenkins build pipeline with Docker

  1. @yveshwang if I understood this correctly you start several docker containers in your Jenkins.
    May I ask if you needed a good server machine to achieve this?
    I failed to achieve this with LXC containers (not docker) because my tests started to failing (processes started to die in the middle of the testing).
    I had a aws m1.medium at the time.

    • at the time of writing, docker was running on bare metal. i am also interested in moving this to AWS. secondly, docker containers are fired up on jenkins slaves, separate machines.

      think i’ll give m1.medium a shot too. keep you updated to see if i run into similar situations.

      • Thanks, I ended up moving my Jenkins to a T2.micro and connecting it to aws slaves. But this is a little costly and I’d like to know what kind of server I need to have to use several LXC containers so if you find anything, please tell me, I’d appreciate it. 🙂

  2. Pingback: Jenkins, jenkins-job-builder and Docker in 2 steps | something else

  3. Was your experience with vagrant up in parallel mode similar to mine?
    1. Have a Jenkins job that allows concurrent builds.
    2. Build on Pull Request
    3. Jenkins job runs vagrant up, vagrant ssh -c build.sh, vagrant destroy -f
    4. The first to run will cause the other VMs to be in the ABORTED state when vagrant destroy -f is called.

    • that is correct. we had to use a throttle plugin (from memory) as a work around to protect vagrant as a resource. this made it of course made it a bottleneck. use Docker instead. now 🙂

      • I’m not familiar with Docker other than the basic concept of it. Are you using it instead of the Vagrant and doing your build processes inside of the docker? I have read your posts, but the lack of Docker has been hurting me.

  4. Pingback: Microservice API on Elastic Beanstalk with Jetty, Jersey, Guice, and Mongodb | something else

  5. Pingback: What #WoW has taught me about building a better #devops team in an enterprise | something else

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s