Jenkins, jenkins-job-builder and Docker in 2 steps

So here is a simple example that will help provision a VM with Jenkins, jenkins-job-builder and Docker all in one with Vagrant.

Gone are those embarrassing git commits with those peskey jenkins-job-builder yaml files! Throwing in Docker install in Vagrant for Mac users who shuns boot2docker.

Some bonus stuff like examples of provisioning Jenkins plugin via the cli interface, creating first time Jenkins user are chucked in too.

Check it out! https://github.com/yveshwang/jenkins-docker-2step

2-steps are as below

  1. vagrant up
  2. point your browser to http://localhost:38080 and enjoy

and then ..

jenkins-jobs --conf jenkins-job-builder-localhost.ini test your-jobs.yaml -o output/

3 steps really… and for the keen learners wanting some intro material for docker, go here, and here for jenkins-job-builder.

Advertisements

Release management, part 3: Supporting and scaling idempotent, isolated, and parallel jobs in Jenkins build pipeline with Docker

This blog post aims to illustrate how Docker is introduced into an existing Jenkins build pipeline for the purpose of performing test automation in parallel whilst guaranteeing isolated and idempotent environments that are scalable.

tldr;

In short, this is a truly exciting example of real world devops tool chaining with Docker as part of a build pipeline. Needlessly to say, by incorporating Docker into our build pipeline, it has has trimmed at least 70% of the build pipeline completion time by enabling lockless and parallel job execution. Since each jobs can run in parallel with Docker guaranteeing an isolated, idempotent environment for test automation, scaling out the build pipeline would be as simple as adding more Jenkins slaves. With clever use of Docker, Vagrant, Git and Jenkins together, this kind of tool chaining provides Platform as a Service (PaaS) that empowers both developers and operations alike, a shining example of devops.

In the first part of the Release Management blog series, the design of the build pipeline was conceived and outlined. Verification of the build pipeline design was covered in part two of the series. Permeating throughout this series is testability and automation. In addressing the correctness of test results and outcomes, the test execution environment must be isolated and test execution idempotent.

Guaranteeing an idempotent and isolated job execution

Ensuring a clean slate and running on a controlled environment when executing tests is paramount to the correctness of the test results. The first blog post discussed Vagrant as the tool for creating VMs, then running said tests on these VMs. Each job would perform vagrant up with a number of vagrant ssh command with specific -c parameter, then vagrant destroy for tear down. These commands guaranteed an idempotent and an isolated environment for each test execution in a Jenkins job. However there was a catch. The command, vagrant up, cannot be executed in parallel. See this discussion. This meant jobs that utilised Vagrant must be specified as a protected resource, accessible by one job, one slave at a time. This locking mechanism is supported by the Concurrent Throttle Builds Jenkins plugin, where jobs will wait and queue for certain resource to be freed before execution. The kicker is that this plugin is broken since October 2013 for Jenkins 1.536 and Concurrent Throttle 1.8. See Jenkins issue #19986 and #20211. Suffice to say, as of October last year, the build pipeline ran on a single Jenkins executor, taking a full hour to complete one pipeline build.

Speeding up and scaling out with parallel job execution

Whilst Vagrant definitely offers isolated and idempotent environments, at the time of writing, firing up parallel Vagrant boxes with VirtualBox was buggy at best. It is later confirmed that VirtualBox as a provider does not support parallel provisioning in Vagrant. Naturally if AWS was utilised as a Vagrant provider, this would be non-issue. Alas, it was not the case.

Secondly, provisioning each box takes a long time. Vagrant startup time are in minutes and increases significantly when compounded by copious amounts of apt-get or yum installs during provisioning. Although this can be mitigated by having pre-built Vagrant boxes with all dependencies installed, but it does become an added maintenance task.

The solution here is Docker. Unlike Vagrant, Docker creates lightweight containers, akin to LXC. Each containers can be pulled and pushed. This is similar to the Git model, with docker.io offering a centralised repository of all Docker images. With the recent Docker 1.0 release, it is ready for prime time and production usage. Indeed, using Docker is quick and easy. Create the necessary Dockerfile with the same provisioning steps as Vagrantfile is a breeze. When the same user as the Jenkins slave is included in the Dockerfile, upon successfully building the image, the Jenkins slave can then mount any directory onto this container and have the same user as Jenkins slave writing to that mounted directory from the container. This is central for reporting back test results.

Docker images, which act as a baseline for a container, provides an idempotent and isolated environment with the major benefit of being able to run containers in parallel, even if they are based on the same image.

Vagrant vs Docker – a moot comparison

The biggest question is does Docker completely replace Vagrant? The answer is a very clear no. Vagrant is extremely useful during development, essentially giving developers the ability to run as many varied platforms as possible. Docker is a container, an isolated kernel namespace running on top of the a shared Linux kernel. The difference is minute, but depending on the testing requirements, when VMs are required, Vagrant is still a valid choice over Docker.

Platform as a service with Git, Jenkins, Docker and Vagrant! Devops to the rescue

Having Vagrantfile and Dockerfile in their own repo is clever. With Git and Jenkins, provisioning updated sets of Vagrant boxes or Docker containers from Vagrantfile and Dockerfile respectively is easy and will give you the ability to track changes in these VMs and containers. When changes are detected, Vagrant boxes can be rebuilt and list of available boxes updated. Likewise, Docker containers can be rebuilt and pushed as the latest tag. This will both enable and empower the development team to maintain their own production VMs and containers for development, testing and releases. This is PaaS with liberal sprinkling of devops goodness.

Selenium & Vagrant: Headless Chrome in the same old 2 steps

This is the second instalment of the Selenium guide to end all Selenium guides.

Selenium has a bad rep for being difficult to setup. With a little Vagrant know-how, this guide will lower the bar for good. The previous blog post illustrated how to seamlessly set up Firefox headless, Xvfb, and Selenium 2.42.1 grid & node using Vagrant. Boom, now we got headless Chrome in the same old two steps. Init.d script included as usual.

The following steps are:

1. git clone this repo https://github.com/yveshwang/selenium-2step
2. Vagrant up selenium and point your test to the port forwarded default hub location.

The awesome sauce continues.

Selenium & Vagrant: Setting up Selenium grid and node, and headless Firefox in 2 steps

This is the Selenium guide to end all Selenium guides.

2 steps is all you need to fire up a potentially production-ready Selenium 2.42.1 grid and node for your test automation awesome sauce. The Vagrantfile in question will install and setup Firefox headless on a precise64 box. Yes, init.d script is included for both Xvfb, Selenium grid and node respectively. Good times!

The following steps are:

1. git clone this repo https://github.com/yveshwang/selenium-2step
2. Vagrant up selenium and point your test to the port forwarded default hub location.

Enjoy!

Docker in OSX via boot2docker or Vagrant: getting over the hump

There are bucket loads of fantastic Docker related blog posts out there. This is yet another tldr guide to get the readers over the hump, particularly illustrating how you can mount local file systems onto a Docker container with Vagrant.

Before we begin, Docker is a client/daemon application. Docker daemon can listen to commands from the Docker client. The client is thin and all containers are hosted on the daemon. This means if you point Docker client to another daemon endpoint, the usual available list of images and containers will change. This is a good thing.

Installing Docker on OSX

As stated on the official Docker installation guide, install via Homebrew is highly recomended.

brew install boot2docker
brew install docker

Note that brew only installs the Docker client on OSX. boot2docker is the VirtualBox wrapper that will fire up Docker daemon. This is important as at the time of writing, commands performed on Docker clients are executed and takes place on the Docker daemon. For example, mounting files to a container really refers to mounting files from the server that the Docker daemon is running to the desired container. This is a cause of many confusion.

Running Docker with boot2docker

The installation guide is really good, and here is a tldr version. Note that the following script uses a different boot2docker port. This is to avoid port conflict with the Docker daemon via Vagrant later.

echo “export DOCKER_PORT=14243” >> ~/.boot2docker/profile 
boot2docker init
boot2docker start
export DOCKER_HOST=tcp://localhost:14243
docker version

If all goes well, doing docker version will yield the following.

docker_version. all good

docker_version. all good

Setting up Docker with Vagrant

The steps are based on John Zanchetta’s blog post, with all of the Vagrant steps trimmed and one network routing step found redundant. This is the tldr version and it assumes you are comfortable with Vagrant

Step 1 – Provision the Docker daemon box
Using the following Vagrant file, fire up the VM called docker

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
    config.vm.define :docker do |docker|
        docker.vm.box = "precise64"
        docker.vm.box_url = "http://files.vagrantup.com/precise64.box"
        docker.vm.network "forwarded_port", guest: 80, host:58080
        docker.vm.network "forwarded_port", guest: 4243, host: 4243
        $script = <<SCRIPT
wget -q -O - https://get.docker.io/gpg | apt-key add -
echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list
apt-get update -qq 
apt-get install -q -y --force-yes lxc-docker
usermod -a -G docker vagrant
sed -e 's/DOCKER_OPTS=/DOCKER_OPTS=\"-H 0.0.0.0:4243\"/g' /etc/init/docker.conf > /vagrant/docker.conf.sed
cp /vagrant/docker.conf.sed /etc/init/docker.conf
rm -f /vagrant/docker.conf.sed
service docker restart
SCRIPT
       docker.vm.provision :shell, :inline => $script
    end
 end

Step 2 – Vagrant or boot2docker, an easy switch
Point the DOCKER_HOST to the Vagrant-based Docker daemon. Note that by simply changing the DOCKER_HOST, you can point your Docker client to whichever Docker daemon you so desire; boot2docker or Vagrant-based for example.

export DOCKER_HOST=tcp://localhost:4243
docker version

If all goes well, you are ready to build!

Step 3 – Build your first Docker container and mount volumes in Docker with Vagrant
Once you have a container built, mounting the /vagrant is no different from mounting any other directories to the container from the Docker daemon. For example

docker run -t -i --name test --volume /vagrant:/vagrant:rw example:test

Easy!

Step 4 – Make your first push, my only advice
This blog post would not be complete without recommending you to perform your first Docker push.

If like me, you have created your container with “example” as the repository, and “test” as the tag based on a Dockerfile

docker build -t example:test .

And upon pushing (assuming the username is “yves”), for example

docker login
docker push yves/example

You will notice the following response:

The push refers to a repository [yves/example] (len: 0)
2014/05/31 00:46:35 No such id: yves/example

This is because the push command needs username, which has to be part of the repository name. However, the above example, the repository name is merely just “example”.

The work around here, based on scouring issues #720, is to create a new tag that includes the username as part of the repository name. For example:

docker tag  yves/example:test_tag
docker push yves/example

then watch the magic unfold in front of your very eyes.

The big take away is to always prefix your repository name with your docker.io username, followed by a /. For example, yves/ubuntu, yves/centos yves/whatever.

My docker image, yves/yves, is built based on the Vagrant file mentioned in https://macyves.wordpress.com/2014/05/20/release-management-part1-test-automation-with-vagrant-and-jenkins-build-pipeline/

Why Vagrant over boot2docker?

At the time of writing, mounting directories from your OSX machine to a container hosted by boot2docker is quite a mind field. See issues #3957. boot2docker is very well supported, but I feel more at ease with Vagrant.

Side note

Docker is really resilient. Simply retry the same command when encountering failures during image building or pushing to docker.io. This is in part thanks to the untagged images that floats around Docker daemon and that an image is really a repository containing a bunch of changesets performed, akin to Git.

However, if you wish to clean up all untagged images, see this blog post.

A quick primer on chroot, sbuild, mock and Vagrant

For me, some of the build tools are used so seldom that when things go pear shaped, it is more often a mad scramble to put out fires. Being big on devops, I wanted to be able to run the all packaging scripts on my own setup (or any other developer’s setup at that) or VMs. Particularly when issues arise, be it dependencies related or error in the packaging process, having a Vagrant box at hand with all the production trimmings mitigate the need to use production Jenkins server for debugging where possible. This is a good thing!

This is a blog post that serves as a quick primer for myself and anybody who feels like dabbling in old school package building for their release management process with a modern twist.

You will need Vagrant, Virtualbox and a couple of reliable Ubuntu and Centos Vagrant boxes. Some knowledge of deb and redhat build scaffolding is also helpful.

Getting started with chroot & sbuild

The following steps will yield a precise64 chroot environment.

Note that I was unsuccessful with setting up sbuild using Vagrant provisioning scripts. So the following steps are executed after the VM has been provisioned.

sudo apt-get update
sudo apt-get install sbuild
sudo sbuild-createchroot --arch=amd64 precise /srv/chroot/precise-amd64 http://no.archive.ubuntu.com/ubuntu --components=main,universe
sudo sbuild-adduser vagrant
cp /usr/share/doc/sbuild/examples/example.sbuildrc /home/vagrant/.sbuildrc

The above steps will create a symlink folder that resides in /etc/sbuild/chroot/precaise-amd64-sbuild that points to /srv/chroot/precise-amd64. You can now list the available chroot by doing

schroot -l

and enter the precise64 jail by doing

schroot -c precise-amd64-sbuild

Ideally now is a good time to generate a gpg key.

sudo sbuild-update --keygen

If you run into entropy issue and require more bytes, put the keygen process in the background, then do the following

sudo apt-get install rng-tools
sudo rngd -r /dev/urandom

At this point, you have almost a happy sbuild setup. However, you probably want to mount a folder to the chroot environments, accessible by the vagrant user.

Add the following line to /etc/chroot/default/fstab, /etc/chroot/mount-defaults/fstab, /etc/chroot/sbuild/fstab

/vagrant        /vagrant        none    rw,bind         0       0

Doing it this way, the mounted Vagrant folder will be accessible by the chroot environments with the default vagrant user.

With using sbuild, it is as simple as issuing the following command.

sbuild --arch-all -v -m 'vagrant user ' --arch "amd64" -d "precise" app_*.dsc

Without -c parameter specified, sbuild will search the default folder /etc/sbuild/chroot/ for an environment matching the architecture and distribution.

Getting started with with chroot & mock

Setting up mock is slightly less complicated. With EPEL repo at hand, do…

sudo yum install mock python-ctypes
sudo usermod -G mock vagrant

mock will install a big bunch of chroot configurations residing in /etc/mock folder.

You can clean and initialise the chroot environment each time as you need them. There are also no mounting of folders to fiddle with. Source and result folders are specified as parameters with the mock command.

For example:

mock -v -r epel-6-x86_64--clean
mock -v -r epel-6-x86_64 --init
mock -v -r epel-6-x86_64 --no-clean --no-cleanup-after --buildsrpm \
    --spec “app.spec" \
    --source "$(pwd)/source"  \
    --resultdir "$(pwd)/result"

Vagrant boxes for reuse

Since mock installs a whole bunch of dependencies on the Centos VM, the resulting box will very quickly blow up to around 7.8 GB in size. Likewise, the aforementioned Ubuntu box with two chroot and sbuild will be around 2.2 GB in size. In addition, provisioning these boxes are time consuming. Therefore I highly recommend that once the VMs are provisioned, a new set of Vagrant boxes are to be created and set aside for reuse.

See https://docs.vagrantup.com/v2/virtualbox/boxes.html for details.

Release management, part2: Designing scalable and testable build pipelines with BPMN and Jenkins Job Builder

This blog post aims to illustrate the process of designing build pipelines with Business Process Modeling Notation (BPMN) that is both scalable and testable. The central premise of this blog post is that Jenkins build pipelines are business processes and BPMN closely aligns with Jenkins and its plugin ecosystem. For example, with some clever use of Jenkins plugin, gateways and swimlanes in BPMN can be supported and realised. In the context of release management, Jenkins provides the executable platform for business processes, which benefit tremendously from automation.

In addition, scaling, testing, version controlling and deploying BPMN build pipelines as Jenkins jobs can be achieved through adopting Jenkins API, Git, and Jenkins Job Builder. All Jenkins jobs in the pipeline are written and templated in YAML and deployed via Git (say via an update-hook) when changes are detected. This significantly sped up the process of implementing and stringing together complex Jenkins jobs. This approach allows for early pipeline verification and ultimately enable devops teams to utilise Jenkins a lot more efficiently with greater freedom and control over their own build pipelines.

A quick BPMN primer

Designing build pipelines for release management is akin to doing any other business process modeling. BPMN offers a rich set of visual language to kick start this process. An example of BPMN is as below. See http://tynerblain.com/blog/2006/07/27/bpmn-gateways/ or http://www.bizagi.com/eng/downloads/BPMNbyExample.pdf for more information. Figure below is from the first link as outlined.

BPMN with joins and forks

BPMN with joins and forks

Nodes in BPMN represent events, activities and gates. The edges represent the relationship and the flow. This is fairly simple and offers a rich set of modeling tool to capture a complete business process. In addition, swimlanes represent multiple parallel flows. There is a strong similarity between UML activity diagrams.

Events are the active objects and they captures the start, end and intermediate events (such as timer events). Activities are split into tasks or subprocesses. They are user defined. Finally, gates are predefined set of logical operators designed to control the flow of the tasks. For example, exclusive, parallel, inclusive gateway and etc.

Designing your own build pipelines

In the context of release management, build pipeline refers to a series of test automation, compilation and deployment. These are tasks that are carried out prior to making the product available to the intended audience.

In the previous blog post in the series, https://macyves.wordpress.com/2014/05/20/release-management-part1-test-automation-with-vagrant-and-jenkins-build-pipeline/, test automation made up of the bulk of the release management process in the start. Each test oriented join job triggered multiple parallel downstream jobs. Only upon successful completion of all downstream jobs can the join job proceed to the next. This is modeled by the parallel gateway, which are used to defined forking and joining points in BPMN as below.

Forked unit testing  jobs

Forked unit testing jobs

Jenkins and plugins

While the previous blog post in the series outline the required plugin. No real explanations were given as to why and when to utilise them. There are a couple of key plugins that enables forking and joining, and the locking of common resources; join trigger plugin and throttle concurrent build plugin respectively. For example, the use of join trigger plugin is key to implementing parallel gates.

Once the plugin is installed, using it is as easy as accessing the usual Jenkins UI. The join plugin wiki page contains all the detailed information. https://wiki.jenkins-ci.org/display/JENKINS/Join+Plugin

For more information on the throttle plugin, see https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin

Roll out your build pipelines with Jenkins UI? NO!!

Armed with Jenkins and a complete pipeline design in BPMN, one is ready to start defining Jenkins job in the UI. However, avoiding the UI all together is strongly recommended at this stage. The reason is that despite the convenience supplemented by Jenkins UI, fiddling around with it s both time consuming and error prone. It is most certainly not scalable if you have multiple people working on the same build pipeline model working in a clunky UI environment.

One solution to the problem is to use Jenkins API and define all jobs in XML. However, arguably it is just as clumsy and error prone. The XML model supported by Jenkins maps to its very own internal object models. To utilise it fully one would require a complete understanding of the underpinning relational model used by Jenkins.

Roll out your build pipelines with Git, and Jenkins Job Builder? Yes!

As with all things in software development life cycle, release management processes need to be repeatable, scalable and resilient. The following are the non-functional requirements for the build pipeline.

  • Testability and debugging support on non-production Jenkins setup
  • Tracking changes and rollback
  • Scalability

Enter Jenkins Job Builder. Jenkins Job Builder is a fantastic tool for writing Jenkins job. The supported format is YAML and is human readable. Further more, templating is supported, which translates to common job definitions and job reuse. For example, you could roll out multiple build pipelines based on branches of the source code. Ultimately, you can very easily setup a personalised build pipeline for each developer and their own personal branch with the same set of job templates. The power of Jenkins Job Builder is astounding as it supports a myriad of Jenkins plugin. Moving away from a UI to a text file oriented way of interacting with Jenkins means productivity increases dramatically and that build pipeline skeleton can be tested.

Coupled with Vagrant, using Jenkin Job Builder allows for the same set of Jenkins jobs to be deployed on any Jenkins server. The implication is that in theory debugging Jenkin related issues or test failures can be easily provisioned and reproduced in a VM. This means that anybody on the team can fire up a fully fledged release management platform with almost all the production trimmings. See previous blog entry for the Vagrantfile that is capable of provisioning the exact Jenkins master node required for building and testing the packages. For more information on Jenkins Job Builder, see http://ci.openstack.org/jenkins-job-builder/

Last but not least, integrating Jenkins Job Builder with Git via the update-hook mechanism would enable automated deployment of newly pushed YAML to production Jenkins. Upon failure, one can simply roll back or just discard that commit. Complete freedom and control.

Testability and debugging support on non-production Jenkins setup

Before mapping out all the detailed scripts and various packaging steps, create an overarching build pipeline template. Run it through and see that all the join jobs triggers the correct downstream jobs. This is essentially functionally testing and verification of the build pipeline itself. This template can of course be easily reused and so on. Furthermore, a Vagrantfile would help the developers create their own build server.

Tracking changes and rollback

By utilising Git, one would immediately gain the ability to track changes in the YAML files and rollback if needed. The content of the Git update-hook script is entirely up to you.

Scalability

Write the YAML as templated as possible. Use parameters and variables supported by Jenkins Job Builder. Scaling out different variants of the same build pipeline should be as easy as defining a new “project” or “job-group” in Jenkins Job Builder.

Examples

If you have read this far, you must be wondering if this is all too good to be true. Answer is yes. However, since each build pipeline is very specific to the product and release processes, these samples from OpenStack is a great starting point. https://github.com/openstack-infra/jenkins-job-builder/tree/master/samples