Very grateful to be invited to speak in the International Forecourt Standards Forum (IFSF) Innovation and Collaboration Conf 2017 in Paris this year 🙂 If you are in town, swing by or hit me up on twitter, and let’s talk #docker #microservices #apis and #devops!
*Speaker Announcement* @yveshwang will be joining us in #Paris to discuss micro-service architecture #IFSFconf2017 https://t.co/3S3uIMO9TG pic.twitter.com/ufKR7aTNsb
— IFSF (@IFSFonline) August 31, 2017
In a short span of time, companies will no longer own or probably never buy new hardware. Gone are the days of PXE booting racks and racks of the stuff. These classical and mysterious oompa loompas are being replaced by automation and tooling. Hardware vendors are cannibalising and consolidating their business all at the same time. While skeptics remain, haters are always going to hate. Then Kubernetes flew in. Like a moth to a flame I was completely drawn at the time. Alas, just scratching the surface lies a giant of a community, oozing with great ideas at an amazing pace. Armed with my own fanboy enthusiasm, I ventured into my first #kubecon this year. One thing for sure, I cannot wait to move stuff to CoreOS and rkt!
These days projects are executed by highly agile guru devops fullstack teams, and we are shipping containers. Containers are a fun new way to package software, as devs are always keen to rewrite SysV init or system.d. I too am guilty of this in the past, at times mustering the courage to create these unique snowflakes rainbow-unicorn of a bash script. Yes, it appears that the dark days are back… maybe… and some of us might be enjoying this a little too much 😀
"docker-entrypoint.sh is init.d scripts for hipsters." @kubernetesio pic.twitter.com/fTC66OH4Eo
— Yves Hwang (@yveshwang) March 10, 2016
Containerisation is the essence of devops as it breaks down the barriers between sysadmins and devs. However Kubernetes does not solve the fact that we are still maintaining or are currently churning out shitty applications despite the improved process around it. The usual remedy is by hammering down on non-functional requirements to level out most of the technical debt. Naturally Kubernetes does not give you free non-functional requirements. Truth is, you never got non-functional requirements for free. This was the clear dividing line between craftsmanship or sipping soup from the big ball of mud. What Kubernetes does give you is elasticity, optimisation and scalability.
Self serviced infra is cool. It was a common theme throughout #kubecon. Having Kubernetes as an integral part of the development process will bring about the next level devops. Empowering these team seems to be the magic sauce and I wholeheartedly agree. Finally we can ditch the last of Elastic Beanstalk.
The biggest validating moment for me was centered around enterprise transformation, based around automation, build pipelines and ChatOps. I knew my strategy was awesome, but this completely made my day. We have in place all three within a short time and we are not alone in this hipster movement! Not to mention we are bloody good at this too! Right up there with the best. A proud moment as I quietly gave myself a pat on the back, swiftly jumped on Slack, shared the good news and congratulated everybody on the team.
#kubecon this year for me was just an amazing experience. It has been a while since I have attended a conference for the sole purpose to just soak up the knowledge from the superheros that I look up to. It was nerdinus maximus, happy hour all day everyday, and I can’t wait to do it all over again.
So here is a simple example that will help provision a VM with Jenkins, jenkins-job-builder and Docker all in one with Vagrant.
Gone are those embarrassing git commits with those peskey jenkins-job-builder yaml files! Throwing in Docker install in Vagrant for Mac users who shuns boot2docker.
Some bonus stuff like examples of provisioning Jenkins plugin via the cli interface, creating first time Jenkins user are chucked in too.
Check it out! https://github.com/yveshwang/jenkins-docker-2step
vagrant upand then ..
jenkins-jobs --conf jenkins-job-builder-localhost.ini test your-jobs.yaml -o output/
3 steps really… and for the keen learners wanting some intro material for docker, go here, and here for jenkins-job-builder.
This blog post aims to illustrate how Docker is introduced into an existing Jenkins build pipeline for the purpose of performing test automation in parallel whilst guaranteeing isolated and idempotent environments that are scalable.
In short, this is a truly exciting example of real world devops tool chaining with Docker as part of a build pipeline. Needlessly to say, by incorporating Docker into our build pipeline, it has has trimmed at least 70% of the build pipeline completion time by enabling lockless and parallel job execution. Since each jobs can run in parallel with Docker guaranteeing an isolated, idempotent environment for test automation, scaling out the build pipeline would be as simple as adding more Jenkins slaves. With clever use of Docker, Vagrant, Git and Jenkins together, this kind of tool chaining provides Platform as a Service (PaaS) that empowers both developers and operations alike, a shining example of devops.
In the first part of the Release Management blog series, the design of the build pipeline was conceived and outlined. Verification of the build pipeline design was covered in part two of the series. Permeating throughout this series is testability and automation. In addressing the correctness of test results and outcomes, the test execution environment must be isolated and test execution idempotent.
Ensuring a clean slate and running on a controlled environment when executing tests is paramount to the correctness of the test results. The first blog post discussed Vagrant as the tool for creating VMs, then running said tests on these VMs. Each job would perform vagrant up with a number of vagrant ssh command with specific -c parameter, then vagrant destroy for tear down. These commands guaranteed an idempotent and an isolated environment for each test execution in a Jenkins job. However there was a catch. The command, vagrant up, cannot be executed in parallel. See this discussion. This meant jobs that utilised Vagrant must be specified as a protected resource, accessible by one job, one slave at a time. This locking mechanism is supported by the Concurrent Throttle Builds Jenkins plugin, where jobs will wait and queue for certain resource to be freed before execution. The kicker is that this plugin is broken since October 2013 for Jenkins 1.536 and Concurrent Throttle 1.8. See Jenkins issue #19986 and #20211. Suffice to say, as of October last year, the build pipeline ran on a single Jenkins executor, taking a full hour to complete one pipeline build.
Whilst Vagrant definitely offers isolated and idempotent environments, at the time of writing, firing up parallel Vagrant boxes with VirtualBox was buggy at best. It is later confirmed that VirtualBox as a provider does not support parallel provisioning in Vagrant. Naturally if AWS was utilised as a Vagrant provider, this would be non-issue. Alas, it was not the case.
Secondly, provisioning each box takes a long time. Vagrant startup time are in minutes and increases significantly when compounded by copious amounts of apt-get or yum installs during provisioning. Although this can be mitigated by having pre-built Vagrant boxes with all dependencies installed, but it does become an added maintenance task.
The solution here is Docker. Unlike Vagrant, Docker creates lightweight containers, akin to LXC. Each containers can be pulled and pushed. This is similar to the Git model, with docker.io offering a centralised repository of all Docker images. With the recent Docker 1.0 release, it is ready for prime time and production usage. Indeed, using Docker is quick and easy. Create the necessary Dockerfile with the same provisioning steps as Vagrantfile is a breeze. When the same user as the Jenkins slave is included in the Dockerfile, upon successfully building the image, the Jenkins slave can then mount any directory onto this container and have the same user as Jenkins slave writing to that mounted directory from the container. This is central for reporting back test results.
Docker images, which act as a baseline for a container, provides an idempotent and isolated environment with the major benefit of being able to run containers in parallel, even if they are based on the same image.
The biggest question is does Docker completely replace Vagrant? The answer is a very clear no. Vagrant is extremely useful during development, essentially giving developers the ability to run as many varied platforms as possible. Docker is a container, an isolated kernel namespace running on top of the a shared Linux kernel. The difference is minute, but depending on the testing requirements, when VMs are required, Vagrant is still a valid choice over Docker.
Having Vagrantfile and Dockerfile in their own repo is clever. With Git and Jenkins, provisioning updated sets of Vagrant boxes or Docker containers from Vagrantfile and Dockerfile respectively is easy and will give you the ability to track changes in these VMs and containers. When changes are detected, Vagrant boxes can be rebuilt and list of available boxes updated. Likewise, Docker containers can be rebuilt and pushed as the latest tag. This will both enable and empower the development team to maintain their own production VMs and containers for development, testing and releases. This is PaaS with liberal sprinkling of devops goodness.
There are bucket loads of fantastic Docker related blog posts out there. This is yet another tldr guide to get the readers over the hump, particularly illustrating how you can mount local file systems onto a Docker container with Vagrant.
Before we begin, Docker is a client/daemon application. Docker daemon can listen to commands from the Docker client. The client is thin and all containers are hosted on the daemon. This means if you point Docker client to another daemon endpoint, the usual available list of images and containers will change. This is a good thing.
As stated on the official Docker installation guide, install via Homebrew is highly recomended.
brew install boot2docker brew install docker
Note that brew only installs the Docker client on OSX. boot2docker is the VirtualBox wrapper that will fire up Docker daemon. This is important as at the time of writing, commands performed on Docker clients are executed and takes place on the Docker daemon. For example, mounting files to a container really refers to mounting files from the server that the Docker daemon is running to the desired container. This is a cause of many confusion.
The installation guide is really good, and here is a tldr version. Note that the following script uses a different boot2docker port. This is to avoid port conflict with the Docker daemon via Vagrant later.
echo “export DOCKER_PORT=14243” >> ~/.boot2docker/profile boot2docker init boot2docker start export DOCKER_HOST=tcp://localhost:14243 docker version
If all goes well, doing docker version will yield the following.
docker_version. all good
The steps are based on John Zanchetta’s blog post, with all of the Vagrant steps trimmed and one network routing step found redundant. This is the tldr version and it assumes you are comfortable with Vagrant
Step 1 – Provision the Docker daemon box
Using the following Vagrant file, fire up the VM called docker
# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.define :docker do |docker|
docker.vm.box = "precise64"
docker.vm.box_url = "http://files.vagrantup.com/precise64.box"
docker.vm.network "forwarded_port", guest: 80, host:58080
docker.vm.network "forwarded_port", guest: 4243, host: 4243
$script = <<SCRIPT
wget -q -O - https://get.docker.io/gpg | apt-key add -
echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list
apt-get update -qq
apt-get install -q -y --force-yes lxc-docker
usermod -a -G docker vagrant
sed -e 's/DOCKER_OPTS=/DOCKER_OPTS=\"-H 0.0.0.0:4243\"/g' /etc/init/docker.conf > /vagrant/docker.conf.sed
cp /vagrant/docker.conf.sed /etc/init/docker.conf
rm -f /vagrant/docker.conf.sed
service docker restart
SCRIPT
docker.vm.provision :shell, :inline => $script
end
end
Step 2 – Vagrant or boot2docker, an easy switch
Point the DOCKER_HOST to the Vagrant-based Docker daemon. Note that by simply changing the DOCKER_HOST, you can point your Docker client to whichever Docker daemon you so desire; boot2docker or Vagrant-based for example.
export DOCKER_HOST=tcp://localhost:4243 docker version
If all goes well, you are ready to build!
Step 3 – Build your first Docker container and mount volumes in Docker with Vagrant
Once you have a container built, mounting the /vagrant is no different from mounting any other directories to the container from the Docker daemon. For example
docker run -t -i --name test --volume /vagrant:/vagrant:rw example:test
Easy!
Step 4 – Make your first push, my only advice
This blog post would not be complete without recommending you to perform your first Docker push.
If like me, you have created your container with “example” as the repository, and “test” as the tag based on a Dockerfile
docker build -t example:test .
And upon pushing (assuming the username is “yves”), for example
docker login docker push yves/example
You will notice the following response:
The push refers to a repository [yves/example] (len: 0) 2014/05/31 00:46:35 No such id: yves/example
This is because the push command needs username, which has to be part of the repository name. However, the above example, the repository name is merely just “example”.
The work around here, based on scouring issues #720, is to create a new tag that includes the username as part of the repository name. For example:
docker tag yves/example:test_tag docker push yves/example
then watch the magic unfold in front of your very eyes.
The big take away is to always prefix your repository name with your docker.io username, followed by a /. For example, yves/ubuntu, yves/centos yves/whatever.
My docker image, yves/yves, is built based on the Vagrant file mentioned in https://macyves.wordpress.com/2014/05/20/release-management-part1-test-automation-with-vagrant-and-jenkins-build-pipeline/
At the time of writing, mounting directories from your OSX machine to a container hosted by boot2docker is quite a mind field. See issues #3957. boot2docker is very well supported, but I feel more at ease with Vagrant.
Docker is really resilient. Simply retry the same command when encountering failures during image building or pushing to docker.io. This is in part thanks to the untagged images that floats around Docker daemon and that an image is really a repository containing a bunch of changesets performed, akin to Git.
However, if you wish to clean up all untagged images, see this blog post.
This blog post focusses on test automation and is one piece of a larger puzzle involving the craft that is release management.
“Real artists ship.” – Steve Jobs
The products discussed in this blog post is written in Java and Javascript. They are shipped as binaries and installed on premise via repositories. However, prior to building and potentially releasing these packages, test automation executed by the continuous integration (CI) server must pass. This this case, Jenkins. These tests include unit tests, integration tests, configuration tests, performance tests and acceptance tests.
Some of the tests are to be executed on complex setup and at times repeated on permutations varying configurations. Enter Vagrant. Vagrant is used heavily for creating virtualised environments that are cheap and can be easily reproduced. In addition, Vagrant enables the set up of complex network topologies that help facilitate specific test scenarios. The result is that tests are executed on freshly prepped environment and results are reproducible on Jenkins as it is on any developers’ machine.
This journey begins, like all other software journeys, on the drawing board. The problem was that despite having hundreds and thousands of regression tests, executing them independently upon new commits has proven to be tricky, let alone repeating the same test suite over different configurations. Lastly, static code analysis were only performed on developers’ personal setup.
A meeting was swiftly called in and plans for test automation was concocted.
where all ideas grow
The major flow of the test automation and build pipeline can be broken down into the following.
Note that each of the build step above is a “join” job, capable of triggering multiple downstream jobs. These downstream jobs can run in parallel, or lock resources on a single Jenkins node if needed. The join job can proceed onto the next join if and only if all triggered jobs are executed completely and successfully. For example, unit tests job will trigger multiple independent jobs for testing data access objects, RESTful APIs, frontend and various other components within the system. Jenkins will only move onwards to integration tests if all unit tests pass.
This is a stock standard Jenkins setup, with the master compiling and unit testing the src. See the Vagrantfile below for full setup.
Required Jenkins Plugins
Note at the time of writing, build pipeline plugin had two bugs that made rendering the pipeline visual really unintelligible. fix was available here https://code.google.com/r/jessica-join-support-build-pipeline-plugin/
Only global configuration are defined for Multi-Project Throttle Categories. This plugin is used as a locking mechanism for shared resources across Jenkins, see https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin
Since Jenkin master will compile the source code, it was then decided to also run the unit tests. All test output are in JUnit format and Jenkins is capable of identifying failed tests and report them appropriately.
Recommendation: Ensuring that all tests can be executed with a single command line, and that test execution order, and setup must be idempotent. Docker can be considered at this point to improve test repeatability and scalability, and avoiding the old “works on Jenkins slaves/masters, but we have no idea what has been installed on it” excuse.
Similar to running unit tests, but on a setup closer to supported production environment. Vagrant is used extensively to setup and teardown fresh VMs for running these tests.
Recommendation: Making these tests run via a single command is highly recommend. This makes setup with Jenkins easy and reproducing failed test result on a developer’s setup more reliable. Note that these tests can take a while to set up since it involves installing 3rd party packages with apt-get or yum through Vagrant. At the time of writing, performing vagrant up in parallel has proven to be buggy. The throttle plugin is utlised to lock Vagrant as a single Jenkins resource. Note that depending on the test, it may be possible to use Docker instead of Vagrant. In addition, using AWS EC2 Vagrant plugin may provide access to additional resource to conduct these tests where needed.
At the time of writing, acceptance tests framework are being developed with Selenium and phantom.js. A mechanism to record user clicks and interactions then replaying them to verify the expected result is critical for this job.
Recommendation: Install packaged binaries on VM from intended repository where possible. This way the full installation process is tested (essentially testing the install guide). Furthermore, my view is that HAR output fits RESTful API testing. Selenium should not be ignored as it tests cross-browser support and provides a simple mechanism for recording user interactions.
At the time of writing, automated performance tests are not in place yet. However, my personal recommendation is to incorporate yslow and phantom.js into the build pipeline. See http://yslow.org/phantomjs/
At the time of writing, different dependencies and configurations are being verified manually and not currently automated. It is highly recommended to do so. A provisioning tool such as Puppet, Chef, or Ansible that is capable of generating Vagrantfile based on supported configuration will help provision the environments for these tests. The same acceptance test suite should be repeated on each supported configuration to verify conformity.
Make available a freshly built Vagrant box with all dependencies and binaries installed. This is highly recommended of you have vendors or consultants, or just wish to pass on a demo box to prospects. It is also useful for guerrilla user testing.
Release is a manual step to avoid any mishaps. If it was a hosted service, this step should be automated and deployment should occur as new binaries pass through these gauntlet of tests.
Only thing missing is automated resilience testing and testing for single point of failures. But for now, behold, a sea of green.
sea of green
# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.define :jenkins do |jenkins|
jenkins.vm.box = "jenkins"
jenkins.vm.box_url = "http://files.vagrantup.com/precise64.box"
$script_jenkins = <<SCRIPT
echo ===== Installing tools, git, devscripts, debhelper, checkinstall, curl, rst2pdf, and unzip =====
sudo apt-get update
sudo apt-get install git -y
sudo apt-get install unzip -y
sudo apt-get install devscripts -y
sudo apt-get install debhelper -y
sudo apt-get install checkinstall -y
sudo apt-get install curl -y
sudo apt-get install rst2pdf -y
echo ===== done ====
echo ===== installing ruby1.8 and gems =====
sudo apt-get install ruby1.8 -y
wget http://production.cf.rubygems.org/rubygems/rubygems-2.0.7.tgz
tar xvf rubygems-2.0.7.tgz
cd rubygems-2.0.7/
sudo checkinstall -y ruby setup.rb
echo ===== done ====
echo ===== Installing compass and zurb-foundation =====
sudo gem1.8 install compass
sudo gem1.8 install zurb-foundation
echo ===== done ====
echo ===== Installing compass and zurb-foundation =====
sudo apt-get install openjdk-7-jdk -y
sudo apt-get install ant -y
echo ===== done ====
echo ===== installing jenkins =====
cd ~
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
echo "deb http://pkg.jenkins-ci.org/debian binary/" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install jenkins -y
echo ===== done ====
echo ===== installing mongodb =====
sudo apt-get install mongodb -y
echo ===== done =====
echo ===== install nodejs and npm via nvm very hackish unless we fix it =====
curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | sh
source /root/.profile
nvm install v0.10.18
nvm use v0.10.18
npm -v
node -v
n=$(which node);n=${n%/bin/node}; chmod -R 755 $n/bin/*; sudo cp -r $n/{bin,lib,share} /usr/local
sudo -s which node
echo ===== done =====
echo ===== Installing grunt-cli =====
sudo npm install -g grunt-cli
echo ===== done ====
echo ===== virtualbox and vagrant =====
cd ~
sudo apt-get install virtualbox
wget http://files.vagrantup.com/packages/db8e7a9c79b23264da129f55cf8569167fc22415/vagrant_1.3.3_x86_64.deb
sudo dpkg -i vagrant_1.3.3_x86_64.deb
echo ===== done =====
echo ===== install jenkins-jobs ===
cd ~
git clone https://github.com/openstack-infra/jenkins-job-builder
cd jenkins-job-builder/
sudo python setup.py install
echo ===== done =====
SCRIPT
jenkins.vm.provision :shell, :inline => $script_jenkins
jenkins.vm.network "forwarded_port", guest: 8080, host:38080
end
end
something else 