Getting rid of monoliths: #lagom and microservice hooha

The road to ridding application monoliths is one wrought with many obstacles, mostly organisational. Whatever your reason behind this brave move, you have convinced yourself that you will fix this big ball of mud, somehow… You will no longer stand for this shanty town of a spaghetti code, riddled with nonsensical donkey poo. The code no longer speaks to you, it has surpassed the point of having a smell. This abomination has a cyclomatic “flavour” that you can almost taste it at the tip of the tongue. No amount of Ballmer’s peak is going to help. So you are going to make good on that promise to your linting tool, probably Sonar, for whom you have whispered hours of sweet nothings to the tunes of Bob Marley, “Don’t worry ‘bout a thing ’cause every little thing gonna be alright…” What once was a good idea has hideously mutated into a giant walking talking abomination that crawls under your skin and haunts your every step. Under the guise of some sound architectural decisions, you are going to pay down this technical debt, hard. You are going to to do microservice, and you are going to ship Docker containers. If this is you, read on.

The biggest modern day monoliths of them all, J2EE

Tasked with the daunting task of ripping apart functional applications into microservices, you will battle bikeshedders whilst debating your silly asses off with armies of monolith fandom. During what seems to be a very long coup d’état, this attempt to evangelise a newer leaner architecture and approach is riddled with skeptics. “What is frontend without JSF or JSP?! How dare you even question server-push technology like XYZFaces? Then how about the JNDI lookups and EJBs?! Surely you cannot replace these things?”

So six years ago, I had a very pleasant experience of pulling out Glassfish with embedded Jetty, replacing I-Hate-My-Faces with some simple JS (nowadays we call it the MEAN stack, or its many other permutations), and started building APIs with microservice principles. Turns out that is what Spotify did too at the time. So there you go haters. Bottom line, do not use J2EE no matter what if you care about having a competitive advantage. But if you need some good reasons, here are some:

  • Most J2EE containers are grounded on the notion of vertical scalability last I checked. Clustering should be idempotent and stateless, and scaling horizontal.
  • J2EE containers are not cloud native. Just look at their clustering! Unless you feel like having VPNs and private networks across different public clouds or data centers, you can probably just forget it.
  • So let’s put it behind the load balancer? No, most J2EE containers don’t do shared session persistence out of the box.
  • Let’s not kid ourselves, you will customise the crap out of this J2EE container; dropping war files upon war files to fix all its shortcomings.
  • Your sysadmins do not work with war/jar/ear files. They are *nix gurus deserving to be treated like one. Ship your product like an actual product, sir! Apt/yum/brew is your friend and please follow Filesystem Hierarchy Standard (FHS) for goodness sake.

Decide exactly what you are going to piecemeal

Have a very clear idea of the different categories or lifecycles of the applications you are going to transform. Timing is important, as with showing results. Afterall, this is the minimum viable product (MVP) approach naturally. Highly recommend avoiding system of records to begin with, those are definitely not quick wins and you will be in an arduous match, going for the full 5 rounds. Your future competitive advantage does not sit in your ERP or CRM systems. If it does, then um, yeah.

  • Isolate and separate clearly the functionality you are transforming or building
  • Ensure the isolation goes all the way down to infra, this is devops after all
  • Think of how to horizontally scale
  • Think of elasticity
  • Think of shared persistence across network

Java is dead, long live Java!

The trend a few years back was relentlessly hating on Java and Java devs. Evidence shows otherwise. Java is still around and it is going to be around for quite some time to come. No need to switch out it just yet even after the programming nuclear winter imposed by object-orientation and Java/.NET alike. Good code, good ecosystem based on tooling, and good solid design patterns go a long way, regardless of application domain or programming languages.

The truth is the same could have been said about node.js. I recall a number of years back, few colleagues quoted Hacker News regarding the state of node.js and how immature it was and Java was the “preferred” choice, even though the sysadmin community bagged the crap out of Java at the time. If you make strategic decisions predominantly based on the whims of HN, then you are just as plonkers as the next troll. What your node.js/Java boys and girls need to remember:

  • Repeatable, testable build pipelines. Think CI/CD.
  • Coding standards and linting, no brainers there
  • Packaging. Do the right thing, treat sysadmins as your end customers.
  • Separate out load balancing or clustering to other applications like nginx or haproxy. TCP stack makes more sense when written in C.
  • Lord forbid you try to do TLS termination in Java. This is really not cool bro. You got a number of other choices, so do not add this complexity to the landscape. There are no OpenSSL implementation in Java, and OpenSSL is already difficult to maintain as it is.
  • Good monitoring and logging practices goes a long way
  • Think network. Think TCP and HTTP.
  • Your JVM will live on top of a kernel. Know them. Tune that JVM and tune that kernel if needed.

So, microservices. Heard of #lagom?

So you have found a good set of business functional requirements to transform into a set of microservices. Heard of #lagom? Maybe Dropwizard or Springboot? The choices are probably all OK, and when doing microservices, there are simply no bad choices in my opinion. The gains outweigh the means here. The kicker is that there are probably a number of customised endpoint you will need to integrate with. This could be HTTP, something-else-over TCP or whatever. There could also be JPA or other nosql data stores you need to use. Pick your microservice framework component knowing that this is a framework and it can easily grow. The microservice strategy can easily bloat into “milliservice” or “services” (SOA?) if you are not careful. So just how do you stop the size of the code base from expanding? Keep distinct business functionalities as separate services and code bases. The sizing is up to you. Also, split up common functionalities into submodules. Both Dropwizard and Springboot has a bunch. Lagom for example has recently being introduced as the microservice framework for Java, it does have quite a lot of these connectors already in place. For me, I opted to homebrew our own microservice framework for maximum flexibility, ownership, and performance tuning.

Either way, armed with your chosen messiah of a framework, the idea here is to rain down free non-functional requirements across multiple projects and dev teams. Cost leadership for all!

  • Ease of hooking up to modern monitoring tools with a configurable metrics set. JVM memory, vmstat, iostat, CPU, JVM gc, etc etc
  • Ease of pulling out logs into say Influxdb or something.
  • Connectors to DBs should be submoduled and shared for future projects. Polyglot persistence ftw.
  • API documentation is super important. Do not assume your API users know your API, and make a point of doing backward compatibility
  • Follow semver.

Keeping your head above water is number one. Hang in there, the good days will come. Just remember, nothing is new in software since 1970, they just get modern marketing hype. And lastly, your fancy new microservice is legacy as soon as you launch. So please do consider the future generations. Let’s end this cycle of spaghetti monster code through old fashion craftsmanship.

Jenkins, jenkins-job-builder and Docker in 2 steps

So here is a simple example that will help provision a VM with Jenkins, jenkins-job-builder and Docker all in one with Vagrant.

Gone are those embarrassing git commits with those peskey jenkins-job-builder yaml files! Throwing in Docker install in Vagrant for Mac users who shuns boot2docker.

Some bonus stuff like examples of provisioning Jenkins plugin via the cli interface, creating first time Jenkins user are chucked in too.

Check it out!

2-steps are as below

  1. vagrant up
  2. point your browser to http://localhost:38080 and enjoy

and then ..

jenkins-jobs --conf jenkins-job-builder-localhost.ini test your-jobs.yaml -o output/

3 steps really… and for the keen learners wanting some intro material for docker, go here, and here for jenkins-job-builder.

Release management, part1: Test automation with Vagrant and Jenkins build pipeline

This blog post focusses on test automation and is one piece of a larger puzzle involving the craft that is release management.

“Real artists ship.” – Steve Jobs

The products discussed in this blog post is written in Java and Javascript. They are shipped as binaries and installed on premise via repositories. However, prior to building and potentially releasing these packages, test automation executed by the continuous integration (CI) server must pass. This this case, Jenkins. These tests include unit tests, integration tests, configuration tests, performance tests and acceptance tests.

Some of the tests are to be executed on complex setup and at times repeated on permutations varying configurations. Enter Vagrant. Vagrant is used heavily for creating virtualised environments that are cheap and can be easily reproduced. In addition, Vagrant enables the set up of complex network topologies that help facilitate specific test scenarios. The result is that tests are executed on freshly prepped environment and results are reproducible on Jenkins as it is on any developers’ machine.

This journey begins, like all other software journeys, on the drawing board. The problem was that despite having hundreds and thousands of regression tests, executing them independently upon new commits has proven to be tricky, let alone repeating the same test suite over different configurations. Lastly, static code analysis were only performed on developers’ personal setup.

A meeting was swiftly called in and plans for test automation was concocted.

where all ideas grow

where all ideas grow

The major flow of the test automation and build pipeline can be broken down into the following.

  1. src compile
  2. unit tests
  3. integration tests
  4. acceptance tests
  5. configuration tests
  6. performance tests
  7. preview
  8. demo
  9. release (manual step)

Note that each of the build step above is a “join” job, capable of triggering multiple downstream jobs. These downstream jobs can run in parallel, or lock resources on a single Jenkins node if needed. The join job can proceed onto the next join if and only if all triggered jobs are executed completely and successfully. For example, unit tests job will trigger multiple independent jobs for testing data access objects, RESTful APIs, frontend and various other components within the system. Jenkins will only move onwards to integration tests if all unit tests pass.

Installing and configuring Jenkins

This is a stock standard Jenkins setup, with the master compiling and unit testing the src. See the Vagrantfile below for full setup.

Required Jenkins Plugins

  • git plugin
  • build pipeline plugin
  • workspace cleanup plugin
  • envinject plugin
  • join trigger plugin
  • throttle concurrent builds plugin
  • copyartifact plugin
  • instant messaging plugin
  • irc plugin
  • build-timeout plugin

Note at the time of writing, build pipeline plugin had two bugs that made rendering the pipeline visual really unintelligible. fix was available here

Only global configuration are defined for Multi-Project Throttle Categories. This plugin is used as a locking mechanism for shared resources across Jenkins, see

Automated unit tests

Since Jenkin master will compile the source code, it was then decided to also run the unit tests. All test output are in JUnit format and Jenkins is capable of identifying failed tests and report them appropriately.

Recommendation: Ensuring that all tests can be executed with a single command line, and that test execution order, and setup must be idempotent. Docker can be considered at this point to improve test repeatability and scalability, and avoiding the old “works on Jenkins slaves/masters, but we have no idea what has been installed on it” excuse.

Automated integration tests

Similar to running unit tests, but on a setup closer to supported production environment. Vagrant is used extensively to setup and teardown fresh VMs for running these tests.

Recommendation: Making these tests run via a single command is highly recommend. This makes setup with Jenkins easy and reproducing failed test result on a developer’s setup more reliable. Note that these tests can take a while to set up since it involves installing 3rd party packages with apt-get or yum through Vagrant. At the time of writing, performing vagrant up in parallel has proven to be buggy. The throttle plugin is utlised to lock Vagrant as a single Jenkins resource. Note that depending on the test, it may be possible to use Docker instead of Vagrant. In addition, using AWS EC2 Vagrant plugin may provide access to additional resource to conduct these tests where needed.

Automated acceptance tests

At the time of writing, acceptance tests framework are being developed with Selenium and phantom.js. A mechanism to record user clicks and interactions then replaying them to verify the expected result is critical for this job.

Recommendation: Install packaged binaries on VM from intended repository where possible. This way the full installation process is tested (essentially testing the install guide). Furthermore, my view is that HAR output fits RESTful API testing. Selenium should not be ignored as it tests cross-browser support and provides a simple mechanism for recording user interactions.

Automated performance tests

At the time of writing, automated performance tests are not in place yet. However, my personal recommendation is to incorporate yslow and phantom.js into the build pipeline. See

Automated configuration tests

At the time of writing, different dependencies and configurations are being verified manually and not currently automated. It is highly recommended to do so. A provisioning tool such as Puppet, Chef, or Ansible that is capable of generating Vagrantfile based on supported configuration will help provision the environments for these tests. The same acceptance test suite should be repeated on each supported configuration to verify conformity.

Preview & Demo

Make available a freshly built Vagrant box with all dependencies and binaries installed. This is highly recommended of you have vendors or consultants, or just wish to pass on a demo box to prospects. It is also useful for guerrilla user testing.


Release is a manual step to avoid any mishaps. If it was a hosted service, this step should be automated and deployment should occur as new binaries pass through these gauntlet of tests.

Putting it all together

Only thing missing is automated resilience testing and testing for single point of failures. But for now, behold, a sea of green.

sea of green

sea of green

Vagrantfile to reproduce the Jenkins master

# -*- mode: ruby -*-
# vi: set ft=ruby :


Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
	config.vm.define :jenkins do |jenkins| = "jenkins"
		jenkins.vm.box_url = ""

        $script_jenkins = <<SCRIPT
echo ===== Installing tools, git, devscripts, debhelper, checkinstall, curl, rst2pdf, and unzip =====
sudo apt-get update
sudo apt-get install git -y
sudo apt-get install unzip -y
sudo apt-get install devscripts -y
sudo apt-get install debhelper -y
sudo apt-get install checkinstall -y
sudo apt-get install curl -y
sudo apt-get install rst2pdf -y
echo ===== done ====
echo ===== installing ruby1.8 and gems =====
sudo apt-get install ruby1.8 -y
tar xvf rubygems-2.0.7.tgz
cd rubygems-2.0.7/
sudo checkinstall -y ruby setup.rb
echo ===== done ====
echo ===== Installing compass and zurb-foundation =====
sudo gem1.8 install compass
sudo gem1.8 install zurb-foundation
echo ===== done ====
echo ===== Installing compass and zurb-foundation =====
sudo apt-get install openjdk-7-jdk -y
sudo apt-get install ant -y
echo ===== done ====
echo ===== installing jenkins =====
cd ~
wget -q -O - | sudo apt-key add -
echo "deb binary/" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install jenkins -y
echo ===== done ====
echo ===== installing mongodb =====
sudo apt-get install mongodb -y
echo ===== done =====
echo ===== install nodejs and npm via nvm very hackish unless we fix it =====
curl | sh
source /root/.profile
nvm install v0.10.18
nvm use v0.10.18
npm -v
node -v
n=$(which node);n=${n%/bin/node}; chmod -R 755 $n/bin/*; sudo cp -r $n/{bin,lib,share} /usr/local
sudo -s which node
echo ===== done =====
echo ===== Installing grunt-cli =====
sudo npm install -g grunt-cli
echo ===== done ====
echo ===== virtualbox and vagrant =====
cd ~
sudo apt-get install virtualbox
sudo dpkg -i vagrant_1.3.3_x86_64.deb
echo ===== done =====
echo ===== install jenkins-jobs ===
cd ~
git clone
cd jenkins-job-builder/
sudo python install
echo ===== done =====

		jenkins.vm.provision :shell, :inline => $script_jenkins "forwarded_port", guest: 8080, host:38080