What #WoW has taught me about building a better #devops team in an enterprise

One does not need to look any further than World of Warcraft (WoW) to appreciate the beautifully enacted artistry that is teamwork. And it is exactly what an enterprise I.T. needs. Picture a cross functional and multidisciplinary team of rockstars working towards the same goal. This blog is about drawing a comparison between staffing up a devops team for product development in the same manner as running a team through an instance in WoW, dispelling any F.U.D. along the way, and getting that epic win for the enterprise.

Why gamers, why WoW?

Suffice to say as a developer, one spends countless hours in front of the PC. It is fair to say that some of us would at times play games together, or WoW to be exact. As Jane McGonigal @ TED 2010 spelled it out for us, that gamers are people whom has picked up the skill of solving extremely difficult problems in an collaborative and multidisciplinary setting. They are self motivated and extremely committed. The rationale can be mapped to the addictive feeling of an epic win. Achieving the insurmountable or being constantly on verge of an epic win is something quite familiar in the context of software development. I would argue that a typical software craftsman share a number of qualities with a successful WoW player:

  • continuous learning
  • application and adaptation of the knowledge they have gained throughout the years
  • mastery of the tools they use
  • deep understanding of the ecosystem they are working with
  • pride in their work and being extremely mindful towards quality

Why software craftsman matters? Because quite frankly, most business are going to be software houses. Digitisation is about doing things right by I.T. and your nerds are now your core business.

Gamers always believe that an epic win is possible, and that it’s always worth trying, and trying now. Gamers don’t sit around. Gamers are virtuosos at weaving a tight social fabric.
Jane McGonigal, Feb 2010 @ TED

Epic wins

The trick here is to make our daily work meaningful and as important as any other world saving quests. Provided time is not spent in meetings or corporate risk management theatre, I would argue most of what we do conjures up the same feeling of achievement. For example:

  • finding that extremely annoying bug and fixing it
  • putting together a super slick build automation pipeline
  • creating that killer UI, designing that awesome architecture then building it
  • ChatOps
  • deploying something automagically 20 to 50 times a day, and etc

These activities are all mini epic wins for me personally. We are your hipster superheroes after all.

This is a load of piss take. How about PCI-DSS, SOX, ITIL or COBIT?

If you are distracted by PCI-DSS, SOX and the like, start writing some code and run a few instances in WoW. Experience the magic first hand. And don’t forget to read this and this during your down time. Get over the smoke screens and get back on track with the epic wins.

“And this argument – that collaboration between silos, or even cross-functional teams, is forbidden by regulation or “best practice” – is an example of what we in the consulting industry call a bullshit smokescreen.”
Jez Humble, 19 October 2012

OK I’m onboard! Let’s go!

Assuming you are being innovative and building things from scratch, typically in WoW you need 1x tank, 1x healer, 3x DPS (Damage per second). Translating to software terms, you will need 1x tech lead, 3x fullstack devs, and 1x sysadmin. A 5-persons multidisciplinary team goes a long way. Let me break this down.

Tank/Tech lead: In WoW terms, your role is to take all direct damage for the team and be the leader of the pact. You draw and keep the attention of the epic boss to youself. You shield the team from a complete wipe. In software terms, you are a SCRUM master or the spiritual tech leader of the team. You take the unnecessary meetings and shield the team away from distractions. You might code a little or contribute to testing and test automation, or you might be a UX specialist. One of your main task is to free up time and space to enable your rockstar coders to code; aka DPS down that epic boss.

DPS/Rockstar coders: In WoW terms, you deal damage, hard and fast and consistently over a period of time. You are much more squishier than the tanks and you should not pull aggro to yourselves and burden the healer. In software terms, you are the rockstar coder with minimal meetings booked in your calendar. You churn out code, good quality code of course. Any rubbish spaghetti code is considered “drawing aggro” and will probably piss off your resident healer and tank. I would naturally recommend fullstack developers for all 3 roles with specific focus/spec on API/backend, frontend/UI, and automation.

Healer/Sysadmin: In WoW terms, you heal the team members, but mostly the tank. You do little damage yourself but are fully capable of keeping the entire team alive throughout the instance. In software terms, you are the sysadmin. You deal with the technical debt directly produced by the team members. You try to ship their work even at times knowing that they are produced way passed their Ballmer’s peak. You try to tune and configure the software to keep it alive. You care about performance and high availability. A healer is arguably the most critical role in the team. You must be more devops than the rest of the devops boys and girls, if you get my drift.

But an enterprise is more like a 40-men Onyxia raid back in vanilla days?!

This is absolutely true. Depending on the goal at hand, coming up with a functional governance model to work with 40 rockstars will be difficult. At a much larger scale, it is more about culture than anything else in my opinion. Your guild and guild master will be important, as their insight and coordination plays a critical role in the success. However, the fundamentals are the same. You still need small agile individual tribes or teams to perform certain tasks.

In summary: teamwork

It is fairly obvious to see that you absolutely require a diverse and cross functional team. No single role alone can build a winning product, or get that epic win, even for Linus Torvald.

“I can’t do UI to save my life. I mean if I was stranded on an island and the only way to get off that island was to make a pretty UI, I’d die there.”
Linux Torvald, 13 April 2016 @ TED

At the end of the day, working together is the essence of devops. In the meantime, I had a great time writing this post 😀 Big it up to the Avengers, our residence craftsman guild.

❤ Cap-A

Advertisements

Getting rid of monoliths: #lagom and microservice hooha

The road to ridding application monoliths is one wrought with many obstacles, mostly organisational. Whatever your reason behind this brave move, you have convinced yourself that you will fix this big ball of mud, somehow… You will no longer stand for this shanty town of a spaghetti code, riddled with nonsensical donkey poo. The code no longer speaks to you, it has surpassed the point of having a smell. This abomination has a cyclomatic “flavour” that you can almost taste it at the tip of the tongue. No amount of Ballmer’s peak is going to help. So you are going to make good on that promise to your linting tool, probably Sonar, for whom you have whispered hours of sweet nothings to the tunes of Bob Marley, “Don’t worry ‘bout a thing ’cause every little thing gonna be alright…” What once was a good idea has hideously mutated into a giant walking talking abomination that crawls under your skin and haunts your every step. Under the guise of some sound architectural decisions, you are going to pay down this technical debt, hard. You are going to to do microservice, and you are going to ship Docker containers. If this is you, read on.

The biggest modern day monoliths of them all, J2EE

Tasked with the daunting task of ripping apart functional applications into microservices, you will battle bikeshedders whilst debating your silly asses off with armies of monolith fandom. During what seems to be a very long coup d’état, this attempt to evangelise a newer leaner architecture and approach is riddled with skeptics. “What is frontend without JSF or JSP?! How dare you even question server-push technology like XYZFaces? Then how about the JNDI lookups and EJBs?! Surely you cannot replace these things?”

So six years ago, I had a very pleasant experience of pulling out Glassfish with embedded Jetty, replacing I-Hate-My-Faces with some simple JS (nowadays we call it the MEAN stack, or its many other permutations), and started building APIs with microservice principles. Turns out that is what Spotify did too at the time. So there you go haters. Bottom line, do not use J2EE no matter what if you care about having a competitive advantage. But if you need some good reasons, here are some:

  • Most J2EE containers are grounded on the notion of vertical scalability last I checked. Clustering should be idempotent and stateless, and scaling horizontal.
  • J2EE containers are not cloud native. Just look at their clustering! Unless you feel like having VPNs and private networks across different public clouds or data centers, you can probably just forget it.
  • So let’s put it behind the load balancer? No, most J2EE containers don’t do shared session persistence out of the box.
  • Let’s not kid ourselves, you will customise the crap out of this J2EE container; dropping war files upon war files to fix all its shortcomings.
  • Your sysadmins do not work with war/jar/ear files. They are *nix gurus deserving to be treated like one. Ship your product like an actual product, sir! Apt/yum/brew is your friend and please follow Filesystem Hierarchy Standard (FHS) for goodness sake.

Decide exactly what you are going to piecemeal

Have a very clear idea of the different categories or lifecycles of the applications you are going to transform. Timing is important, as with showing results. Afterall, this is the minimum viable product (MVP) approach naturally. Highly recommend avoiding system of records to begin with, those are definitely not quick wins and you will be in an arduous match, going for the full 5 rounds. Your future competitive advantage does not sit in your ERP or CRM systems. If it does, then um, yeah.

  • Isolate and separate clearly the functionality you are transforming or building
  • Ensure the isolation goes all the way down to infra, this is devops after all
  • Think of how to horizontally scale
  • Think of elasticity
  • Think of shared persistence across network

Java is dead, long live Java!

The trend a few years back was relentlessly hating on Java and Java devs. Evidence shows otherwise. Java is still around and it is going to be around for quite some time to come. No need to switch out it just yet even after the programming nuclear winter imposed by object-orientation and Java/.NET alike. Good code, good ecosystem based on tooling, and good solid design patterns go a long way, regardless of application domain or programming languages.

The truth is the same could have been said about node.js. I recall a number of years back, few colleagues quoted Hacker News regarding the state of node.js and how immature it was and Java was the “preferred” choice, even though the sysadmin community bagged the crap out of Java at the time. If you make strategic decisions predominantly based on the whims of HN, then you are just as plonkers as the next troll. What your node.js/Java boys and girls need to remember:

  • Repeatable, testable build pipelines. Think CI/CD.
  • Coding standards and linting, no brainers there
  • Packaging. Do the right thing, treat sysadmins as your end customers.
  • Separate out load balancing or clustering to other applications like nginx or haproxy. TCP stack makes more sense when written in C.
  • Lord forbid you try to do TLS termination in Java. This is really not cool bro. You got a number of other choices, so do not add this complexity to the landscape. There are no OpenSSL implementation in Java, and OpenSSL is already difficult to maintain as it is.
  • Good monitoring and logging practices goes a long way
  • Think network. Think TCP and HTTP.
  • Your JVM will live on top of a kernel. Know them. Tune that JVM and tune that kernel if needed.

So, microservices. Heard of #lagom?

So you have found a good set of business functional requirements to transform into a set of microservices. Heard of #lagom? Maybe Dropwizard or Springboot? The choices are probably all OK, and when doing microservices, there are simply no bad choices in my opinion. The gains outweigh the means here. The kicker is that there are probably a number of customised endpoint you will need to integrate with. This could be HTTP, something-else-over TCP or whatever. There could also be JPA or other nosql data stores you need to use. Pick your microservice framework component knowing that this is a framework and it can easily grow. The microservice strategy can easily bloat into “milliservice” or “services” (SOA?) if you are not careful. So just how do you stop the size of the code base from expanding? Keep distinct business functionalities as separate services and code bases. The sizing is up to you. Also, split up common functionalities into submodules. Both Dropwizard and Springboot has a bunch. Lagom for example has recently being introduced as the microservice framework for Java, it does have quite a lot of these connectors already in place. For me, I opted to homebrew our own microservice framework for maximum flexibility, ownership, and performance tuning.

Either way, armed with your chosen messiah of a framework, the idea here is to rain down free non-functional requirements across multiple projects and dev teams. Cost leadership for all!

  • Ease of hooking up to modern monitoring tools with a configurable metrics set. JVM memory, vmstat, iostat, CPU, JVM gc, etc etc
  • Ease of pulling out logs into say Influxdb or something.
  • Connectors to DBs should be submoduled and shared for future projects. Polyglot persistence ftw.
  • API documentation is super important. Do not assume your API users know your API, and make a point of doing backward compatibility
  • Follow semver.

Keeping your head above water is number one. Hang in there, the good days will come. Just remember, nothing is new in software since 1970, they just get modern marketing hype. And lastly, your fancy new microservice is legacy as soon as you launch. So please do consider the future generations. Let’s end this cycle of spaghetti monster code through old fashion craftsmanship.

#kubecon ’16 for me: inspiration, community, and strategic validation

In a short span of time, companies will no longer own or probably never buy new hardware. Gone are the days of PXE booting racks and racks of the stuff. These classical and mysterious oompa loompas are being replaced by automation and tooling. Hardware vendors are cannibalising and consolidating their business all at the same time. While skeptics remain, haters are always going to hate. Then Kubernetes flew in. Like a moth to a flame I was completely drawn at the time. Alas, just scratching the surface lies a giant of a community, oozing with great ideas at an amazing pace. Armed with my own fanboy enthusiasm, I ventured into my first #kubecon this year. One thing for sure, I cannot wait to move stuff to CoreOS and rkt!

These days projects are executed by highly agile guru devops fullstack teams, and we are shipping containers. Containers are a fun new way to package software, as devs are always keen to rewrite SysV init or system.d. I too am guilty of this in the past, at times mustering the courage to create these unique snowflakes rainbow-unicorn of a bash script. Yes, it appears that the dark days are back… maybe… and some of us might be enjoying this a little too much 😀

Containerisation is the essence of devops as it breaks down the barriers between sysadmins and devs. However Kubernetes does not solve the fact that we are still maintaining or are currently churning out shitty applications despite the improved process around it. The usual remedy is by hammering down on non-functional requirements to level out most of the technical debt. Naturally Kubernetes does not give you free non-functional requirements. Truth is, you never got non-functional requirements for free. This was the clear dividing line between craftsmanship or sipping soup from the big ball of mud. What Kubernetes does give you is elasticity, optimisation and scalability.

Self serviced infra is cool. It was a common theme throughout #kubecon. Having Kubernetes as an integral part of the development process will bring about the next level devops. Empowering these team seems to be the magic sauce and I wholeheartedly agree. Finally we can ditch the last of Elastic Beanstalk.

The biggest validating moment for me was centered around enterprise transformation, based around automation, build pipelines and ChatOps. I knew my strategy was awesome, but this completely made my day. We have in place all three within a short time and we are not alone in this hipster movement! Not to mention we are bloody good at this too! Right up there with the best. A proud moment as I quietly gave myself a pat on the back, swiftly jumped on Slack, shared the good news and congratulated everybody on the team.

#kubecon this year for me was just an amazing experience. It has been a while since I have attended a conference for the sole purpose to just soak up the knowledge from the superheros that I look up to. It was nerdinus maximus, happy hour all day everyday, and I can’t wait to do it all over again.

Yay! My first “enterprisey” blog post!

Not that the world needs any more “enterprisey” blogging but I thought I would throw in my two cents in this arena anyway as it is a new genre for me. Consider this my feeble attempt to disrupt Gartner from the perspective of the t-shirt brigade 😀 And thus I have started this new blog series aptly labeled “enterprisey.”

This is a slightly more architectural and strategic take than my usual blogging style. Naturally this is entirely my own opinion and nobody else’s.

For those that is already part this hipster movement and need a monolith-battle-hardened sparring partner, or if you are keen to jump on for the sake of carrying out tactics such as devops, microservices or polyglot persistence, then stay tuned and watch this space!

Microservice API on Elastic Beanstalk with Jetty, Jersey, Guice, and Mongodb

This blog aims to outline how one can very easily ship microservice APIs on Elastic Beanstalk with Jetty, Jersey, Guice and Mongodb. This piece of work written in Java was an inspiration over a weekend, where I finally found a coherent red thread based on some of the work I did in the past. So if you are reading this, be fully aware that this stack is no longer cool and hip since the conception of RxJava and Akka 😀 http://blog.circleci.com/its-the-future/

Code lives here. https://github.com/yveshwang/api-example

Why Elastic Beanstalk

Platform as a service (PaaS) is rather nice. A little similar to Heroku, Elastic Beanstalk (EB) environment can be tailored to run Docker containers amongst other things. Where EB truly shine is that the platform is accessible and completely configurable as if it was IaaS. This is made possible by ebextensions. For example, each EB instance by default ship with a local Nginx instance as designed by AWS. If you really want to switch it out, in theory you can do so by hijacking the initialisation process through ebextensions, uninstall Nginx, and install something like Varnish instead 😀 Yep, you can well and truly extend and tweak the platform to your heart’s content and potentially break everything.

Shipping to Elastic Beanstalk is fairly straight forward as it mostly involves a Dockerfile or a Docker image hosted in a public or private docker.io repository. If you have a Dockerfile or a Docker image, and a bit savvy with the eb cli tool, you are sorted and ready to deploy to Elastic Beanstalk.

Having a Docker container also makes testing repeatable and standardised. See the previous build pipeline blog series. It is by intention that the infrastructure part of the setting up eb is left out of this post for now as I believe Elastic Beanstalk deserves a blog post of its own. So let’s keep it old school and talk UML and code a little bit in this blog post.

Why Jetty

Jetty is a lightweight container as a naive attempt to defeat the monolithic J2EE containers because let’s be honest, most of us do not use half the functionalities in J2EE and the clustering of these J2EE containers goes against the very principle of microservices in my opinion. Instead, we should adhere to HTTP and RESTful API everything! Note that Jetty is most certainly cool, but it is not RxNetty or Akka-http cool.

Why Guice

Inversion of control is neat. On a grander scale, Guice can be used to inject mock layers en mass. For example, using Mockito to configure an entire mock data access layer and injecting that in context of unit or integration testing thereby allowing more tests to be written with less dependencies. Guice is also a nice way to help address the separation of concerns by keeping configuration away from business logic. Lastly, being able to do @Inject anywhere is powerful and allows us to construct a lot of templates and basically scale out horizontally through scaffolding code. When used properly, this is the little unsung hero of the Java world in my opinion.

Why Mongodb

Expect endless devops discussion on this very topic. Ever since the hacker news trolls came out of the woodworks against 10gen, the discussion has never ended. I like Mongo. I like it because it is fast to bang out a prototype.

DBs can vastly differ in ACID properties and thus address different combinations of CAP. I think I will save my opinion on Mongodb for another blog post another time. For now, Morphia is nice to work with in Java.

Why Jersey

Jersey is a pretty well structured way to write RESTful endpoints.

Putting it all together

Busting out some sick UML skills here.

Class diagram for api-example

Class diagram for api-example

Some basic principle by convention are as follows:

  • Each entity lives in its own collection in Mongodb
  • Each entity has one data access object (DAO)
  • Facade pattern is applied and should only contain business logic (no DB related operations)
  • Each DAO then at the very least as its own facade that can be extended to support business logic
  • You can freely inject other facades into one another.
  • Each facade maps to one HTTP resource supporting typical CRUD routines for that entity’s
    RESTful interfaces, GET, PUT, POST, DELETE and PATCH (ha!)
  • Caching headers, ETag, IMF headers can live in filters
  • Basic auth is also supported here as an example, that should live in filters too

Benefits

  • Loose coupling between the layers. You can replace Mongo quickly by just replacing the DAO implementations.
  • Most code is scaffolding or pure business logic. All connector code and basic CRUD support, including PATCH, lives in its respective base classes for entity, DAO, facade to resource layer that can be easily extended and reused.
  • Easy to test. All layers are tested, all entities are tested. And the test code can be easily extended
  • You can ship this code to an offshore team and expect that they can easily create new entities and new HTTP endpoints in a short time by simply copypasta some scaffolding code, reuse some templated CRUD classes, follow the basic CRUD routines, and pump out some basic business logic 🙂 Good times!
  • If you actually have a good team to work with, then this stack is very easy to extend by simply following the Facade pattern. Build cool stuff like your own in-memory RRDs for statistics then inject that statistics to other business logic!
  • Clustering is easy because this stack speaks HTTP and you would simply need a load balancer and some minor (or major?) Mongodb config.

Cons

  • At the core, Facade pattern is used liberally. This is not an event driven or reactive approach at all. When using facade, think shared memory, which means threading and parallelism will require due diligence. This is one reason I believe an event-driven, message based approach would improve the Facade pattern.
  • The stack compiles against JDK7. It would work fine with JDK8.
  • Not reactive, and by definition, not hipster enough.

Websocket: the road to hell is paved with good intentions

“Server-push technology” was an elusive sales pitch half a decade ago. COMET, Flash and websockets promised ways for servers to “push” messages onto clients. We sold our souls to heavy weight libraries like Icefaces, GWT, or some other JSF abomination that included hixie-75/hixie-76 support. We became oblivious to this devil’s promise and started blaming browsers when stuff did not work. Now with RFC 6455, the realities of websocket for me is still as disjointed as ever. I started off as an evangelist and a staunch supporter for websocket, but as Captain Hindsight would say, “you are doing it wrong”, and I quickly learnt from my mistakes.

This blog is my personal journey with websocket. In the era of devops and cloud (just as elusive and sales pitchy as ever), in my opinion, I find it really hard to see how this protocol would fit elegantly into an elastic, edge-side, micro-service cloud architecture/infrastructure.

tldr;

Websocket does not work out of the box for most load balancers. Furthermore, when you upgrade the connection directly to the backend (see this piece I wrote for Varnish a while back) you lose edge-side load balancing and caching, thus essentially piping the backend directly to the browser one connection per client at the time. Without some additional clever connection multiplexing components in between the load balancer and the websocket server, like websocket-multiplex or some tricked out node-http-proxy, the websocket server will not scale. For those that prefer sales and marketing lingo, this means it is not “cloud enabled” or “web scale.” Furthermore, websocket’s binary protocol, implemented by libraries such as jWebsocket, is extremely hard to debug in production. Unless you are super savvy with Wireshark and regularly eat TCP dumps for breakfast, and not to mention a bit of a masochist, I highly recommend staying away from websocket all together at the time of writing.

Websocket in practice, and asking some hard devops questions

In the past, I have had the displeasure of working with Icefaces and GWT. These Java front end frameworks abstract away the nitty gritty of the network and protocols such as websocket versions, handshakes, messaging format and error handling with elegant MVC models. This is all well and good on the drawing board, but MVC is a design pattern for separation of concerns on the UI level. Not exactly applicable when talking about the complexity and reality of running a websocket server in this internet and mobile driven world.

I have spent past 4 years developing and supporting an application that first utilised websocket. I have to admit, it was fun building directly ontop of jWebsocket, but it was painful and nearly impossible to debug in production. This was alleviated when we went full blown backbone.js, require.js and d3.js. Keeping things simple pays off in the long run. From that experience, I have devised a checklist for any future websocket projects, and help potentially avoiding the same situation from happening again.

  • Are there any authentication with this websocket implementation? If so, how does this tie into the user model? (jWebsocket for example requires you to specify a couple of system users for authentication and namespaces/roles for authorisation. These are separated from the existing authentication/authorisation model used by the webapp)
  • If it runs within browser, can you tweak the client-side retry logic or connection timeout variables?
  • If this runs outside of a browser (and this gets weird fast) are there any TCP client that can be used for automated testing or health checks in production?
  • How do you test this automatically? Is it supported by Selenium or phantom.js?
  • Can this webserver server be plugged into existing load-balancer? Any additional tweaks and settings required?
  • Does this need multiplexing?
  • How do you debug this on client-side in production? This is usually not possible because the connection is now elevated into a TCP connection and browsers no longer cares for it.
  • How do you debug this on server-side in production? This gets even more tricky as you include multiplexers and load balancers and various other nodes that speaks this binary protocol.
  • How do you debug this at all? Not possible if browser gives up the connection and everything in between is a mystery.
  • Ok so we can do TCP dump and Wireshark. So are you ready to do TCP dump between somebody’s browser and the origin server and everything else in between?
  • Catering to older browsers means Flash fallback. Are you prepared to open an additional Flash port and start supporting this? (and repeat the set of debug and test questions for Flash fallback)
  • Does this thing scale? Yes if you multiplex the connection.
  • How does it handle reconnection and stateful connection?
  • How does the server side handle connections and thread pools handling these logics behind each connection?
  • Does the network infrastructure block upgrade headers? Any other firewall rules that might break websocket?
  • Lastly, you must be prepared to give up caching.

Benefits with websocket

  • True server-push! Yes, this is true. Long-polling or COMET is not exactly “pushing” messages per definition. You are safe from the definition police.
  • You get to play with fun new websocket multiplexing code. This is quite cool actually. Mucking around with node.js is always fun. Perhaps you are thinking about building a Varnish VMOD to support websocket multiplexing. Perhaps you are thinking about building some kind of HTTP cacheable stream for websocket messages before stitching them back out as websocket payloads? This is all very exciting indeed!
  • Ability to build cool HipChat-like applications

Jenkins, jenkins-job-builder and Docker in 2 steps

So here is a simple example that will help provision a VM with Jenkins, jenkins-job-builder and Docker all in one with Vagrant.

Gone are those embarrassing git commits with those peskey jenkins-job-builder yaml files! Throwing in Docker install in Vagrant for Mac users who shuns boot2docker.

Some bonus stuff like examples of provisioning Jenkins plugin via the cli interface, creating first time Jenkins user are chucked in too.

Check it out! https://github.com/yveshwang/jenkins-docker-2step

2-steps are as below

  1. vagrant up
  2. point your browser to http://localhost:38080 and enjoy

and then ..

jenkins-jobs --conf jenkins-job-builder-localhost.ini test your-jobs.yaml -o output/

3 steps really… and for the keen learners wanting some intro material for docker, go here, and here for jenkins-job-builder.