#leetspeak ’14 for me: big ball of mud, devops, architectural validation

Nothing in I.T. is new. We merely learn to call it different names. We continue to battle with big ball of mud. As this guerilla warfare thickens, we learn to scope down and manage smaller balls of mud through tighter collaboration and empathy between dev and ops. Devops anyone? The war against object-oriented programming rages on, backed by experts and Hacker News alike. It seems that we need an escape goat but even as objects are denounced as the new evil one thing remains constant, Python is a great language with a shit runtime and Java is a shit language with a great runtime. Oh and there is still no love for Visual Basic or Sharepoint developers. Actors, messages and events are the new hipster but have they not been around since Erlang or Cobol? We give it a new name, a new stage like C10K and the internet and mobility, forgetting the lessons of old. Nothing in I.T. is new.

It is not often that one gets a total validation to previous architectural design and implementation by the likes of Spotify. I have long defended and championed my Java expertise in the land of sysadmins dinosaurs, where empathy seems to be part of a bike shedding exercise. From this baptism of fire I realised it is not the language, it is the approach and architecture design that matters. A choice we make that will determine the construct and the size of this ball of mud. I walked out of Niklas Gustavsson’s talk thrilled, overwhelmed with joy and kinship. For I am not alone. For the stuff I architected in the past are composed of the same technical choices, principles and components. A big pat on my own back, and we need to do that from time to time 🙂

Then this brings us to devops, collaboration and throwaway code. Well, if Gartner said that it is here in 2015, then the pitch is even easier to convince the execs.

Devop is not a position, title or an engineering degree. I have written in the past that devops is culture. It is empathy and collaboration. For us grunts and peons navigating the corporate waters, this is the ammunition we need to break down silos, poor communications, ridding ourselves of these monolithic non-scalable balls of muds. This is a grass root movement that will change the faces of I.T. departments worldwide.

Indeed it is a rarer occasion to be surrounded by software craftsman of similar background that are driven, passionate and willing to share their experiences. For that I am grateful to both Tretton37 and Gothenburg. If you have missed leetspeak 2014, fear not, it was recorded. So thank you again and I will definitely be back for next year.

Now time to assemble my devop Avengers. We got work to do.

Microservice architecture and startups: a thought excercise

All buzz words aside, microservice architecture or microservices, refer to a style of software architecture where components are built as small and self confined services. These services, when integrated as part of a larger stack, can be arranged to deliver functionalities akin to a traditional monolithic application. This blog post is part of the devops discourse, an exploration and a thought exercise to further examine the best fit of microservice architecture for startups.

Quick micro service summary

Martin Fowler is an authoritative voice on this topic, and he has a series of blog posts outlining the finer details of microservices. Microservice architecture are two peas in a pod with the taxonomy of devops (see https://macyves.wordpress.com/2014/03/08/taxonomy-of-devops-in-startups/), where in addition the former exudes the following characteristics:

  • Componentization via services, service for everything!
  • Organised around business capabilities, cross functional teams
  • Products not projects, full SDLC responsibilities from teams
  • Smart endpoints and dumb pipes, common integration protocols
  • Decentralised governance, common components/libraries
  • Decentralised data management, data redundancy and distribution
  • Infrastructure automation, scalable platforms
  • Design for failure, resilience and high QoS
  • Evolutionary design, continuous integration

Mass adoption?

These aforementioned characteristics are in fact not exclusive to microservice architectures. They can be found in monoliths too. For example, I believe you can have smart endpoints and dumb pipes even when running a large clusters of heavy weight J2EE containers (J2EE is arguably the epitome monolithic platform). You can certainly build decentralised reusable components, which can be put together to form new WAR/EAR packages that can be deployed on an elastic cloud infrastructures. The point here is that enterprises and monoliths are not the mark of the beast. Much like microservices and devops are not the messiah. In fact Fowler stamps his cautious optimism by saying

…you shouldn’t start with a microservice architecture. Instead begin with a monolith, keep it modular, and split it into microservices once the monolith becomes a problem.

So for a start up?

Assuming that resources are scarce and VC funding aplenty, the following are some of the common grounds that could help in the endevour of moving monolithic architecture to a microservice one.

  • Cross functional teams. Build devops mentality by avoiding silos from the get go. This extends even to mixing individual with varying programming languages background at times.
  • Product overheads (release, packaging, maintenance, testing, etc). Avoid pushing out too many tiny products that can lead to overwhelming overheads and potential technical debt. Naturally do not build competing products.
  • If several product lines exists, then the implication is that technical teams do not spend time contributing to the same code base. Camaraderie may be amiss and teams maybe in loggerheads. Promote better cross-team communication and be aware of unintended silos.
  • Cross functional team requires quite a wide spread of competencies. Invest in your resources and promote accountability and dogfooding.
  • Build monoliths and utilise common integration protocols (HTTP is always a good choice). Commonly adopted protocols tends to be easier to debug.
  • Keep the size of the teams small. Large teams build larger system.

Take-aways

Microservice architecture attempts to programmatically address devops via architectural design and giving it a name that resonate with enterprise developers and architects. I predict that adoption of microservices will not be hampered by its inherent complexity or costs, but by human factors. Further more, currently microservices read like a silver bullet to everything, but underneath the surface there looms a fair amount of overhead and potential technical debts. I believe that a technical organisation needs to reach a certain critical mass and attain a certain level of competency in order to fully adopt microservices and reap the full benefits.

Indeed I do not believe a startup company is fully capable of building microservice architecture from the get go. I would advice for a transition strategy from monolith to microservice for small technical organisations. Naturally, startups can easy build components that can be used as a microservice too. Perhaps thats where microservices shines the most for a startup.

Commitment-driven SCRUM

“They always say time changes things, but you actually have to change them yourself.”
― Andy Warhol, The Philosophy of Andy Warhol

Burndown at the end of a sprint is indicative to a team’s velocity. This equates to the amount of work a team can squeeze out of a single sprint. After a couple of sprints, as a team gains domain experience and knowledge, velocity is expected to improve. With this in mind, how does one go about answering these quinella?

  • When is the release date?
  • How much is this costing us?

This blog post aims to answer the above questions, and provide a SCRUM spreadsheet as a tool we use for commitment-drive SCRUM.

The real-world lo-down

Let’s take a step back. If you do SCRUM properly, the question of release dates is moot. That is having a potentially releasable product, which can be pushed out to the world at moments notice, is a utopian scenario for any product managers. This is achievable only through hard yakka and paying down the technical debt, one -ilities at a time.

Externals in the SCRUM team? By adding billable consultants to the team one may start to wonder how much is any of this costing us and is this going to eat into our slush fund?!

Further complicating the overall picture, in the real world, engineers are usually not confined to a single role, task, or team. Some of us do presales, training, support, consulting and generally split our working efforts in various activities (even blogging!). The annoying thing with velocity is that, these “other” activities that are not part of the sprint backlog are categorised as interruptions and/or blockers. These interruptions ultimately impact the team’s velocity. Traditionally if you are a product development team, your central role is to build features, fix reported bugs, and push out releases. Not presale, not training, not support and not consulting.

Support is indeed tricky. Typically one is unable to estimate how much support may flood in. This is at loggerheads with the notation of an agreed sprint backlog. Though this can be mitigated with lean principles, WIP and cumulative burndown, but that is not SCRUM and the discussion is for another blog post, another time.

User story points vs man hours

If you come from the SCRUM school of Mike Cohn, like me, you should already be tut-tutting, arms crossed and ranting, “one should not measure velocity in man hours but in user story points!” However if you have a steering group asking for the release dates and cost of development, or you have external consultants billing hours on the project, you know the only measurement that really matters is man hours, which can be directly translated into a monetary cost.

Using man hours is a double edged sword. The clear benefit is that you are immediately given a sense of expected delivery. This contradicts the central premise of user story points, that a task should be measured on its complexity and size based on a common understanding and analysis. At this point we are unsure if time-based estimates are better suited for us than user story points.

The other issue with time-based estimation is that team members may find themselves giving dishonest estimates, be it too optimistic or pessimistic. Be weary of people touting “oh they build xyz in just one day.” Or, “it’ll take me 10 years to do xyz”. Both quotes should trigger alarm bells.

Commitment-drive

Most engineers are used to percentages as an indicator of commitment. For example, “I’d spend 50% on project A, 20% on project B, 10% on meetings and doing expense reports, and 20% on water cooler conversations this week.” is not unheard of. This simple model fits our sprint schedule, where one could identify their non-working days ahead of time, percentage of working hours for each projects and tasks, and a preliminary prediction on admin and meeting hours.

Naturally none of the existing SCRUM tools of the world currently support the notion of commitment. Therefore as part of evaluating our tools and processes, old school spreadsheet was deemed worthy of a revisit. I have come across some of the most comprehensive SCRUM support and prediction models that was entirely written in Excel. So I was not too surprised that we stumbled on this stirling little pearler of a SCRUM spreadsheet, by Xavie Esteve. The spreadsheet has a great burndown chart and uses estimation in hours. It is simple to get started and we loved it. I extended it with calculations based on commitment, helping to determine if we have fit the right amount of tasks based on the predicted maximum velocity. The end result is here and the following were added.

  • Theoretical maximum team velocity
  • Reduced max team velocity based on planned engagements
  • Adjusted max team velocity based on adjusted capacity based on unplanned engagements
  • Recommended daily burndown per person

All calculations above take into account vacation days, public holidays or any other planned non-working days.

The biggest benefit is that we used commitment to incorporate other tasks in our estimation and sprint planning. For example, individuals on weekly support are capped at max 50% commitment to the team.

Shortcomings

The spreadsheet is an excellent starting point. But it really lacks the overall picture. The following would be fantastic to have.

  • Task assignment and notification
  • A sprint board
  • Re-adjusted daily burndown prediction based on unplanned non-working days or engagements
  • Calculating expected velocity based on previous sprints, perhaps based on an average or median

As a side note, we are still evaluating other options and tools. Stay tuned.

Taxonomy of DevOps in startups

Software engineering is changing and DevOps is at the heart of it. There are myriad of DevOps blogs and conference talks on transforming enterprises to more DevOps, usually centered around incorporating operational concerns early on software development lifecycle (SDLC).

An Enterprise Transformation

On a larger scale, Mike Kavis, VP/Principal Architect for Cloud Technology Partners, puts it quite bluntly on a post titled No, You Are Not a DevOps Engineer.

“Enterprises are struggling with DevOps. They all want DevOps even though many do not know what it is. In many cases, I see infrastructure teams who are calling themselves DevOps leading a grassroots initiative. When I ask them where the development team is, they often say either “we did not invite them,” or even worse, “we don’t talk to them.””

Kavis goes on saying:

“Another reoccurring pattern I see is that a “DevOps” team’s first step is often to figure out if they are going to use Chef or Puppet (or Salt or Ansible or whatever else is hot). They have not even defined the problems that they are setting out to solve, but they have the tools in hand to solve them. Often these teams wind up building thousands of lines of the scripts, which raises the question, “are we in the business of writing Chef scripts or in the business of getting to market faster with better quality and more reliability?” Too often, these teams code themselves into a corner with mountains of proprietary scripts that actually add more waste to the system, instead of removing waste from the system, which is what the driving forces behind the DevOps movement are all about.”

Arguably these are symptoms specific to big enterprises, where traditionally massive engineering silos exists with minimal communication coupled with ticketing/queuing systems in between departments. But are startup basically immune to these symptoms?

Definition of DevOps

I agree with Kavis that DevOps is not a role or job title, it is not an engineering discipline nor a methodology or process framework. I also agree with this post, titled What Is (Not) DevOps, and How do We Get There, that DevOps is a culture shift. It is a movement that encourages and improves communication, collaboration, and fosters quality software product delivery through tight knit teamwork.

Daemon Edward, Co-Founder of DTO Solutions stated in QCon London 2014, titled Dev “Programming” Ops for DevOps Success, that the definition of DevOps to him is

  • a way of seeing your problems
  • a way of evaluation solutions
  • a way of communicating these things
  • always evolving

I wholeheartedly agree.

DevOps taxonomy in startups

Arguably the size of startups and the limited resources requires every technical members to be all-hands-on-deck, participating in all aspects of SDLC. Often those that are slightly more interested in Ops would take the lead in addressing the many -ilities in requirements and release management. Those more interested in Devs would take the lead on churning out code and test automation. Based on the lack of resource and the inherent drive and focus of a startup, the taxonomy of DevOps in startups would be the following:

  • -ilities are addressed as backlog items during development
  • Release/production concerns are on the table from day one
  • Teams have a tighter bond, communication channel transparent
  • All efforts are tightly mapped with tangible values (be it business or customer)

The DevOps sweet spot

It would appear that startups are DevOps havens. This would not be the case. Startups are not immune to the aforementioned problems in larger enterprises. Silos do exist in startups, be it a single man team or hundreds. There are plenty of pitfalls along the way. In my experience, I believe in the existence of a DevOps sweet spot. The points below are my own guidelines for chasing that utopian DevOps shangri la.

  • Non-functional requirements, -ilities as features residing in backlogs
  • Build systems that makes managing easy for Ops
  • Foster a nurturing company environment for thriving Devs
  • Treat Ops as customer facing, sales/consultancy alignment
  • Treat Devs as business value delivery, marketing/R&D alignment
  • Adopt lean/agile principles
  • Last but not least, Stop, collaborate, and listen! (no wiser words have come out of the 90s)