#leetspeak ’14 for me: big ball of mud, devops, architectural validation

Nothing in I.T. is new. We merely learn to call it different names. We continue to battle with big ball of mud. As this guerilla warfare thickens, we learn to scope down and manage smaller balls of mud through tighter collaboration and empathy between dev and ops. Devops anyone? The war against object-oriented programming rages on, backed by experts and Hacker News alike. It seems that we need an escape goat but even as objects are denounced as the new evil one thing remains constant, Python is a great language with a shit runtime and Java is a shit language with a great runtime. Oh and there is still no love for Visual Basic or Sharepoint developers. Actors, messages and events are the new hipster but have they not been around since Erlang or Cobol? We give it a new name, a new stage like C10K and the internet and mobility, forgetting the lessons of old. Nothing in I.T. is new.

It is not often that one gets a total validation to previous architectural design and implementation by the likes of Spotify. I have long defended and championed my Java expertise in the land of sysadmins dinosaurs, where empathy seems to be part of a bike shedding exercise. From this baptism of fire I realised it is not the language, it is the approach and architecture design that matters. A choice we make that will determine the construct and the size of this ball of mud. I walked out of Niklas Gustavsson’s talk thrilled, overwhelmed with joy and kinship. For I am not alone. For the stuff I architected in the past are composed of the same technical choices, principles and components. A big pat on my own back, and we need to do that from time to time 🙂

Then this brings us to devops, collaboration and throwaway code. Well, if Gartner said that it is here in 2015, then the pitch is even easier to convince the execs.

Devop is not a position, title or an engineering degree. I have written in the past that devops is culture. It is empathy and collaboration. For us grunts and peons navigating the corporate waters, this is the ammunition we need to break down silos, poor communications, ridding ourselves of these monolithic non-scalable balls of muds. This is a grass root movement that will change the faces of I.T. departments worldwide.

Indeed it is a rarer occasion to be surrounded by software craftsman of similar background that are driven, passionate and willing to share their experiences. For that I am grateful to both Tretton37 and Gothenburg. If you have missed leetspeak 2014, fear not, it was recorded. So thank you again and I will definitely be back for next year.

Now time to assemble my devop Avengers. We got work to do.


Outsourcing and Scrum: our tools, our process, and some pitfalls

Part 2 of the outsourcing series on Bangladesh.

Last blog we touched on the difficulties in hiring and managing an outsourced team. This blog entry will highlight the tools we used, our processes and some of the pitfalls.

Cloud everything

Due to the time difference and the distributed nature of outsourcing, solid communication channels are critical. The general lack of stable infrastructure in Bangladesh meant that we could not have tools hosted there due to the high risks of down time. Furthermore, power outages would frequently plague the Dhaka office until a backup generator was installed.

We used Google for our emails and documentation, at times Dropbox. We used Jira Studio, particularly Jira, Confluence and Greenhopper for bug tracking, Wiki, Scrum tool respectively. In addition, we use Balsamiq and Gliffy plugins. The Jira Studio packages seamlessly integrates with our Google accounts, which made access control easy. The one thing I loved about Jira Studio is that all the tools are integrated; this includes our source control system. Note that Jira Studio is now migrated to Atlassian OnDemand.

Github is where we store the code. It allows for local cache copies and caters for your very own push process. i.e. Setup a local git repo in Dhaka, team pushes there. A pull request can be submitted upstream and reviewed for example. It can be as simple or as sophisticated as you want.

For sprint demos, we used Webex and TeamViewer for remote access.

Last but not least, Instant Messages and video conferences such as Skype or GTalk was very useful. Did I mention that Voice-over-IP is blocked in Bangladesh? One needs a government permit to get through the great firewall of Dhaka.

Planning and running sprints

Before any sprint planning occurs, the team needs to agree on a “definition of done”. This is the notion that a task is deemed complete. The team needs to come up with this themselves and agree to it. For example, unit testing, documentation and compile-error-free code is pushed to repo as part of the definition of done.

For this project, requirements were written as epics initially, then broken down in user stories. Each user stories are estimated in user story points by the technical team, as well as attaching a business value from our steering group.

Each user story is then further dissected into into technical tasks and estimated in man hours no more than 2 days worth of work. A user story is completed when all associated tasks is deemed complete.

The product owner will prioritise the user stories and provide a product backlog that can be potentially accepted as sprint backlog.

During sprint planning, we used an online planning poker provided by Mountain Goat Software and each task has an initial estimate and a remaining estimate field. These two fields allows us to get a good overview of the burndown and thus the associated velocity in each sprint. Note that an estimate is only accepted if it is a unanimous decision. You will be surprised how people will come to an agreement eventually.

With this approach, you will get burndown in user story points, business value and man hours. The benefit is that after a few sprint, you will get a sense of the complexity of a user story through user story points, and the team`s general performance via man hours.

At the end of each sprint, both a sprint demo and a retrospective is conducted and any unfinished user story is pushed to the following sprint. Note that re-estimation is not performed on delayed user story. In fact, re-estimation is something I do not recommend at all. For example, it is a known fact that estimation is quite inaccurate in the start of any project. Not to mention that there will be a varying degree of competency and perhaps confidence level. As the team progresses, estimation becomes a closer reflection of the complexity of the task at hand, as well as team`s proficiency with the problem domain. Thus the estimation will converge as individuals` own competency improves. In order to fully illustrate this and correctly reflect the increase in velocity, the old crappy estimates must remain. If the project is intended to be very short, then re-estimation *might* make sense.

Sprint demo is the chance where the team can really shine, or fail miserably. Though the Scrum Master should try to protect the team, if the sprint is faced with a ton of road blocks, it is inevitable that the demo will not be their proudest moment. However, that should improve with time. The Scrum Master should be the best communicator in English and host these demos. Anybody from the organisation can attend these demos and encourage them to do so. I have heard that Google often wheel out cake and coffee during sprint demos to set a celebratory tone. And indeed it should be a celebration.

At the end of each sprint, the product needs to be in a shape where it is potentially releasable. This is extremely important as I believe it is the central concept of any agile processes, particularly Scrum. Even if the product does not have a packing or release process, you can still convey to the team that test users needs to be able to check out the code and run the program via some simple scripts. Being able to potentially release at any moment is critical to the success and pacing of the project.

Some pitfalls

  • Estimation. Some team members estimate in perfect days (7,5 hours without any interruption), some include lax time in their estimates. The unanimous estimation aims to find a balance between the two. In time, the team itself will understand and identify correctly the difference between a 2 hour vs 8 hour tasks.
  • Business value estimation is of no use to the business. This is a typical case where an idea is good, but have no real life use usage. Establishing a steering group to iron out these type of features is great to avoid erroneous requirements will hopefully ensure the invested features are with the support of marketing and sales. Once this becomes a routine, showing how much business value deliver per sprint is as easy as looking at the burndown.
  • Inherent hierarchies in Bangladesh. There is an intricate and ingrained notion of hierarchies in the Dhaka team. An interesting observation for me is that a junior engineer cannot be seen having a cigarette when a senior engineer member is in the vicinity. The definition of a junior is merely somebody who has joined the company later, or is of a younger age. Nothing to do with their technical skills. To further exacerbate this symptom, the junior is rarely seen correcting the seniors. This is a huge problem because I would like the team to operate in a flat hierarchy, where ideas and innovation can be freely discussed. There is not easy cure for this as it is part of establishing a local company culture that promotes innovation and discussion, while leaving the old world baggage behind. Our approach was to build a good team rapport and encourage the juniors to speak their mind. One last suggestion is to talk to the juniors directly on a 1-1 basis to find out their thoughts and suggestions. This is quite time consuming of course, but it can be rewarding and often serve as an early indicator if there is something fishy going on with the team.
  • User stories are vague and thus form a horrible basis for requirements. This is the general pitfall of Scrum, not particularly an outsourcing issue. Establish concrete tasks and implementation based on an idea is never straightforward. A vision can be as ambiguous as “As a sysadmin, I want to create a user in this system, so that this newly created user and login and work with the system.” This is a typical user story that any CRUD application will support. But what does that user story really entails? How much work is this? How much business value does this actually give? These are questions that both the team and the steering group will need to iron out for themselves. However, I cannot recommend the use of condition of satisfaction enough for user stories. Be as meticulous as possible when writing condition of satisfaction. Your QA will thank you. If you are the borderlining the formal methods type, then throwing in a UML design methodology would not hurt. It will bolster the understanding of both the business requirements and the implementation itself.
  • The team used powerpoint slides for their sprint demo. This is generally not encourage as the idea of having a potentially releasable product is obviously not met. However, there are at times where this is perfectly ok. I encourage good documentation throughout the development cycle and if extracting power point slides from a few wiki pages will do the trick. So be it. Be sure to not make this a habbit.
  • Certain tasks such as refactoring will render the product in a potentially unshippable state. Create a new branch and merge it back to the release branch is your solution. No excuses that a product cannot be shipped.