DevOps, OpsDev or a bit of both?

At a recent meetup a group of us were joking around with the term OpsDev because in our experience DevOps is more often than not pushed from the operations side rather than development. So the question came up "why do the devs get to come first?" Googling the term OpsDev later I discovered it is a real thing.

The UK website Business Computing World carries a passionate piece by Randy Clark from 2012 in which he argues:

For me, in whichever context Ops is talked about, the industry is still getting it back to-front. It should be OPSdev, OPSweb, OPScloud. Ops isn’t just important; it’s essential. Putting Ops second is a sign that although businesses realise its role, they don’t fully understand the far reaching effect of Ops.

This HP white paper from 2013 states:

After engaging with many enterprises, we learned that the adoption of DevOps practices and methodologies in large enterprises is often driven by IT operations, and not necessarily by the developers of the Line of Businesses. We call this OpsDev.

The paper goes on to talk about how DevOps and OpsDev often address the same business problems but they are often driven by different concerns. I found this image a particularly interesting illustration of this.


None of the goals listed here under DevOps can be realized without those under OpsDev and vice versa. This is the key to understanding the collaboration component of the DevOps culture. For better or worse, Patrick Debois chose to call his conference DevOpsDays in 2009 and the now infamous twitter hash tag #DevOps was borne. Along with it emerged a massive cultural movement that can not be turned back.

What is DevOps?

For a lot of people in operations, myself included, it was difficult to get what the hype was about back in 2009. Moreover the question a lot of ops folks asked is "Why does dev have to come first?"
Automation, continuous integration and deployment, collaboration between ops and dev, aren't these all just operations best practices? This is what good Sys Admins have advocated for years. Add to this the marketing hype attached to DevOps in recent years and we begin to see some justification for the skepticism.

But the movement has proved the skeptics wrong in a number of ways. The DevOps movement brought to the fore frustrations felt within many engineering organizations. Developers were frustrated with traditional operations teams inability to respond with the perceived urgency demanded. Operations people were equally frustrated with being treated like human keyboards and other people's poor planning becoming their emergency. What had been discussed for years over beers now became the topic of industry wide conferences. Tools began to emerge to help solve some of these problems in a more generalized way.

The DevOps movement enabled software engineers and engineering managers to begin thinking about what goes on after the code is deployed. The Agile movement had taken us a long way towards this with automated testing and continuous delivery. Developers became more involved in deployments but it was still the responsibility of the operations team to run and maintain production. The proverbial wall was still there and the code was essentially still thrown over it. Where it landed was the responsibility of operations.

Virtualization and cloud computing has taken us further along the road of true collaboration. It has made possible the full realization of Infrastructure as Code. This in turn has made it possible to enable developers to provision their own infrastructure. Some have interpreted this as NoOps and even gone so far as to eliminate their operations teams. That is a big mistake.

DevOps + Cloud do not eradicate the need for operations. Rather they illustrate the need for OpsDev -- Operations people who get DevOps alongside DevOps -- Developers who get DevOps. I am not proposing here to start using the term OpsDev in place of DevOps. It is way too late for a name change here. What I am trying to illustrate is that DevOps practices and culture need to be embraced by both operations teams and development teams. Once the teams themselves are collaborating better the next challenge is the cross team collaboration. When we begin to bring together the acquired skills and best practices of operations with those of software engineering we can build some truly awesome systems. This is what DevOps done right enables.

What DevOps can learn from Kanban

One of the great strengths of Kanban over other Agile methodologies is its great flexibility. For operations, the ability to incorporate interrupts easily is essential. While the goal of any operations team should always be to plan as much as possible, the fact is interrupts do happen. Kanban suggests itself as a natural system for organizing operations workflow but it is far more than that. I believe there is much to be learned from Kanban for anyone leading a DevOps program.

Principles of Kanban

The principles of Kanban align closely with the culture of DevOps. One of the best and more detailed expositions of the Kanban method I have read recently is Kanban from the Inside by Mike Burrows.

Burrows avoids introducing the Kanban method from its Foundation Practices and Core Principles in favor of a system of nine values which he lists as follows:

• Transparency
• Balance
• Collaboration
• Customer Focus
• Flow
• Leadership
• Understanding
• Agreement
• Respect

These nine values could well be listed as the components of a DevOps culture. Burrows goes on to elaborate four foundational principles which he labels as follows:

FP1: Start with what you do now.
FP2: Agree to pursue evolutionary change
FP3: Initially, respect current processes, roles, responsibilities, and job titles.
FP4: Encourage acts of leadership at every level in your organization -- from individual contributor to senior management.

Again these are very useful foundation principles, not only for Kanban but for DevOps in general. FP2 and FP3 on the pursuit of evolutionary change and respecting initial processes are particularly important. More often than not a company turns to DevOps in recognition of broken practices and procedures. Few companies are completely homogeneous and different teams will have their own ideas about what must change and how quickly. The best recipe for success with ah DevOps program is not to seek drastic changes overnight but to encourage new processes as the culture develops.

The DevOps toolset can help tremendously with this this. A move towards configuration management implies a change in practice by both operations and development teams. Self enablement of developers implies changes to roles, responsibilities and in some cases even job titles. But we need the tools in place before we can change process. This is how I interpret FP3: "Initially, respect current processes." Once the tools are in place and we have buy in from some teams and individuals, it becomes easier to convince others.

Encouraging acts of leadership is also central to the DevOps culture. It encourages individuals at every level of an organization to take ownership of the product. Not simply their part of it but the whole product.

Burrows also elaborates six Core Practices as follows:

CP1: Visualize.
CP2: Limit Work-in-Progress (WIP).
CP3: Manage flow.
CP4: Make policies explicit.
CP5: Implement feedback loops.
CP6: Improve collaboratively, evolve experimentally (using models and the scientific method).

As with the foundation principles, these core practices can be of enormous benefit to a DevOps program. Let's look at how each one of these can assist with DevOps.

Kanban and DevOps

Visualize

A friend of mine asked me recently how one measures the success of a DevOps program and the first thing that came to mind was the need for visualization. The Kanban board is the the most basic visualization tool. Anyone interested can see the progression of work through the system. They can see what is blocked, what is in progress and get a rough approximation of when their "most urgent of all tasks"  are likely to be completed. In addition I like to produce metrics relating to team velocity that can illustrate improvements resulting from the introduction of DevOps.

Of course  the Kanban board doesn't account for all the work being done by a team. While there certainly should be a card associated with every task, there are other metrics that become useful when one is implementing infrastructure as code. Tools such as github make it simple to generate reports relating to code checkins, pull requests etc. This should be part of your velocity metrics, as should mean time to acknowledge alerts from your monitoring system and mean time to resolve production issues.

This is all part of the Visualization advocated by Kanban. Make it clear what your team does and where they are at in the workflow.

Limit WiP

Operations teams and even some development teams are invariably pulled every which way by demands from multiple stakeholders. Every stakeholder naturally thinks their issue is the most important issue of the day. Limiting Work-in-Progress in a very visual way can be a useful asset in explaining why people sometimes just need to be patient.

By taking stakeholders to the Kanban board and pointing out the current issues in progress, we can initiate the conversation about what should be dropped in favor of that most urgent issue you are being asked to work on. It is then possible to explain to the person demanding you drop everything that they need to talk to the stakeholders affected if you do so.

Feedback Loops

The success of "pushing back" as I suggest above will be heavily affected by the feedback loops in place. If stakeholders are left wondering if their request is just sitting in some backlog somewhere to never be reviewed, the above push back will not be very successful. We need to provide continuous feedback. Could the stakeholder have done anything different that would make the processing of the request smoother? Could we have done anything different? All of this relates strongly to the first principle of Kanban, transparency. Be totally transparent about your workflows, WiP limits and team velocity. It will pay big dividends in the end.


Collaboration

It should not be necessary to explain to DevOps folks the relationship of CP6 to DevOps. It is the very heart of a DevOps culture. We need to improve collaboratively, both within our team and between teams. That is really what DevOps is about.

Conclusion

In conclusion I want to emphasize that I have presented here only a brief synopsis of ideas of Mike Burrows as elaborated in his work Kanban from the Inside. I strongly advise everyone involved with DevOps to read this book. There is much to learn from the agile movement and Kanban in particular.