Why I created DevOps Advocates

After recently leaving my full time gig as a DevOps Manager in a large enterprise I had some time to think about my future career goals and where I fit within the DevOps ecosystem.

I've worked at a number of interesting companies over the years and as regular readers of my blog will know, I have developed my own opinions of what DevOps is and how to do it right.

Surveying thousands of job posts with "DevOps" in the title, it struck me that few, if any, spoke about culture. Job posts for release engineers, automation engineers, systems administrators, even software engineers and developers are all commonly prepended with DevOps. These posts call for skills that have existed years before the term DevOps was coined, so what makes them DevOps?

This got me thinking about some of the difficulties I have encountered in trying to bring DevOps and more generally agile concepts into companies large and small. The problems I encountered were never around tools but culture. Sure, there maybe issues to address such as tool adoption, the use of pull requests to help collaboration, Kanban boards and so on but that was never the source of the problems.

I've seen various approaches, embedded DevOps, DevOps teams, transformations of existing teams and in all cases the cultural aspect of DevOps is a much bigger issue than the tools. Why is this?

At DevOps Days Boston there was an excellent speech by Katie Rose titled GridIronOps: What Women's Tackle Football Can Teach Us About Competitive Software. In the course of this unique and lively presentation of the cultural challenges in DevOps, Katie made the point that one similarity between DevOps and women's football is that we are trying to change the culture of adults. Young boys learn the collaborative required for successful team work at early age but these are grown women taking up the sport as adults.

This comparison left a powerful impression on me. Coming into a company to "implement DevOps" inevitable meets with cultural resistance. If you build a separate DevOps team it is immediately perceived as a challenge to other teams that existed before you got there. If you start merging operations people with development teams then you are bringing together what are, more likely than not, conflicting ways of working. If you are a development manager seeking to take on automation and configuration management, there is the danger that you will be seen as a threat by the operations team. If you are an operations team trying to encourage developers to take on their own deployments you run the risk of being seen as trying to pass off responsibilities. All of some of theses problems will arise in the course of trying to implement DevOps.

So, to get back to the title of this post, why did I create DevOps Advocates and what is it? Thinking about how to avoid some of the problems cited above, I was reminded of the early days of agile and the concept of agile coaching. Companies that were most successful with agile often hired outside consultants to come in and review existing workflows. These consultants would then work with the teams to figure out where agile would fit within the business. What were they doing right? and wrong?

The use of a third party avoided the defensiveness one often sees in trying to move a company over to a DevOps culture and practice. The agile coach was seen as an independent voice who's only agenda was to improve process and make a company more successful. So I decided there is a need for DevOps coaching along the same lines as the Agile coaching of an early age.

I am seeking out consulting gigs from companies who need more than tools. Lots of companies have intelligent people who, with a little nudge, can become excellent advocates and practitioners of DevOps. The companies I am interested in are those willing to have an independent assessment of where they are currently on their DevOps journey and where they need to get. Initial conversations at DevOps Days and other meetups tell me there is a demand for what I am pitching. It's early days so only time will tell if I am right but it would be stupid not to give it a try.

For more information or to get in touch visit http://devopsadvocates.com/.

DevOps, OpsDev or a bit of both?

At a recent meetup a group of us were joking around with the term OpsDev because in our experience DevOps is more often than not pushed from the operations side rather than development. So the question came up "why do the devs get to come first?" Googling the term OpsDev later I discovered it is a real thing.

The UK website Business Computing World carries a passionate piece by Randy Clark from 2012 in which he argues:

For me, in whichever context Ops is talked about, the industry is still getting it back to-front. It should be OPSdev, OPSweb, OPScloud. Ops isn’t just important; it’s essential. Putting Ops second is a sign that although businesses realise its role, they don’t fully understand the far reaching effect of Ops.

This HP white paper from 2013 states:

After engaging with many enterprises, we learned that the adoption of DevOps practices and methodologies in large enterprises is often driven by IT operations, and not necessarily by the developers of the Line of Businesses. We call this OpsDev.

The paper goes on to talk about how DevOps and OpsDev often address the same business problems but they are often driven by different concerns. I found this image a particularly interesting illustration of this.


None of the goals listed here under DevOps can be realized without those under OpsDev and vice versa. This is the key to understanding the collaboration component of the DevOps culture. For better or worse, Patrick Debois chose to call his conference DevOpsDays in 2009 and the now infamous twitter hash tag #DevOps was borne. Along with it emerged a massive cultural movement that can not be turned back.

What is DevOps?

For a lot of people in operations, myself included, it was difficult to get what the hype was about back in 2009. Moreover the question a lot of ops folks asked is "Why does dev have to come first?"
Automation, continuous integration and deployment, collaboration between ops and dev, aren't these all just operations best practices? This is what good Sys Admins have advocated for years. Add to this the marketing hype attached to DevOps in recent years and we begin to see some justification for the skepticism.

But the movement has proved the skeptics wrong in a number of ways. The DevOps movement brought to the fore frustrations felt within many engineering organizations. Developers were frustrated with traditional operations teams inability to respond with the perceived urgency demanded. Operations people were equally frustrated with being treated like human keyboards and other people's poor planning becoming their emergency. What had been discussed for years over beers now became the topic of industry wide conferences. Tools began to emerge to help solve some of these problems in a more generalized way.

The DevOps movement enabled software engineers and engineering managers to begin thinking about what goes on after the code is deployed. The Agile movement had taken us a long way towards this with automated testing and continuous delivery. Developers became more involved in deployments but it was still the responsibility of the operations team to run and maintain production. The proverbial wall was still there and the code was essentially still thrown over it. Where it landed was the responsibility of operations.

Virtualization and cloud computing has taken us further along the road of true collaboration. It has made possible the full realization of Infrastructure as Code. This in turn has made it possible to enable developers to provision their own infrastructure. Some have interpreted this as NoOps and even gone so far as to eliminate their operations teams. That is a big mistake.

DevOps + Cloud do not eradicate the need for operations. Rather they illustrate the need for OpsDev -- Operations people who get DevOps alongside DevOps -- Developers who get DevOps. I am not proposing here to start using the term OpsDev in place of DevOps. It is way too late for a name change here. What I am trying to illustrate is that DevOps practices and culture need to be embraced by both operations teams and development teams. Once the teams themselves are collaborating better the next challenge is the cross team collaboration. When we begin to bring together the acquired skills and best practices of operations with those of software engineering we can build some truly awesome systems. This is what DevOps done right enables.

What DevOps can learn from Kanban

One of the great strengths of Kanban over other Agile methodologies is its great flexibility. For operations, the ability to incorporate interrupts easily is essential. While the goal of any operations team should always be to plan as much as possible, the fact is interrupts do happen. Kanban suggests itself as a natural system for organizing operations workflow but it is far more than that. I believe there is much to be learned from Kanban for anyone leading a DevOps program.

Principles of Kanban

The principles of Kanban align closely with the culture of DevOps. One of the best and more detailed expositions of the Kanban method I have read recently is Kanban from the Inside by Mike Burrows.

Burrows avoids introducing the Kanban method from its Foundation Practices and Core Principles in favor of a system of nine values which he lists as follows:

• Transparency
• Balance
• Collaboration
• Customer Focus
• Flow
• Leadership
• Understanding
• Agreement
• Respect

These nine values could well be listed as the components of a DevOps culture. Burrows goes on to elaborate four foundational principles which he labels as follows:

FP1: Start with what you do now.
FP2: Agree to pursue evolutionary change
FP3: Initially, respect current processes, roles, responsibilities, and job titles.
FP4: Encourage acts of leadership at every level in your organization -- from individual contributor to senior management.

Again these are very useful foundation principles, not only for Kanban but for DevOps in general. FP2 and FP3 on the pursuit of evolutionary change and respecting initial processes are particularly important. More often than not a company turns to DevOps in recognition of broken practices and procedures. Few companies are completely homogeneous and different teams will have their own ideas about what must change and how quickly. The best recipe for success with ah DevOps program is not to seek drastic changes overnight but to encourage new processes as the culture develops.

The DevOps toolset can help tremendously with this this. A move towards configuration management implies a change in practice by both operations and development teams. Self enablement of developers implies changes to roles, responsibilities and in some cases even job titles. But we need the tools in place before we can change process. This is how I interpret FP3: "Initially, respect current processes." Once the tools are in place and we have buy in from some teams and individuals, it becomes easier to convince others.

Encouraging acts of leadership is also central to the DevOps culture. It encourages individuals at every level of an organization to take ownership of the product. Not simply their part of it but the whole product.

Burrows also elaborates six Core Practices as follows:

CP1: Visualize.
CP2: Limit Work-in-Progress (WIP).
CP3: Manage flow.
CP4: Make policies explicit.
CP5: Implement feedback loops.
CP6: Improve collaboratively, evolve experimentally (using models and the scientific method).

As with the foundation principles, these core practices can be of enormous benefit to a DevOps program. Let's look at how each one of these can assist with DevOps.

Kanban and DevOps

Visualize

A friend of mine asked me recently how one measures the success of a DevOps program and the first thing that came to mind was the need for visualization. The Kanban board is the the most basic visualization tool. Anyone interested can see the progression of work through the system. They can see what is blocked, what is in progress and get a rough approximation of when their "most urgent of all tasks"  are likely to be completed. In addition I like to produce metrics relating to team velocity that can illustrate improvements resulting from the introduction of DevOps.

Of course  the Kanban board doesn't account for all the work being done by a team. While there certainly should be a card associated with every task, there are other metrics that become useful when one is implementing infrastructure as code. Tools such as github make it simple to generate reports relating to code checkins, pull requests etc. This should be part of your velocity metrics, as should mean time to acknowledge alerts from your monitoring system and mean time to resolve production issues.

This is all part of the Visualization advocated by Kanban. Make it clear what your team does and where they are at in the workflow.

Limit WiP

Operations teams and even some development teams are invariably pulled every which way by demands from multiple stakeholders. Every stakeholder naturally thinks their issue is the most important issue of the day. Limiting Work-in-Progress in a very visual way can be a useful asset in explaining why people sometimes just need to be patient.

By taking stakeholders to the Kanban board and pointing out the current issues in progress, we can initiate the conversation about what should be dropped in favor of that most urgent issue you are being asked to work on. It is then possible to explain to the person demanding you drop everything that they need to talk to the stakeholders affected if you do so.

Feedback Loops

The success of "pushing back" as I suggest above will be heavily affected by the feedback loops in place. If stakeholders are left wondering if their request is just sitting in some backlog somewhere to never be reviewed, the above push back will not be very successful. We need to provide continuous feedback. Could the stakeholder have done anything different that would make the processing of the request smoother? Could we have done anything different? All of this relates strongly to the first principle of Kanban, transparency. Be totally transparent about your workflows, WiP limits and team velocity. It will pay big dividends in the end.


Collaboration

It should not be necessary to explain to DevOps folks the relationship of CP6 to DevOps. It is the very heart of a DevOps culture. We need to improve collaboratively, both within our team and between teams. That is really what DevOps is about.

Conclusion

In conclusion I want to emphasize that I have presented here only a brief synopsis of ideas of Mike Burrows as elaborated in his work Kanban from the Inside. I strongly advise everyone involved with DevOps to read this book. There is much to learn from the agile movement and Kanban in particular.

When you can't see the forrest for the trees

How often do you work at a company and see only the bad? I see this a lot. In operations particularly we are dealing with so many problems that we often lose site of the awesome, cool stuff we are doing.

I was at a meet-up recently where a company was demoing some orchestration tools they had built for wrangling AWS. This was of particular interest to me because my present company has spent a lot of time doing just that. As a primary user of these tools there has been a lot of frustration. Infrastructure orchestration is complex. Any tool built will have bugs. When you are the one fighting through the bugs, you tend to have a very negative opinion of a tool set. But getting out there and seeing what others are doing can be very refreshing.

This particular tool that was being presented was very good as far as it went. But I soon realized it was doing about ten percent of what we are doing in my present company. The purpose of this short post is not to brag about what we are doing -- although it really is pretty awesome! My intent is to caution against getting bogged down by the problems.

The problems are a way of life and operations people see more of them than any other team. My advice; take a step back. Get out there on the meet-up circuit, look at what others are doing and you may well get a new perspective on how awesome your company and your team really is.

After some 30 years in the IT industry I can say with some authority that all companies have their fair share of good and bad. I can also say that the bad tends to dominate in any bar room discussion. We need to do a better job of advertising the good. The DevOps movement has helped a lot with this but we can do more.

Look for the cool in what you are doing and talk about it more. It will make you feel good and others around you.

Some thoughts on planning

Anyone working in the IT industry has seen, heard or said this phrase more than once. Unfortunately, if you in operations then someone else's inability to plan more than likely DOES become your emergency.This remains true even in companies preaching DevOps. The larger and more complex the company, the greater the chances that there is a team somewhere, Operations, DevOps, whatever the name, who's workload is driven almost entirely by interrupts.

These teams often adopt Kanban as a means of making the problem visible but Kanban alone won't fix the problem. Kanban highlights the demands being made upon an operations team and can be useful in improving the work flow but the real problem is in planning.

DevOps at its heart is about collaboration between teams, avoiding silos etc. In a few startups DevOps results in single teams made up of developers and operations people who are in on the plan from the beginning and everyone gets to celebrate at the same time due to a common understanding of "done". At its best DevOps encourages an understanding that a feature is only completed when it is running in production and in the hands of the end user.

In far too many cases, however, companies proclaiming they are doing DevOps have not redefined their definition of "done". In these companies, development teams consider work done when the code builds successfully, or when QA has signed off on it. The task of making this run in production is not considered part of the development process, as a result, what is required to make and keep this running in production is often not part of the plan.

The need for planning is certainly understood by software development teams and forms an integral part of agile methodologies such as Scrum. Developers would revolt if they spent four hours in a planning meeting only to be told three days into a two week sprint, "forget the plan, something has come up and we need you to do this instead." Yet it is considered perfectly acceptable to assign tasks to an operations team towards the end of a two sprint for work that could easily have been predicted in planning.

DevOps Planning

Clearly there is a need for better planning. Many organizations seek to integrate operations people into development sprint planning sessions. This can help but too often discussion focuses around feature requirements from a code perspective and doesn't leave room for considerations of infrastructure. It is often discovered late in the sprint that a new feature requires a change to the Load Balancer for example and this becomes an emergency change for the operations team. To allow operations teams to plan and minimize interrupt driven work, it is necessary to ensure that they are involved in the earliest stages of design discussions.

New features need to be considered in their entirety. Design discussions can not be limited to code implementation. How does the new feature integrate into the application as a whole? What configuration changes are required? Are there new ingress/egress points? Can the existing infrastructure serve the new feature and meet performance requirements? These are all questions that need to be considered as early as possible if we are to make our operations teams more proactive than reactive.

Monitoring and Visibility

I have been thinking a lot recently about the role of monitoring in a DevOps role versus traditional operations.

Monitoring has traditionally been the purview of Operations. To the extent that Development was involved in monitoring, it was to provide the applications metrics that could indicate a problem. These would then be sucked into a monitoring platform managed by operations and appropriate alert thresholds set up with the necessary run books for how respond.

With the evolution of DevOps, production support is often a joint responsibility. Lots of tools have emerged, providing more insight into the application from a user or code perspective and these metrics can be integrated into the monitoring and alerting platform. It is not uncommon to have different escalation polices for different types of alerts, some going to developers first, others to ops first and frequently passing between the two.

This collaborative effort can lead to a much quicker resolution of production issues but it is still reactive. How can the monitoring tools we implement assist in teams proactively preventing production issues?

Visibility is often talked about as an integral part of DevOps. Tools such as Ansible or Puppet pride themselves on giving a clear view into how systems are managed. Code Coverage tools provide visibility into testing. Continuous Integration systems provide visibility into build stability. All of this helps to increase confidence among those working together to build and release systems that those systems will meet requirements: functionality, performance, security and stability.

Monitoring Tools should be seen as an integral part of this effort to provide visibility. They should present information in a simple visual format that can inform the development process.

Developers working on a new feature for example can check systems metrics to watch for issues such as a spike in CPU or memory that may indicate code inefficiencies not covered by unit tests. User simulations can indicate problems with response times deep within a web application. Are these the result of new code, or are we just seeing them now because we are looking more closely? Either way it is probably something we want to fix before releasing to production.

Taking benchmark snapshots with each release is important. Measuring against these benchmarks can be incorporated into the unit testing. When we are confident that we have a good understanding of what is optimal performance of an application we can break the build if code changes degrade performance.

Thus our monitoring tools move from something that becomes important at the end of the process -- when code is deployed to production -- to an integral part of the development process itself, in true DevOps fashion.

Defining DevOps

In this blog I will talk about all things DevOps. As the blog title suggests, these are my own views on what constitutes DevOps culture and practices. So it seems appropriate in this first post to present my own definition of DevOps.

For me DevOps describes best practices that have actually existed in the most successful agile companies for a long time. It is the process of removing walls between departments and ensuring a close collaboration between operations and engineering departments involved in the creation of systems.

What has become known as DevOps culture or practice is actually central to the success of agile development.

Agile, like the term DevOps, can mean all things to all people. According to the Agile Alliance:

In the late 1990’s several methodologies began to get increasing public attention. Each had a different combination of old ideas, new ideas, and transmuted old ideas. But they all emphasized close collaboration between the programmer team and business experts; face-to-face communication (as more efficient than written documentation); frequent delivery of new deployable business value; tight, self-organizing teams; and ways to craft the code and the team such that the inevitable requirements churn was not a crisis.

Taking this definition of Agile, DevOps becomes a logical extension of Agile. Whereas the Agile Manifesto speaks of “deployable business value” DevOps speaks of “delivering operational systems.”

The challenge is to bring operational skill sets to projects earlier in order to encourage development to think about what is required to build scalable systems that can be managed operationally.

Hiring for DevOps

Coming from the operations side of DevOps, I am seeking to recruit ops guys who “get” DevOps. It has become almost obligatory to include the term DevOps engineer in a job description, if not in the actual title, in order to differentiate from the perception of old school sys admins who act as the gate keepers of production, waiting for code to be thrown over the wall to be deployed in their Kingdom.A good candidate for working in a DevOps culture is not someone who knows Puppet or Chef or other tools. Chances are the right candidate will have a good tool set but that’s not my main criteria for hiring. A good sys admin will have actually been using the main practices contained within the DevOps model, without necessarily applying a name to it. Breaking down walls between operations and engineering, automating processes, advocating continuous integration and frequent releases. These are all things that make a sys admin’s life a little less painful.

When recruiting developers for a DevOps environment, experience with configuration management and even specific tools, Puppet, Chef etc., becomes important. Also important is an understanding of a collaborative workflow, git pull requests, discussing with operations early in the development life cycle. These ways of working are often far newer to the dev side of the house than they are to operations. Recruiting the right people on both sides of DevOps is the single most important factor in a successful implementation of DevOps. When tasked with introducing DevOps in an organization that is fully staffed, look for ways to bring in "new blood". In any company there is a natural turn over of staff, take advantage of this to introduce the right type of people to the teams. These new forces, whether on the ops side or development can then become advocates for DevOps within their own teams.

Culture

Many companies, particularly start-ups, spend a lot of time seeking to define company culture, values etc. Often this involves a top down approach. Senior executives come up with some set of values that define a company culture that is then to be adopted by the ranks. For more successful companies, those values develop organically out of the day to day collaboration of people with shared interests and goals. It is no different with DevOps culture. It is extremely helpful to have executive buy-in to a DevOps program and support for the ways of working that go with it. But trying to define a DevOps culture top down is a mistake. If we are really adopting DevOps methods and practices, the culture will flow from that.

The DevOps Team

Many blogs and presentations advise “Do NOT call your team DevOps or yourself a DevOps manager”. This arises from a couple of major concerns.

DevOps is a culture and as such requires buy-in from the entire company. No team has the single responsibility for introducing a new culture.
There is a very real danger that a DevOps team becomes just another silo between Dev and Ops.

These are very valid concerns and if you are building DevOps in a new startup I would strongly encourage you not to call a team DevOps. Implementing DevOps in a large enterprise with dispersed operations teams who have traditionally seen themselves as production gate keepers, all the while using DevOps terminology without understand it, is a little different. In such an environment, introducing a DevOps team made up of people who have "done DevOps" before can have some advantages.

There can be some value to labeling a team DevOps in order to emphasize an outlook and commitment to DevOps practices. The team can establish itself as a center of excellence for DevOps best practices across the company. The name does not automatically make it a new silo anymore than not using the name guarantees against silos.

Measuring DevOps adoption

Companies are often asked, what is the current state of adoption of DevOps. In some cases they attempt to answer this in the manner of a check list: Are we doing CI? Do we have configuration management etc.

I don’t think this approach gets us very far. A better question would be, how close are we to solving the problems we are seeking to solve by adopting DevOps? For me, DevOps is not about everyone getting along and dev and ops hanging out at the bar together. That’s often useful and desirable but more importantly what we are trying to do is work together to solve the problems which call for a DevOps culture and practices.

I’ve seen various approaches to “implementing” DevOps ranging from management sub-committees to discuss endlessly what DevOps means for a given company, what should we be doing etc., to gorilla movements by devs or more commonly ops people to enforce change from below. What’s the correct route? Whatever works for you. It’s all about breaking down barriers to get the job done.

There are many ingredients to moving to a DevOps culture but to really succeed with DevOps, I believe we have to define goals. What are we trying to achieve with the DevOps program?

My goal is to deliver scalable systems that are supportable in production. This is something that programmers, QA testers, ops people and even product managers need to be on board with and play a role in, for DevOps, and ultimately the entire company, to be successful.