My DevOps Reading List

Here I am publishing some of the titles that have most influenced my thinking on DevOps in the hopes that others find it useful. The books are ordered by year of publication and where possible I provide a link to Amazon.com or other sites they where they can be purchased. I will try to update this list as I read more or remember other works that influenced me.

• The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps (2005)
• Kevin Behr, Gene Kim, George Spafford
• Continuous Integration: Improving Software Quality and Reducing Risk (2007)
• Paul M. Duvall, Steve Matyas, Andrew Glover
• Release It: Design Production-Ready Software (2007)
• Michael T. Nygard
• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (2010)
• Jez Humble, David Farley
• Personal Kanban: Mapping Work | Navigating Life (2011)
• Jim Benson, Tonianne DeMaria Barry 
• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win (2013)
• Gene Kim, Kevin Behr, George Spafford
• Kanban from the Inside: Understand the Kanban Method, connect it to what you already know, introduce it with impact (2014)
• Mike Burrows
• The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations (2016)
• Gene Kim, Patrick Debois, John Willis , Jez Humble, 

 John Allspaw

Book Review: The DevOps Handbook

The DevOps Handbook by Gene Kim, Jez Humble, Patrick Debois and John Willis is the product of five years of intensive research, conferences, discussion with IT professionals, including between the authors themselves.

The Handbook builds on a number of other significant works addressing the problems which DevOps seeks to fix that have been published over more than a decade, including but not limited to:

• The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps
• Gene Kim, Kevin Behr and George Spafford  (2005) 
• Continuous Integration: Improving Software Quality and Reducing Risk 
• Paul M Duvall, Steve Matyas and Andrew Glover (2007) 
• Continuous Delivery: Reliable Software Releases through Build, Test and Deployment Automation 
• Jez Huble and David Farley (2010) 
• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win
• Gene Kim, Kevin Behr, George Spafford (20123)

It also draws heavily on the annual Puppet Labs State of DevOps Reports which examine what contributes to the development of High-performing IT organizations.

The preface documents the Aha! moments of each of the authors that were critical in bringing them to what is now knows as DevOps. Gene Kim writes about the outsourced IT Operations of a large airline company with its "large, annual software releases: each of which would cause immense chaos and disruption" resulting in the layoffs of talented people. Kim writes:

"The sense of hopelessness and futility that resulted created for me the beginnings of a crusade. Development seemed to always be viewed as strategic, but IT Operations was viewed as tactical, often delegated away or outsourced entirely, only to return in five years in worse shape than it was first handed over."

Each of the authors tell similar stories of thinking there must be a better way. Jez Humble cites a project with ThoughtWorks where the team automated deployments. "That project inspired a lot of the ideas in both the Continuous Integration book and this one," Humble writes. Patrick Debois writes that for him it was a "collection of moments" beginning with a data center migration project in 2007 and culminating in the 2009 Velocity Conference with the "10 Deploys Per Day" presentation by John Allspaw and Paul Hammond. John Willis writes about when he first met Luke Kanies, the founder of Puppet Labs at an O'Reilly open source conference.

The book clearly draws upon the vast collective experience of the authors in seeking to "describe how to replicate the DevOps transformations we've been a part of or observed." The rest of the preface focuses on dispelling common myths about DevOps such as it only being for startups.

In his forward, John Allspaw, the CTO of Etsy gives his view that the handbook represents a "synthesis" of a "diverse set of perspectives" on the origins of the DevOps movement.

The introduction gives a very good overview of DevOps and the problems it is trying to solve. It also points to the rapidly acceleration trend towards "faster, cheaper, low-risk-delivery of software." It includes a table that details the declining risk from the 1970s and '80s to the 1990s and 2000 to the present pointing to the evolution of both infrastructure, from Mainframes to Client/Server and now Commoditization on the Cloud, and software development languages from COBOL to C++ to Java, Ruby on Rails, PHP etc.

In elaborating the business value of DevOps, the introduction sketches out the value of DevOps for both business as a whole and individual teams. Introducing the six parts to the book the authors write:

"The purpose of the DevOps Handbook is to give you the theory, principles, and practices you need to successfully start your DevOps initiative and achieve your desired outcomes."

 In my opinion the authors have certainly succeeded in this purpose. The handbook avoids a discussion of tools or even specific technical implementations in favor of more fundamental practices such as moving from large to small batch delivery, or the importance of the Deployment Pipeline.

Part one of the handbook is titled Agile, Continuous Delivery, and the Three Ways. Readers familiar with The Phoenix Project will know that it presents the Three Ways as the set of principles upon which all the observed patterns and behaviors of DevOps are based and this point is reiterated by the authors of the handbook.

After introducing the Three Ways, the book gives a very thorough guide to starting a DevOps transformation in part two, appropriately titled Where To Start. This section talks about picking the right value stream to give early successes, citing real world experiences from the retail company Nordstrom.  It offers valuable tips for how to expand DevOps across the organization. The handbook gets into detail on "organizational archetypes", reviewing the pros and cons of functional versus market-oriented teams and discusses various DevOps patterns such as embedding operations engineers into the development teams and assigning an ops liaison to each service team.

The bulk of the book is taken up with an elaboration of The Three Ways from the standpoint of technical practices. In this way it elaborates technical principles that form the bedrock of a successful DevOps program. For example in part three The Technical Practices of Flow we find a discussion of the deployment pipeline, automated testing, continuous integration, automation and continuous delivery. All of this is backed up by detailed case studies.

A similar level of detail is giving to The Second Way in The Technical Practice of Feedback and The Third Way,  The Technical Practices of Continual Learning and Experimentation. This is followed by a discussion on integrating Information Security, Change Management and Compliance into DevOps.

I would recommend a careful reading of this valuable work for anyone involved in or thinking about a DevOps transformation. This is equally true for managers and individual contributors. DevOps is gaining more momentum in the enterprise thanks to events such as the DevOps Enterprise Summit, now in its third year as well as the numerous DevOps Days that have taken place since the first one organized by Patrick Debois in Belgium - Ghent in 2009. The case studies provide ample evidence of the benefits of DevOps as well as the consequences of ignoring it.

One minor gripe I have with an otherwise excellent book is the presentation of the Endnotes. They provide valuable sources for each chapter with page numbers referenced by a quote from the text. Here's an example from the Introduction:

xxv      Dr. Eliyahu M. Goldratt...      Goldrat, Beyond the Goal.

One can find what this refers to by going to the page and looking for that reference but there is no indication of the existence of the note while reading the chapter. On discovering the Notes section, I at first thought that the superscript numbers normally indicating the presence of end notes had been accidentally eliminated in production but it seems this was a design decision, a bad one in my opinion, but don't let that stop you from buying the book.

Why I created DevOps Advocates

After recently leaving my full time gig as a DevOps Manager in a large enterprise I had some time to think about my future career goals and where I fit within the DevOps ecosystem.

I've worked at a number of interesting companies over the years and as regular readers of my blog will know, I have developed my own opinions of what DevOps is and how to do it right.

Surveying thousands of job posts with "DevOps" in the title, it struck me that few, if any, spoke about culture. Job posts for release engineers, automation engineers, systems administrators, even software engineers and developers are all commonly prepended with DevOps. These posts call for skills that have existed years before the term DevOps was coined, so what makes them DevOps?

This got me thinking about some of the difficulties I have encountered in trying to bring DevOps and more generally agile concepts into companies large and small. The problems I encountered were never around tools but culture. Sure, there maybe issues to address such as tool adoption, the use of pull requests to help collaboration, Kanban boards and so on but that was never the source of the problems.

I've seen various approaches, embedded DevOps, DevOps teams, transformations of existing teams and in all cases the cultural aspect of DevOps is a much bigger issue than the tools. Why is this?

At DevOps Days Boston there was an excellent speech by Katie Rose titled GridIronOps: What Women's Tackle Football Can Teach Us About Competitive Software. In the course of this unique and lively presentation of the cultural challenges in DevOps, Katie made the point that one similarity between DevOps and women's football is that we are trying to change the culture of adults. Young boys learn the collaborative required for successful team work at early age but these are grown women taking up the sport as adults.

This comparison left a powerful impression on me. Coming into a company to "implement DevOps" inevitable meets with cultural resistance. If you build a separate DevOps team it is immediately perceived as a challenge to other teams that existed before you got there. If you start merging operations people with development teams then you are bringing together what are, more likely than not, conflicting ways of working. If you are a development manager seeking to take on automation and configuration management, there is the danger that you will be seen as a threat by the operations team. If you are an operations team trying to encourage developers to take on their own deployments you run the risk of being seen as trying to pass off responsibilities. All of some of theses problems will arise in the course of trying to implement DevOps.

So, to get back to the title of this post, why did I create DevOps Advocates and what is it? Thinking about how to avoid some of the problems cited above, I was reminded of the early days of agile and the concept of agile coaching. Companies that were most successful with agile often hired outside consultants to come in and review existing workflows. These consultants would then work with the teams to figure out where agile would fit within the business. What were they doing right? and wrong?

The use of a third party avoided the defensiveness one often sees in trying to move a company over to a DevOps culture and practice. The agile coach was seen as an independent voice who's only agenda was to improve process and make a company more successful. So I decided there is a need for DevOps coaching along the same lines as the Agile coaching of an early age.

I am seeking out consulting gigs from companies who need more than tools. Lots of companies have intelligent people who, with a little nudge, can become excellent advocates and practitioners of DevOps. The companies I am interested in are those willing to have an independent assessment of where they are currently on their DevOps journey and where they need to get. Initial conversations at DevOps Days and other meetups tell me there is a demand for what I am pitching. It's early days so only time will tell if I am right but it would be stupid not to give it a try.

For more information or to get in touch visit http://devopsadvocates.com/.

What DevOps can learn from Kanban

One of the great strengths of Kanban over other Agile methodologies is its great flexibility. For operations, the ability to incorporate interrupts easily is essential. While the goal of any operations team should always be to plan as much as possible, the fact is interrupts do happen. Kanban suggests itself as a natural system for organizing operations workflow but it is far more than that. I believe there is much to be learned from Kanban for anyone leading a DevOps program.

Principles of Kanban

The principles of Kanban align closely with the culture of DevOps. One of the best and more detailed expositions of the Kanban method I have read recently is Kanban from the Inside by Mike Burrows.

Burrows avoids introducing the Kanban method from its Foundation Practices and Core Principles in favor of a system of nine values which he lists as follows:

• Transparency
• Balance
• Collaboration
• Customer Focus
• Flow
• Leadership
• Understanding
• Agreement
• Respect

These nine values could well be listed as the components of a DevOps culture. Burrows goes on to elaborate four foundational principles which he labels as follows:

FP1: Start with what you do now.
FP2: Agree to pursue evolutionary change
FP3: Initially, respect current processes, roles, responsibilities, and job titles.
FP4: Encourage acts of leadership at every level in your organization -- from individual contributor to senior management.

Again these are very useful foundation principles, not only for Kanban but for DevOps in general. FP2 and FP3 on the pursuit of evolutionary change and respecting initial processes are particularly important. More often than not a company turns to DevOps in recognition of broken practices and procedures. Few companies are completely homogeneous and different teams will have their own ideas about what must change and how quickly. The best recipe for success with ah DevOps program is not to seek drastic changes overnight but to encourage new processes as the culture develops.

The DevOps toolset can help tremendously with this this. A move towards configuration management implies a change in practice by both operations and development teams. Self enablement of developers implies changes to roles, responsibilities and in some cases even job titles. But we need the tools in place before we can change process. This is how I interpret FP3: "Initially, respect current processes." Once the tools are in place and we have buy in from some teams and individuals, it becomes easier to convince others.

Encouraging acts of leadership is also central to the DevOps culture. It encourages individuals at every level of an organization to take ownership of the product. Not simply their part of it but the whole product.

Burrows also elaborates six Core Practices as follows:

CP1: Visualize.
CP2: Limit Work-in-Progress (WIP).
CP3: Manage flow.
CP4: Make policies explicit.
CP5: Implement feedback loops.
CP6: Improve collaboratively, evolve experimentally (using models and the scientific method).

As with the foundation principles, these core practices can be of enormous benefit to a DevOps program. Let's look at how each one of these can assist with DevOps.

Kanban and DevOps

Visualize

A friend of mine asked me recently how one measures the success of a DevOps program and the first thing that came to mind was the need for visualization. The Kanban board is the the most basic visualization tool. Anyone interested can see the progression of work through the system. They can see what is blocked, what is in progress and get a rough approximation of when their "most urgent of all tasks"  are likely to be completed. In addition I like to produce metrics relating to team velocity that can illustrate improvements resulting from the introduction of DevOps.

Of course  the Kanban board doesn't account for all the work being done by a team. While there certainly should be a card associated with every task, there are other metrics that become useful when one is implementing infrastructure as code. Tools such as github make it simple to generate reports relating to code checkins, pull requests etc. This should be part of your velocity metrics, as should mean time to acknowledge alerts from your monitoring system and mean time to resolve production issues.

This is all part of the Visualization advocated by Kanban. Make it clear what your team does and where they are at in the workflow.

Limit WiP

Operations teams and even some development teams are invariably pulled every which way by demands from multiple stakeholders. Every stakeholder naturally thinks their issue is the most important issue of the day. Limiting Work-in-Progress in a very visual way can be a useful asset in explaining why people sometimes just need to be patient.

By taking stakeholders to the Kanban board and pointing out the current issues in progress, we can initiate the conversation about what should be dropped in favor of that most urgent issue you are being asked to work on. It is then possible to explain to the person demanding you drop everything that they need to talk to the stakeholders affected if you do so.

Feedback Loops

The success of "pushing back" as I suggest above will be heavily affected by the feedback loops in place. If stakeholders are left wondering if their request is just sitting in some backlog somewhere to never be reviewed, the above push back will not be very successful. We need to provide continuous feedback. Could the stakeholder have done anything different that would make the processing of the request smoother? Could we have done anything different? All of this relates strongly to the first principle of Kanban, transparency. Be totally transparent about your workflows, WiP limits and team velocity. It will pay big dividends in the end.


Collaboration

It should not be necessary to explain to DevOps folks the relationship of CP6 to DevOps. It is the very heart of a DevOps culture. We need to improve collaboratively, both within our team and between teams. That is really what DevOps is about.

Conclusion

In conclusion I want to emphasize that I have presented here only a brief synopsis of ideas of Mike Burrows as elaborated in his work Kanban from the Inside. I strongly advise everyone involved with DevOps to read this book. There is much to learn from the agile movement and Kanban in particular.

When you can't see the forrest for the trees

How often do you work at a company and see only the bad? I see this a lot. In operations particularly we are dealing with so many problems that we often lose site of the awesome, cool stuff we are doing.

I was at a meet-up recently where a company was demoing some orchestration tools they had built for wrangling AWS. This was of particular interest to me because my present company has spent a lot of time doing just that. As a primary user of these tools there has been a lot of frustration. Infrastructure orchestration is complex. Any tool built will have bugs. When you are the one fighting through the bugs, you tend to have a very negative opinion of a tool set. But getting out there and seeing what others are doing can be very refreshing.

This particular tool that was being presented was very good as far as it went. But I soon realized it was doing about ten percent of what we are doing in my present company. The purpose of this short post is not to brag about what we are doing -- although it really is pretty awesome! My intent is to caution against getting bogged down by the problems.

The problems are a way of life and operations people see more of them than any other team. My advice; take a step back. Get out there on the meet-up circuit, look at what others are doing and you may well get a new perspective on how awesome your company and your team really is.

After some 30 years in the IT industry I can say with some authority that all companies have their fair share of good and bad. I can also say that the bad tends to dominate in any bar room discussion. We need to do a better job of advertising the good. The DevOps movement has helped a lot with this but we can do more.

Look for the cool in what you are doing and talk about it more. It will make you feel good and others around you.

Defining DevOps

In this blog I will talk about all things DevOps. As the blog title suggests, these are my own views on what constitutes DevOps culture and practices. So it seems appropriate in this first post to present my own definition of DevOps.

For me DevOps describes best practices that have actually existed in the most successful agile companies for a long time. It is the process of removing walls between departments and ensuring a close collaboration between operations and engineering departments involved in the creation of systems.

What has become known as DevOps culture or practice is actually central to the success of agile development.

Agile, like the term DevOps, can mean all things to all people. According to the Agile Alliance:

In the late 1990’s several methodologies began to get increasing public attention. Each had a different combination of old ideas, new ideas, and transmuted old ideas. But they all emphasized close collaboration between the programmer team and business experts; face-to-face communication (as more efficient than written documentation); frequent delivery of new deployable business value; tight, self-organizing teams; and ways to craft the code and the team such that the inevitable requirements churn was not a crisis.

Taking this definition of Agile, DevOps becomes a logical extension of Agile. Whereas the Agile Manifesto speaks of “deployable business value” DevOps speaks of “delivering operational systems.”

The challenge is to bring operational skill sets to projects earlier in order to encourage development to think about what is required to build scalable systems that can be managed operationally.

Hiring for DevOps

Coming from the operations side of DevOps, I am seeking to recruit ops guys who “get” DevOps. It has become almost obligatory to include the term DevOps engineer in a job description, if not in the actual title, in order to differentiate from the perception of old school sys admins who act as the gate keepers of production, waiting for code to be thrown over the wall to be deployed in their Kingdom.A good candidate for working in a DevOps culture is not someone who knows Puppet or Chef or other tools. Chances are the right candidate will have a good tool set but that’s not my main criteria for hiring. A good sys admin will have actually been using the main practices contained within the DevOps model, without necessarily applying a name to it. Breaking down walls between operations and engineering, automating processes, advocating continuous integration and frequent releases. These are all things that make a sys admin’s life a little less painful.

When recruiting developers for a DevOps environment, experience with configuration management and even specific tools, Puppet, Chef etc., becomes important. Also important is an understanding of a collaborative workflow, git pull requests, discussing with operations early in the development life cycle. These ways of working are often far newer to the dev side of the house than they are to operations. Recruiting the right people on both sides of DevOps is the single most important factor in a successful implementation of DevOps. When tasked with introducing DevOps in an organization that is fully staffed, look for ways to bring in "new blood". In any company there is a natural turn over of staff, take advantage of this to introduce the right type of people to the teams. These new forces, whether on the ops side or development can then become advocates for DevOps within their own teams.

Culture

Many companies, particularly start-ups, spend a lot of time seeking to define company culture, values etc. Often this involves a top down approach. Senior executives come up with some set of values that define a company culture that is then to be adopted by the ranks. For more successful companies, those values develop organically out of the day to day collaboration of people with shared interests and goals. It is no different with DevOps culture. It is extremely helpful to have executive buy-in to a DevOps program and support for the ways of working that go with it. But trying to define a DevOps culture top down is a mistake. If we are really adopting DevOps methods and practices, the culture will flow from that.

The DevOps Team

Many blogs and presentations advise “Do NOT call your team DevOps or yourself a DevOps manager”. This arises from a couple of major concerns.

DevOps is a culture and as such requires buy-in from the entire company. No team has the single responsibility for introducing a new culture.
There is a very real danger that a DevOps team becomes just another silo between Dev and Ops.

These are very valid concerns and if you are building DevOps in a new startup I would strongly encourage you not to call a team DevOps. Implementing DevOps in a large enterprise with dispersed operations teams who have traditionally seen themselves as production gate keepers, all the while using DevOps terminology without understand it, is a little different. In such an environment, introducing a DevOps team made up of people who have "done DevOps" before can have some advantages.

There can be some value to labeling a team DevOps in order to emphasize an outlook and commitment to DevOps practices. The team can establish itself as a center of excellence for DevOps best practices across the company. The name does not automatically make it a new silo anymore than not using the name guarantees against silos.

Measuring DevOps adoption

Companies are often asked, what is the current state of adoption of DevOps. In some cases they attempt to answer this in the manner of a check list: Are we doing CI? Do we have configuration management etc.

I don’t think this approach gets us very far. A better question would be, how close are we to solving the problems we are seeking to solve by adopting DevOps? For me, DevOps is not about everyone getting along and dev and ops hanging out at the bar together. That’s often useful and desirable but more importantly what we are trying to do is work together to solve the problems which call for a DevOps culture and practices.

I’ve seen various approaches to “implementing” DevOps ranging from management sub-committees to discuss endlessly what DevOps means for a given company, what should we be doing etc., to gorilla movements by devs or more commonly ops people to enforce change from below. What’s the correct route? Whatever works for you. It’s all about breaking down barriers to get the job done.

There are many ingredients to moving to a DevOps culture but to really succeed with DevOps, I believe we have to define goals. What are we trying to achieve with the DevOps program?

My goal is to deliver scalable systems that are supportable in production. This is something that programmers, QA testers, ops people and even product managers need to be on board with and play a role in, for DevOps, and ultimately the entire company, to be successful.