My DevOps Reading List

Here I am publishing some of the titles that have most influenced my thinking on DevOps in the hopes that others find it useful. The books are ordered by year of publication and where possible I provide a link to Amazon.com or other sites they where they can be purchased. I will try to update this list as I read more or remember other works that influenced me.

• The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps (2005)
• Kevin Behr, Gene Kim, George Spafford
• Continuous Integration: Improving Software Quality and Reducing Risk (2007)
• Paul M. Duvall, Steve Matyas, Andrew Glover
• Release It: Design Production-Ready Software (2007)
• Michael T. Nygard
• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (2010)
• Jez Humble, David Farley
• Personal Kanban: Mapping Work | Navigating Life (2011)
• Jim Benson, Tonianne DeMaria Barry 
• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win (2013)
• Gene Kim, Kevin Behr, George Spafford
• Kanban from the Inside: Understand the Kanban Method, connect it to what you already know, introduce it with impact (2014)
• Mike Burrows
• The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations (2016)
• Gene Kim, Patrick Debois, John Willis , Jez Humble, 

 John Allspaw

Book Review: The DevOps Handbook

The DevOps Handbook by Gene Kim, Jez Humble, Patrick Debois and John Willis is the product of five years of intensive research, conferences, discussion with IT professionals, including between the authors themselves.

The Handbook builds on a number of other significant works addressing the problems which DevOps seeks to fix that have been published over more than a decade, including but not limited to:

• The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps
• Gene Kim, Kevin Behr and George Spafford  (2005) 
• Continuous Integration: Improving Software Quality and Reducing Risk 
• Paul M Duvall, Steve Matyas and Andrew Glover (2007) 
• Continuous Delivery: Reliable Software Releases through Build, Test and Deployment Automation 
• Jez Huble and David Farley (2010) 
• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win
• Gene Kim, Kevin Behr, George Spafford (20123)

It also draws heavily on the annual Puppet Labs State of DevOps Reports which examine what contributes to the development of High-performing IT organizations.

The preface documents the Aha! moments of each of the authors that were critical in bringing them to what is now knows as DevOps. Gene Kim writes about the outsourced IT Operations of a large airline company with its "large, annual software releases: each of which would cause immense chaos and disruption" resulting in the layoffs of talented people. Kim writes:

"The sense of hopelessness and futility that resulted created for me the beginnings of a crusade. Development seemed to always be viewed as strategic, but IT Operations was viewed as tactical, often delegated away or outsourced entirely, only to return in five years in worse shape than it was first handed over."

Each of the authors tell similar stories of thinking there must be a better way. Jez Humble cites a project with ThoughtWorks where the team automated deployments. "That project inspired a lot of the ideas in both the Continuous Integration book and this one," Humble writes. Patrick Debois writes that for him it was a "collection of moments" beginning with a data center migration project in 2007 and culminating in the 2009 Velocity Conference with the "10 Deploys Per Day" presentation by John Allspaw and Paul Hammond. John Willis writes about when he first met Luke Kanies, the founder of Puppet Labs at an O'Reilly open source conference.

The book clearly draws upon the vast collective experience of the authors in seeking to "describe how to replicate the DevOps transformations we've been a part of or observed." The rest of the preface focuses on dispelling common myths about DevOps such as it only being for startups.

In his forward, John Allspaw, the CTO of Etsy gives his view that the handbook represents a "synthesis" of a "diverse set of perspectives" on the origins of the DevOps movement.

The introduction gives a very good overview of DevOps and the problems it is trying to solve. It also points to the rapidly acceleration trend towards "faster, cheaper, low-risk-delivery of software." It includes a table that details the declining risk from the 1970s and '80s to the 1990s and 2000 to the present pointing to the evolution of both infrastructure, from Mainframes to Client/Server and now Commoditization on the Cloud, and software development languages from COBOL to C++ to Java, Ruby on Rails, PHP etc.

In elaborating the business value of DevOps, the introduction sketches out the value of DevOps for both business as a whole and individual teams. Introducing the six parts to the book the authors write:

"The purpose of the DevOps Handbook is to give you the theory, principles, and practices you need to successfully start your DevOps initiative and achieve your desired outcomes."

 In my opinion the authors have certainly succeeded in this purpose. The handbook avoids a discussion of tools or even specific technical implementations in favor of more fundamental practices such as moving from large to small batch delivery, or the importance of the Deployment Pipeline.

Part one of the handbook is titled Agile, Continuous Delivery, and the Three Ways. Readers familiar with The Phoenix Project will know that it presents the Three Ways as the set of principles upon which all the observed patterns and behaviors of DevOps are based and this point is reiterated by the authors of the handbook.

After introducing the Three Ways, the book gives a very thorough guide to starting a DevOps transformation in part two, appropriately titled Where To Start. This section talks about picking the right value stream to give early successes, citing real world experiences from the retail company Nordstrom.  It offers valuable tips for how to expand DevOps across the organization. The handbook gets into detail on "organizational archetypes", reviewing the pros and cons of functional versus market-oriented teams and discusses various DevOps patterns such as embedding operations engineers into the development teams and assigning an ops liaison to each service team.

The bulk of the book is taken up with an elaboration of The Three Ways from the standpoint of technical practices. In this way it elaborates technical principles that form the bedrock of a successful DevOps program. For example in part three The Technical Practices of Flow we find a discussion of the deployment pipeline, automated testing, continuous integration, automation and continuous delivery. All of this is backed up by detailed case studies.

A similar level of detail is giving to The Second Way in The Technical Practice of Feedback and The Third Way,  The Technical Practices of Continual Learning and Experimentation. This is followed by a discussion on integrating Information Security, Change Management and Compliance into DevOps.

I would recommend a careful reading of this valuable work for anyone involved in or thinking about a DevOps transformation. This is equally true for managers and individual contributors. DevOps is gaining more momentum in the enterprise thanks to events such as the DevOps Enterprise Summit, now in its third year as well as the numerous DevOps Days that have taken place since the first one organized by Patrick Debois in Belgium - Ghent in 2009. The case studies provide ample evidence of the benefits of DevOps as well as the consequences of ignoring it.

One minor gripe I have with an otherwise excellent book is the presentation of the Endnotes. They provide valuable sources for each chapter with page numbers referenced by a quote from the text. Here's an example from the Introduction:

xxv      Dr. Eliyahu M. Goldratt...      Goldrat, Beyond the Goal.

One can find what this refers to by going to the page and looking for that reference but there is no indication of the existence of the note while reading the chapter. On discovering the Notes section, I at first thought that the superscript numbers normally indicating the presence of end notes had been accidentally eliminated in production but it seems this was a design decision, a bad one in my opinion, but don't let that stop you from buying the book.

Why I created DevOps Advocates

After recently leaving my full time gig as a DevOps Manager in a large enterprise I had some time to think about my future career goals and where I fit within the DevOps ecosystem.

I've worked at a number of interesting companies over the years and as regular readers of my blog will know, I have developed my own opinions of what DevOps is and how to do it right.

Surveying thousands of job posts with "DevOps" in the title, it struck me that few, if any, spoke about culture. Job posts for release engineers, automation engineers, systems administrators, even software engineers and developers are all commonly prepended with DevOps. These posts call for skills that have existed years before the term DevOps was coined, so what makes them DevOps?

This got me thinking about some of the difficulties I have encountered in trying to bring DevOps and more generally agile concepts into companies large and small. The problems I encountered were never around tools but culture. Sure, there maybe issues to address such as tool adoption, the use of pull requests to help collaboration, Kanban boards and so on but that was never the source of the problems.

I've seen various approaches, embedded DevOps, DevOps teams, transformations of existing teams and in all cases the cultural aspect of DevOps is a much bigger issue than the tools. Why is this?

At DevOps Days Boston there was an excellent speech by Katie Rose titled GridIronOps: What Women's Tackle Football Can Teach Us About Competitive Software. In the course of this unique and lively presentation of the cultural challenges in DevOps, Katie made the point that one similarity between DevOps and women's football is that we are trying to change the culture of adults. Young boys learn the collaborative required for successful team work at early age but these are grown women taking up the sport as adults.

This comparison left a powerful impression on me. Coming into a company to "implement DevOps" inevitable meets with cultural resistance. If you build a separate DevOps team it is immediately perceived as a challenge to other teams that existed before you got there. If you start merging operations people with development teams then you are bringing together what are, more likely than not, conflicting ways of working. If you are a development manager seeking to take on automation and configuration management, there is the danger that you will be seen as a threat by the operations team. If you are an operations team trying to encourage developers to take on their own deployments you run the risk of being seen as trying to pass off responsibilities. All of some of theses problems will arise in the course of trying to implement DevOps.

So, to get back to the title of this post, why did I create DevOps Advocates and what is it? Thinking about how to avoid some of the problems cited above, I was reminded of the early days of agile and the concept of agile coaching. Companies that were most successful with agile often hired outside consultants to come in and review existing workflows. These consultants would then work with the teams to figure out where agile would fit within the business. What were they doing right? and wrong?

The use of a third party avoided the defensiveness one often sees in trying to move a company over to a DevOps culture and practice. The agile coach was seen as an independent voice who's only agenda was to improve process and make a company more successful. So I decided there is a need for DevOps coaching along the same lines as the Agile coaching of an early age.

I am seeking out consulting gigs from companies who need more than tools. Lots of companies have intelligent people who, with a little nudge, can become excellent advocates and practitioners of DevOps. The companies I am interested in are those willing to have an independent assessment of where they are currently on their DevOps journey and where they need to get. Initial conversations at DevOps Days and other meetups tell me there is a demand for what I am pitching. It's early days so only time will tell if I am right but it would be stupid not to give it a try.

For more information or to get in touch visit http://devopsadvocates.com/.

DevOps, OpsDev or a bit of both?

At a recent meetup a group of us were joking around with the term OpsDev because in our experience DevOps is more often than not pushed from the operations side rather than development. So the question came up "why do the devs get to come first?" Googling the term OpsDev later I discovered it is a real thing.

The UK website Business Computing World carries a passionate piece by Randy Clark from 2012 in which he argues:

For me, in whichever context Ops is talked about, the industry is still getting it back to-front. It should be OPSdev, OPSweb, OPScloud. Ops isn’t just important; it’s essential. Putting Ops second is a sign that although businesses realise its role, they don’t fully understand the far reaching effect of Ops.

This HP white paper from 2013 states:

After engaging with many enterprises, we learned that the adoption of DevOps practices and methodologies in large enterprises is often driven by IT operations, and not necessarily by the developers of the Line of Businesses. We call this OpsDev.

The paper goes on to talk about how DevOps and OpsDev often address the same business problems but they are often driven by different concerns. I found this image a particularly interesting illustration of this.


None of the goals listed here under DevOps can be realized without those under OpsDev and vice versa. This is the key to understanding the collaboration component of the DevOps culture. For better or worse, Patrick Debois chose to call his conference DevOpsDays in 2009 and the now infamous twitter hash tag #DevOps was borne. Along with it emerged a massive cultural movement that can not be turned back.

What is DevOps?

For a lot of people in operations, myself included, it was difficult to get what the hype was about back in 2009. Moreover the question a lot of ops folks asked is "Why does dev have to come first?"
Automation, continuous integration and deployment, collaboration between ops and dev, aren't these all just operations best practices? This is what good Sys Admins have advocated for years. Add to this the marketing hype attached to DevOps in recent years and we begin to see some justification for the skepticism.

But the movement has proved the skeptics wrong in a number of ways. The DevOps movement brought to the fore frustrations felt within many engineering organizations. Developers were frustrated with traditional operations teams inability to respond with the perceived urgency demanded. Operations people were equally frustrated with being treated like human keyboards and other people's poor planning becoming their emergency. What had been discussed for years over beers now became the topic of industry wide conferences. Tools began to emerge to help solve some of these problems in a more generalized way.

The DevOps movement enabled software engineers and engineering managers to begin thinking about what goes on after the code is deployed. The Agile movement had taken us a long way towards this with automated testing and continuous delivery. Developers became more involved in deployments but it was still the responsibility of the operations team to run and maintain production. The proverbial wall was still there and the code was essentially still thrown over it. Where it landed was the responsibility of operations.

Virtualization and cloud computing has taken us further along the road of true collaboration. It has made possible the full realization of Infrastructure as Code. This in turn has made it possible to enable developers to provision their own infrastructure. Some have interpreted this as NoOps and even gone so far as to eliminate their operations teams. That is a big mistake.

DevOps + Cloud do not eradicate the need for operations. Rather they illustrate the need for OpsDev -- Operations people who get DevOps alongside DevOps -- Developers who get DevOps. I am not proposing here to start using the term OpsDev in place of DevOps. It is way too late for a name change here. What I am trying to illustrate is that DevOps practices and culture need to be embraced by both operations teams and development teams. Once the teams themselves are collaborating better the next challenge is the cross team collaboration. When we begin to bring together the acquired skills and best practices of operations with those of software engineering we can build some truly awesome systems. This is what DevOps done right enables.

What DevOps can learn from Kanban

One of the great strengths of Kanban over other Agile methodologies is its great flexibility. For operations, the ability to incorporate interrupts easily is essential. While the goal of any operations team should always be to plan as much as possible, the fact is interrupts do happen. Kanban suggests itself as a natural system for organizing operations workflow but it is far more than that. I believe there is much to be learned from Kanban for anyone leading a DevOps program.

Principles of Kanban

The principles of Kanban align closely with the culture of DevOps. One of the best and more detailed expositions of the Kanban method I have read recently is Kanban from the Inside by Mike Burrows.

Burrows avoids introducing the Kanban method from its Foundation Practices and Core Principles in favor of a system of nine values which he lists as follows:

• Transparency
• Balance
• Collaboration
• Customer Focus
• Flow
• Leadership
• Understanding
• Agreement
• Respect

These nine values could well be listed as the components of a DevOps culture. Burrows goes on to elaborate four foundational principles which he labels as follows:

FP1: Start with what you do now.
FP2: Agree to pursue evolutionary change
FP3: Initially, respect current processes, roles, responsibilities, and job titles.
FP4: Encourage acts of leadership at every level in your organization -- from individual contributor to senior management.

Again these are very useful foundation principles, not only for Kanban but for DevOps in general. FP2 and FP3 on the pursuit of evolutionary change and respecting initial processes are particularly important. More often than not a company turns to DevOps in recognition of broken practices and procedures. Few companies are completely homogeneous and different teams will have their own ideas about what must change and how quickly. The best recipe for success with ah DevOps program is not to seek drastic changes overnight but to encourage new processes as the culture develops.

The DevOps toolset can help tremendously with this this. A move towards configuration management implies a change in practice by both operations and development teams. Self enablement of developers implies changes to roles, responsibilities and in some cases even job titles. But we need the tools in place before we can change process. This is how I interpret FP3: "Initially, respect current processes." Once the tools are in place and we have buy in from some teams and individuals, it becomes easier to convince others.

Encouraging acts of leadership is also central to the DevOps culture. It encourages individuals at every level of an organization to take ownership of the product. Not simply their part of it but the whole product.

Burrows also elaborates six Core Practices as follows:

CP1: Visualize.
CP2: Limit Work-in-Progress (WIP).
CP3: Manage flow.
CP4: Make policies explicit.
CP5: Implement feedback loops.
CP6: Improve collaboratively, evolve experimentally (using models and the scientific method).

As with the foundation principles, these core practices can be of enormous benefit to a DevOps program. Let's look at how each one of these can assist with DevOps.

Kanban and DevOps

Visualize

A friend of mine asked me recently how one measures the success of a DevOps program and the first thing that came to mind was the need for visualization. The Kanban board is the the most basic visualization tool. Anyone interested can see the progression of work through the system. They can see what is blocked, what is in progress and get a rough approximation of when their "most urgent of all tasks"  are likely to be completed. In addition I like to produce metrics relating to team velocity that can illustrate improvements resulting from the introduction of DevOps.

Of course  the Kanban board doesn't account for all the work being done by a team. While there certainly should be a card associated with every task, there are other metrics that become useful when one is implementing infrastructure as code. Tools such as github make it simple to generate reports relating to code checkins, pull requests etc. This should be part of your velocity metrics, as should mean time to acknowledge alerts from your monitoring system and mean time to resolve production issues.

This is all part of the Visualization advocated by Kanban. Make it clear what your team does and where they are at in the workflow.

Limit WiP

Operations teams and even some development teams are invariably pulled every which way by demands from multiple stakeholders. Every stakeholder naturally thinks their issue is the most important issue of the day. Limiting Work-in-Progress in a very visual way can be a useful asset in explaining why people sometimes just need to be patient.

By taking stakeholders to the Kanban board and pointing out the current issues in progress, we can initiate the conversation about what should be dropped in favor of that most urgent issue you are being asked to work on. It is then possible to explain to the person demanding you drop everything that they need to talk to the stakeholders affected if you do so.

Feedback Loops

The success of "pushing back" as I suggest above will be heavily affected by the feedback loops in place. If stakeholders are left wondering if their request is just sitting in some backlog somewhere to never be reviewed, the above push back will not be very successful. We need to provide continuous feedback. Could the stakeholder have done anything different that would make the processing of the request smoother? Could we have done anything different? All of this relates strongly to the first principle of Kanban, transparency. Be totally transparent about your workflows, WiP limits and team velocity. It will pay big dividends in the end.


Collaboration

It should not be necessary to explain to DevOps folks the relationship of CP6 to DevOps. It is the very heart of a DevOps culture. We need to improve collaboratively, both within our team and between teams. That is really what DevOps is about.

Conclusion

In conclusion I want to emphasize that I have presented here only a brief synopsis of ideas of Mike Burrows as elaborated in his work Kanban from the Inside. I strongly advise everyone involved with DevOps to read this book. There is much to learn from the agile movement and Kanban in particular.

When you can't see the forrest for the trees

How often do you work at a company and see only the bad? I see this a lot. In operations particularly we are dealing with so many problems that we often lose site of the awesome, cool stuff we are doing.

I was at a meet-up recently where a company was demoing some orchestration tools they had built for wrangling AWS. This was of particular interest to me because my present company has spent a lot of time doing just that. As a primary user of these tools there has been a lot of frustration. Infrastructure orchestration is complex. Any tool built will have bugs. When you are the one fighting through the bugs, you tend to have a very negative opinion of a tool set. But getting out there and seeing what others are doing can be very refreshing.

This particular tool that was being presented was very good as far as it went. But I soon realized it was doing about ten percent of what we are doing in my present company. The purpose of this short post is not to brag about what we are doing -- although it really is pretty awesome! My intent is to caution against getting bogged down by the problems.

The problems are a way of life and operations people see more of them than any other team. My advice; take a step back. Get out there on the meet-up circuit, look at what others are doing and you may well get a new perspective on how awesome your company and your team really is.

After some 30 years in the IT industry I can say with some authority that all companies have their fair share of good and bad. I can also say that the bad tends to dominate in any bar room discussion. We need to do a better job of advertising the good. The DevOps movement has helped a lot with this but we can do more.

Look for the cool in what you are doing and talk about it more. It will make you feel good and others around you.

Some thoughts on planning

Anyone working in the IT industry has seen, heard or said this phrase more than once. Unfortunately, if you in operations then someone else's inability to plan more than likely DOES become your emergency.This remains true even in companies preaching DevOps. The larger and more complex the company, the greater the chances that there is a team somewhere, Operations, DevOps, whatever the name, who's workload is driven almost entirely by interrupts.

These teams often adopt Kanban as a means of making the problem visible but Kanban alone won't fix the problem. Kanban highlights the demands being made upon an operations team and can be useful in improving the work flow but the real problem is in planning.

DevOps at its heart is about collaboration between teams, avoiding silos etc. In a few startups DevOps results in single teams made up of developers and operations people who are in on the plan from the beginning and everyone gets to celebrate at the same time due to a common understanding of "done". At its best DevOps encourages an understanding that a feature is only completed when it is running in production and in the hands of the end user.

In far too many cases, however, companies proclaiming they are doing DevOps have not redefined their definition of "done". In these companies, development teams consider work done when the code builds successfully, or when QA has signed off on it. The task of making this run in production is not considered part of the development process, as a result, what is required to make and keep this running in production is often not part of the plan.

The need for planning is certainly understood by software development teams and forms an integral part of agile methodologies such as Scrum. Developers would revolt if they spent four hours in a planning meeting only to be told three days into a two week sprint, "forget the plan, something has come up and we need you to do this instead." Yet it is considered perfectly acceptable to assign tasks to an operations team towards the end of a two sprint for work that could easily have been predicted in planning.

DevOps Planning

Clearly there is a need for better planning. Many organizations seek to integrate operations people into development sprint planning sessions. This can help but too often discussion focuses around feature requirements from a code perspective and doesn't leave room for considerations of infrastructure. It is often discovered late in the sprint that a new feature requires a change to the Load Balancer for example and this becomes an emergency change for the operations team. To allow operations teams to plan and minimize interrupt driven work, it is necessary to ensure that they are involved in the earliest stages of design discussions.

New features need to be considered in their entirety. Design discussions can not be limited to code implementation. How does the new feature integrate into the application as a whole? What configuration changes are required? Are there new ingress/egress points? Can the existing infrastructure serve the new feature and meet performance requirements? These are all questions that need to be considered as early as possible if we are to make our operations teams more proactive than reactive.