Some time back, on my first day supporting a CIO on an Agile transformation, the CIO said that he wanted to implement DevOps. In his mind, they were the same thing - that DevOps was merely the latest Agile model for how to arrange IT functions. Indeed, it is.
But it is also different. On the one hand, Agile is defined by a set of values and principles enshrined in the Agile Manifesto. In that sense, DevOps is Agile. On the other hand, Agile has become defined by a set of practices that are almost universal to Agile implementations - things such as standups, team rooms, test-driven development, a product owner, iterations, and so on. In that sense, DevOps does change Agile, in a very substantial way.
Pipeline Versus Team
Agile project culture is very team focused. DevOps does not change that, but it shifts the focus to something broader: the end-to-end "value stream", also known traditionally as a "value chain", that consists of the sequence of activities that happen from requirement inception to actual deployment of the consequent features. That value stream is best thought of as a pipeline of activities, the term being borrowed from the computer science concept of "pipelining". If drawn, a DevOps pipeline looks like a waterfall process, consisting of requirements followed by implementation, followed by various kinds of system level testing, followed by deployment. What makes is not waterfall, however, is the fact that the pipeline operates continuously. That is, any any moment, every portion of the pipeline has something in it - it is a non-stop flow.To make a value stream work, the development team must enlarge its horizon, to consider actors beyond the team - such as enterprise security, operations, and so on. The world does not revolve around the development team: it revolves around the value pipeline, and that means there must be lots of ongoing collaboration with parties beyond the team. This is perfectly consistent with Agile values and principles, but it is a different viewpoint than is traditional for Agile in the way that it is normally practiced. For example, should those parties that are external to the development team be in the standup? Normal Agile practices do not answer that question, and there are many other similar questions that need to be answered to make DevOps work.
Behavior-Driven Development Versus Test -Driven Development
The practice of Test-Driven Development (TDD) is deeply entrenched in Agile culture. Teams that practice it are often viewed as advanced, whereas teams that do not are considered by many in the Agile community to be less advanced. Alas, TDD has come under fire, and there are many people who feel that it is not the best approach for everyone in all cases. Regardless of that, it turns out that DevOps does not need TDD for an effective pipeline. What DevOps needs is Behavior-Driven Development (BDD).Loosely speaking, BDD consists of defining tests from a user perspective. Thus, tests are "end-to-end" because they are defined in terms of the behavior of the system - not the behavior of system components or even more granular "units". Historically it has been difficult to implement a BDD approach because one needs to assemble the entire system as an integrated unit to perform the tests. However, virtualization - the technology that has enabled cloud computing and DevOps - makes it possible - even easy - to create local instances of system components, so that one can assemble an integrated system very early. That makes BDD possible. Thus, one can use a test-first approach in which tests are defined at a behavior level, instead of a very granular unit level, as is the case with TDD.
TDD is still useful, but the focus shifts to BDD.
TDD might still be useful - that is a different discussion - but it is no longer necessary for having a high coverage automated test process. One can measure test coverage for behavioral tests just as easily as for unit level tests. In fact, DevOps teams typically use BDD as the foundation of their continuous integration test cycle: developers run the BDD tests - which are end-to-end - for a story before checking their code in. This is shown in figure 1.
Figure 1: Comparison of traditional continuous integration (CI) and how CI is often done today in a DevOps setting.
To be able to perform end-to-end tests locally before checking one's code in, developers need a local build process that instantiates an integrated system locally (or perhaps in a developer-specific cloud instance - see figure 2). Linux container technology now makes this even easier: one merely starts a container for each system component and runs one's tests - starting a container takes a fraction of a second, so there is little delay from instantiating the integrated system locally. If developers work in Linux (locally or in a cloud), they can run docker and create all the containers they need in a fraction of a second, test with those, and then destroy the containers. Indeed, testing in the cloud right from the beginning of the CI process is increasingly useful given that one of the main things that needs to be tested is the orchestration definition - which is currently tied to the type of target operating environment (e.g., AWS, Azure, GCE, etc.), although there are efforts to unify orchestration definition.
Figure 2: Traditional development environment versus DevOps development environment.
Again, all this is perfectly consistent with Agile values and principles. What it conflicts with, however, is a longstanding Agile practice of focusing on comprehensive unit testing as the bedrock of the test automation cycle. DevOps does not preclude unit tests, which are still valuable, but it shifts the focus to test-first development using end-to-end tests, done early and continually, using locally provisioned instances of all system components instead of shared test instances or "mocks".
Testing the System Versus Testing Stories
The "user story" is the cornerstone of any Agile process. A story defines a requirement, from a user's perspective, in terms of an end result. Agile teams work against a backlog of stories, and a story is considered to be "done" when it has passed all of its tests; the tests in turn map one-to-one to a set of acceptance criteria that are attached to the story. A release of the system is considered to be "done" when all of its stories are "done".DevOps changes the last part. A DevOps pipeline begins with stories, but as soon as work passes through the continuous integration portion of the pipeline, the idea of a story ceases to have meaning: at that point, the entire system is being tested, and a test failure might be easy to correlate to a story or it might not be. As an example, consider figure 3: suppose that a story's tests have passed when run in the continuous integration (CI) environment, but subsequently the same tests are run again "downstream" is a more production-like environment, and one of the tests fails in that environment. Is the problem with the story, or the environment? It is not clear. Also, suppose a performance test fails: such a failure might be difficult to trace to a particular story. Thus, tests performed after continuous integration are system-level tests - not story level tests. This means that the release is not "done" when all the story tests have passed - the release is "done" when all of the story tests have passed, and all of the system level tests have passed, in all of the applicable environments. A typical full DevOps pipeline is shown in figure 3.
Figure 3: A typical full DevOps pipeline.
And again, this is consistent with Agile values and principles, but it departs from traditional Agile practice.
Is DevOps Itself a Sign Of Immaturity?
Organizations that deploy continuously have fully automated processes and so one might consider whether there are even separate "development" and "operations" teams. There are. A man who used to work for me went on to become head of deployment at Amazon for several years. In a large organization, you still need someone - a team - to focus on the challenges of operations. Leaving it to each team without any oversight or centralized support is a recipe for disintegration and a "tragedy of the commons" where things that are not urgent for a team but that are urgent for the organization as a whole fall by the wayside.Netflix has a very large "tools" team that supports all of the many different development teams. Netflix focuses on automating things, so that instead of manual monitoring they have alert systems that let the right people know when something is wrong. As Adrian Cockroft of Netflix puts it, "[dev teams] never have to have a meeting with ITops, or file a ticket asking someone from ITops to make a change to a production system", but Josh Evans is Director of Operations Engineering and he oversees the way that their hundreds of microservices are integrated, so much of operations has, in effect, been replaced by an operational architecture team: the "Ops" has shifted to defining an architecture that is highly decoupled, that alerts the right people, and that is elastic and self healing. This approach is sometimes called "NoOps" because the ops people are more engineers than operations people: they build and operate automated deployment systems that enable development teams to deploy continuously with minimal human intervention. What NoOps is not is a group of autonomous development teams that operate without any centralized operational support.
DevOps entails a considerable shift in responsibility, in that operational responsibility is now shared by development: programmers need to build systems that are easier to operate, and they need to test in production-like environments so that deployment does not reveal problems that could have been found earlier. Similarly, operations needs to focus mostly on automating operations, and needs to deploy with the same scripts and processes that are used by development. These two camps must design the pipeline together, and work together to continually improve it. The pipeline becomes their shared platform.
Some Agile Ideas That Are Even More Important Now
There is one idea that is not in the Agile Manifesto but has nevertheless been strongly embraced by the Agile community and that is even more important for DevOps: that is the idea of continuous learning. In order to implement a DevOps cloud-based testing pipeline, one must use a-lot of tools. It is a fact that those tools turn over at a rapid rate. For example, two years ago the most important tools for provisioning environments were chef, puppet, vagrant, and a few others. Today, the entire idea of provisioning environments is in question, because containers have emerged as a better alternative to VMs, and so the need to deploy software to VMs is going away - instead, one deploys container images. Those images are built locally, and so the need for remote software provisioning has been replaced by remote container management. This all happened essentially overnight, and it means that DevOps teams are going to have to eventually replace all of their chef/puppet/vagrant code with orchestration files and dockerfiles. Surely something else will take the place of these in a few years. We are now in an era in which tools change more rapidly than ever before, and teams need to accept that they are always learning new tools. The ability and willingness to abandon one's expertise and plunge into a new one is essential. This point is highlighted in Cal Newport's bestseller, Deep Work. According to him,"...because these technologies change rapidly, this process of mastering hard things never ends: You must be able to do it quickly, again and again." (page 30)
