The purpose of release management is to ensure that the risks associated with deploying software releases are managed.
Waterfall software
development is a phased and gated process, and so waterfall release management
is also phased and implemented as a gate. Waterfall processes also seldom use
much automation for testing and deployment, and so test reports tend to be
manually assembled. In the end, the release management gate consists of the
various stakeholders attesting that their test processes have completed
satisfactorily and that all governance artifacts have been completed.
These methods work
well for releases that occur a few times per year. However, organizations today
are moving toward “continuous delivery”, in which releases are frequent,
possibly monthly, possibly many times per day. (Amazon releases software to
production every ten seconds.) Manual processes, governed by attestation
meetings are too cumbersome for continuous delivery. A new approach is needed.
Managing Risk in a Continuous Manner
Continuous delivery
requires that nearly all software and system tests are automated, and that all
deployment processes are also automated. As such, it is expected that all of
those processes are fully tested and exercised prior to production deployment.
The primary means of managing risk for automated processes is that those
processes are tested, and the primary metric for risk management is test
coverage. This applies to all areas of risks that are amenable to automation, including
functional testing, performance testing, failure mode testing, and security
scanning.
The primary metric for risk management
is test
coverage.
The goal of continuous
delivery is that release to production should be a “business decision” – that
is, any software build that passes all of its tests should be considered
releasable, and the decision whether to release (and deploy) it is therefore
based only on whether stakeholders feel that the user community and other
business stakeholders are ready for the new release. Risk management has been
automated!
For the above process
to work, the software tests must be trustworthy: that is, there must be
confidence that the tests are accurate, and that they are adequate. Adequacy is
normally expressed as a “coverage” metric. Accuracy is typically ensured by
code review of the tests, or spot checking them, and ensuring a separation of
duties so that acceptance tests for a given code feature are not written by the
same people who write the application code for that feature. For very high risk
code, additional methods can be used to ensure accuracy. In the end, however,
tangible metrics should be used, and preferably metrics that can be measured
automatically. (See the article series, Real Agile Testing in Large Organizations.)
Is Attestation Still Needed?
In a continuous
delivery process, attestation is still needed, but the attestation should be on
the testing process – not on the result. Specifically, risk management
attestation focuses on whether the process for creating and assessing tests
ensures that the tests are accurate and that they have sufficient coverage.
Attestation does not occur for the releases themselves, because they arrive
with too much frequency. Instead, attestation is done at a process level.
Are Gates Still Needed?
Since release to production is a business decision, humans
make the decision about whether to release a given software build. In addition,
there are sometimes tests that fail, or quality criteria that are not fully
met, but release to production might still be justified. Therefore, for most
businesses, release to
production will still be governed by a gated process, even when all tests have
been automated. Release to production can only be fully automated and gateless if one automates all of the production release decision criteria and places quality control on those automated decision criteria.
What About Documents?
Some things cannot be automated. For example, design
documentation must be created by hand. Design documentation is important for
managing the risks associated with maintainability.
The continuous delivery approach to such things is to shift
assessment into the Agile sprint. As such, updating artifacts such as a design
document are part of the “definition of done” for each team’s Agile stories. To
manage the risk that these documents might not be updated, one embeds risk
mitigation practices into the Agile team’s workflow. For example, one way to
ensure that design documents are updated is to include a design review in the
workflow for the team’s Agile stories. Thus, overall assessment and attestation
of the risk should occur on the process – not on the set of documents that are
produced. If the process is shown to be robust, then when the code passes its
tests, it should be “good to go” – ready to release – assuming that releasing
it makes business sense.
The surest path to unreliability is to provide
direct access to static test environments.
What About “Lower Environments”?
In many organizations that are early in their adoption of DevOps methods and therefore use static test environments, teams push code to the various
test environments. That is a legacy practice that we plan to eliminate. In a DevOps approach, build artifacts (not code) are pulled into a test environment.
Further, each test process is performed by a single build orchestration task (VSTS task or Jenkins job), and only
that task should have the security permission required to pull into that
environment. Thus, it should not be possible to push into an environment. This
eliminates the need for any kind of release management for the lower
environments.
Many of these issues go away once one starts to use dynamically provisioned environments. Until then, it is absolutely critical that one control updates to the various test environments, using an orchestrated process as described here. The surest path to unreliability is to provide direct access to static test environments.
-->