Develop on Cadence. Release on Demand.
—A SAFe mantra
Release on Demand
Release on Demand is the process by which features deployed into production are released incrementally or immediately to Customers based on market demand.
Release on Demand is the fourth and last element in the four-part Continuous Delivery Pipeline of Continuous Exploration (CE), Continuous Integration (CI), Continuous Deployment (CD) and Release on Demand, as can be seen in Figure 1.
The processes that precede Release on Demand, CE and CI, help ensure that we always have value ready to be deployed in the production environment. But since tangible development value occurs only when end users are successfully operating the Solution in their environment, releasing value at the right time is critical for the enterprise to gain the true benefits of agility. This is why the decision to release should never be taken lightly. While there may be situations where deployed features are immediately available to all end users, more often the release is a decoupled on-demand activity, occurring for specific users, timed for when they need it.
Note: This article is the companion article to Develop On Cadence.
The ability to Release on Demand is integral to the Continuous Delivery Pipeline. It preserves three critical options:
- When should we release?
- What elements of the system should be released?
- Which end users should receive the release?
Having all three alternatives accelerates and creates possibility and flexibility in delivering value.
Further, Release on Demand allows us to ‘close the loop’ and evaluate the benefit hypothesis proposed during the CE stage. It provides the information needed to decide whether further exploration of an idea is warranted, or if new functionality should be added, or perhaps a different approach is warranted. Achieving the goals of Release on Demand requires:
- An understanding of how to decouple the release from the development cadence
- An understanding of how to decouple Solution elements from a monolithic release
- The ability to architect and re-architect as necessary to support incremental, and partial, delivery
Each of these is described in the following sections.
Decouple on Cadence. Release on Demand
Develop on Cadence is the other half of a strategy that allows Agile Release Trains (ARTs) and the Solution Trains to operate in a predictable pattern and synchronize the efforts of multiple development teams. But when it comes to actually releasing value, a different set of rules may apply. Once we have a reliable stream of value through the Continuous Deployment process, the next and even larger consideration is when and how to actually release all that accumulating value. The release strategy is decoupled from the development cadence, as can be seen in Figure 2.
There are many release strategies that may apply in different situations:
Releasing on the Program Increment Cadence. The simplest case is when an enterprise can release on the Program Increment (PI) boundaries. All release dates are known in advance, and PI planning, releasing, and Inspect and Adapt (I&A) are coordinated by the same cadence and calendar dates. In addition, Innovation and Planning Iterations can be timed, designed, and coordinated to support the more extensive release activities. This can include final Verification and Validation (V&V), User Accepted Testing (UAT), release documentation, etc.
Releasing Less Frequently. In many cases, however, releasing on a fast PI cadence may not be possible, or even desirable. For example, in some enterprise settings, deployed systems constitute critical infrastructure for an operating environment. Even if the customer would like to have the new software, the service level and license agreements may be prohibitive. And then there is the overhead and disruption of installation. In other cases, the timelines of enterprises that are building systems with both software and hardware (mobile phones or geophysical mapping satellites) are driven by their long-lead hardware items (displays, chip sets, and the like). The new hardware must be available first, which means releasing early and incrementally is not an option. In these cases, releasing on a PI cadence may not be practical, and the planning and releasing activities may be completely decoupled.
Releasing More Frequently. For many, the goal is simply to release as frequently as possible—hourly, daily, weekly, whatever. Achieving that requires the right DevOps capability, an effective Continuous Delivery Pipeline, and an architecture that supports incremental releases. Even then, the periodic planning function still provides the cadence, synchronization, and alignment the enterprise needs to manage variability and limit deviations from expectations.
Release On Demand. For enterprises building complex solutions (e.g., systems of systems), the cases above are probably overly simplistic. Big systems are not homogeneous. They contain different types of components and subsystems, each of which may leverage its own release model. In that case, the most general model is: Release whatever you want, and whenever it makes sense within the governance and business model.
Decouple Release Elements from the Solution
The understanding that we might have different release frequencies also begs another question: Is a release a monolithic, all-or-none process? If so, the release strategy would be limited. Fortunately, that’s not the case. In fact, even fairly simple solutions will have multiple release elements, each operating on different release strategies, as Figure 3 illustrates.
For example, the SAFe website that’s hosting this article has multiple release cycles:
- We can make a fix to deployment or security at any time (ad hoc, but expedited).
- We can update any article at any time, and simply notify readers via blog post (high frequency).
- We can add new articles to the Guidance section whenever significant new content is available (medium frequency).
- But we update the major revision to the framework only periodically, based on new content, new ways to render the Big Picture, and most importantly, the readiness of the market for a major new release (low frequency).
We call these separate flows “Value Streamlets,” as they represent a full, end-to-end flow of value within the Value Stream. Each streamlet can and should be managed to deliver value according to its own needs and pace. Identifying those streamlets is critical to enable Release on Demand, as they allow the different elements of the solution to be released independently on their own cadence. They also allow for value stream mapping and improving each streamlet separately.
Architect the Solution for Incremental Release
To achieve these release strategies, and to decouple release elements, the system must be designed for component or service-based deploy-ability, release-ability, and fast recovery. Designing intentionally to realize these attributes requires the collaboration of System Architects, Agile Teams, and deployment operations. The goal is a fast and flawless delivery process, ideally requiring little manual effort. Capabilities to achieve this more flexible release process include:
Feature toggles –These enable deployment of features into production without making them immediately available to the user. They avoid the need for multiple code branches, allowing developers to deploy new features to production, but to activate them only when the situation warrants. It’s important to note, that if feature toggles are not cleaned up and removed after they are deemed stable in production, they can become a source of technical debt and reduce the agility of the trains.
Canary Releases – Feature toggles also help enable “canary releases,” which are releases made available only to a subset of users. This allows selective production testing and feature validation without impacting the entire user base.
Dark launches – In some situations, it’s important to test new functionality in a production setting before making it available to customers. This is primarily the case where it’s uncertain how well the system will meet its Nonfunctional Requirements (NFRs) under production load. Again, feature toggles help manage this risk.
In order to create these capabilities, the organization will typically need to undertake certain enterprise-level architectural initiatives. These may include moving to common technology stacks, decoupling monolithic applications with micro-services, data preparation and normalization, third-party interfaces and data exchange, logging and reporting tools, and more.
‘Close the Loop’ on the Feature hypothesis
Finally, after the feature is released, it’s time to evaluate the benefit hypothesis. Was achieved the intended outcome achieved? For example, should a canary release be extended to more customers? Turn off the feature toggles? Once we have an understanding of the outcome of the release, that feature is ‘done.’
Building the Release
Building large-scale systems that are ready for release and fit for use is a process most easily considered in stages, as Figure 4 illustrates.
The first, and hopefully, continuous step in this process is that each Agile Team ensures that it produces a working increment for the Stories, Features, and components they are responsible for. They do this by:
- Completing user stories from their Team Backlog, assuring that each meets its story (local) Definition of Done (DoD)
- Applying Continuous Integration practices with test automation to monitor and ensure progress
Every two weeks, teams build a system increment, an integrated stack of new functionality. This is the sum total of all the backlog items completed by release train teams during the current and all previous Iterations. Each system increment is the subject of the System Demo. In this way, new Features are added to the system, one story at a time, typically a few per program increment.
In large solutions, ARTs typically build only a part of the solution, whether it’s a set of subsystems, some solution capabilities, or a mix. For that reason, the real progress of the solution development effort can only be objectively evaluated based on the solution increment that comprises all the capabilities collaboratively delivered by all ARTs in the Solution Train. Integrated, verified, and validated, they must satisfy both the functional and Nonfunctional Requirements (NFRs). This solution increment is the subject of the all-important Solution Demo, which brings all the system increments together into a working system.
Building the release in this incremental fashion mainly affects the release of the solution as a whole. As discussed above, many elements can and should be decoupled from the solution to be released independently. These elements might be at the team, system, or solution levels; so the same logic should be applied across the entire integration stack.
Some activities might be required when releasing. These include Validation and Verification (V&V), release notes, User Accepted Testing (UAT), final documentation, and finalized training materials, as well as marketing activities.
As much as possible, these activities should be included in the DoD of previous increments. Some, however, will remain as release activities.
A Scaled Definition of Done
The continuous buildup of system functionality, along with the final solution itself and the ongoing V&V of its elements, can be reflected in a scaled DoD, as shown below in Table 1.
Release management is the process of planning, managing, and governing Solution releases, which helps guide the value stream toward the business goals. In some enterprises, especially those with significant regulatory and compliance criteria, this is a centralized, portfolio-level team or function that assures releases meet all the relevant business criteria. In other circumstances, release management and governance is a responsibility shared by the leadership of the ART or Solution Train, where stakeholders from development operations, quality, sales and other stakeholders take part in the responsibility.
In either case, the release management function helps coordinate and facilitate the activities necessary to help internal and external stakeholders receive and deploy the new solution. It also ensures that the most critical governance quality elements have been appropriately addressed prior to deployment —particularly internal and external security, regulatory, and other compliance guidelines.
Release planning happens during PI planning. But planning is the easy part; the hard part is coordinating the implementation of all the capabilities and features over the multiple Iterations within a release. This is especially true when new issues, roadblocks, dependencies, over-scopes, and gaps in Vision and Backlogs are uncovered. This is the challenge for the ARTs; the scope of each release must be continually managed, revalidated, and communicated. Primary considerations include:
- Ensuring that the organization’s release governance is understood and adhered to
- Communicating release status to external stakeholders
- Ensuring that an appropriate deployment/distribution plan is in place
- Coordinating with marketing and with Product and Solution Management on internal and external communications
- Validating that the solution meets relevant quality and Compliance criteria
- Participating in Inspect and Adapt (I&A) to improve the release process, value stream productivity, and solution quality
- Providing final authorization for the release
- Acting as a liaison with Lean Portfolio Management (LPM), as appropriate
- Participating in, and overseeing, the final release activities
Many enterprises hold release management meetings weekly to address the following questions:
- Is the vision still understood, and are the trains and teams aligned to that purpose?
- Does everyone understand what they are building, and is it aligned with the understanding of the purpose of the value stream and current Strategic Themes?
- Are the scheduled releases still largely tracking?
- Is the appropriate quality built in to the solution?
- What impediments must be addressed to facilitate progress?
The weekly meeting provides senior management with regular visibility into the release status. This meeting is also the place to approve any scope, timing, or resource adjustments necessary to ensure the release. In a more continuous delivery environment, the participants monitor the release section of the Program Kanban. Their role is making sure items are released when needed, to the right audiences, managing dark and canary releases, verifying that hypotheses are evaluated and that feature toggles, etc. are removed after they have been verified.
Learn More Kim, Gene and Jez Humble, Patrick Debois, John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press.
Last update: 3 May, 2017