Application Architecture Summit

Modern Application Development for Digital Business Success

REGISTER

Online Conference

The frequency in which new releases are being deployed is increasing dramatically.  It appears DevOps practices are taking hold more widely.

But, before we pop the champagne, there’s a frustratingly related stat. We’re seeing such escalating frequency doesn’t always translate to efficiency. Sadly, in fact, there are signs that rapid releases can actually raise the risk of software failures.

Evidence is mounting from two different surveys that the pressure for DevOps teams to deploy features more frequently is having a negative impact on the caliber and efficiency of the features being released.

However, not all is dark.  In this post, we’ll review the evidence and share some tips your organization can use to keep DevOps efforts on track – and significantly decrease in the amount of downtime and issues caused by frequent feature releases.

First, let me share some findings from these surveys that illustrate the depth of the issue – and how widespread this phenomenon has become.

Here at Split.io, we conducted a survey of Operations / SRE / DevOps professionals.  Entitled “The State of Feature Delivery,” our survey revealed that 67% of organizations are releasing new features every two weeks. Even more dramatic - nearly 20% are releasing new features daily.

Our survey included more than 100 responses from a wide range of companies. The survey showed that the big-named Fortune 100 companies (like Walmart and Netflix) are working with apps in terms of hundreds of releases a month. 

And while that may not sound like a big surprise, what was eye-popping from the survey is that even smaller firms are not that far behind the big guys. Everyone appears to be pushing their DevOps pedal down hard as they can.

This increase in the number and frequency of releases is a clear indication of the advancements within DevOps teams and their ability to release new code faster.  And that should be a good thing. 

But our survey revealed something else – something not so good.Unfortunately, the survey also revealed that an astounding 82% of organizations commonly discover bugs in production. Even worse, fixing those bugs can take more than a day to tackle for some 40% of organizations.

This matters a lot to productivity and business outcomes.  As an IDC research report put it: Just one hour of downtime can cost organizations upwards of $500,000!  This means extended downtime caused by bugs could easily cause millions of dollars in revenue losses.

Our Split survey also revealed that 27% of organizations experience downtime after releasing a feature. No surprise then that rollbacks and hotfixes are becoming more common practice in DevOps. To put a number on this trend, Split’s survey found 41% of DevOps teams rollback or hotfix more than 10% of new features.

This leads me to another recent piece of research. 

The well-respected Accelerate State of DevOps Report from DORA (DevOps Research and Assessment) recently revealed that almost one-fourth (23%) of organizations that deploy code weekly admit to experiencing some type of software failure.

So, in 2020, evidence is mounting about DevOps inconvenient truth. So, not to leave the reader on a down note, let’s explore what can be done – without disrupting the great momentum DevOps is building up.

Demand from customers for new features is only going to increase. So that is a strong case to support the DevOps approach to rapid updates and releases.  Given that reality – both for software and for business -- there are some things you can do.

Let’s look at three main tips you can implement to assist the growing need for speed when it comes to deployment of new features.

The first tip is Gradual Rollouts.

With a gradual rollout, a new feature is rolled out to small subsets of users first, allowing for previously undetected bugs and other issues to be discovered in a way that impacts fewer users. This is what is meant by “limiting the blast radius” of unforeseen issues. Practical patterns for gradual rollouts include releasing new features to only internal users and/or beta testers, and then progressively to larger portions of users. Feature flags are an ideal tool to use for gradual rollouts, as they provide granular targeting of a release to specific subsets of your users, and changes to targeting can be made instantly without a new deployment.

The second tip is Integrated Feature Monitoring.

This technique requires participants to view application monitoring and error tracking from the context of the gradual rollout itself. By default, traditional monitoring will miss issues in an early stage of a gradual rollout due to weak signals. The fix is to compare user behavior and system health between the user populations that are in an early rollout and the rest of the population, looking for significant differences. This practice dramatically improves the speed and efficiency of detecting bugs and issues at an early stage because it can simultaneously detect small changes in performance or errors and determine the root cause (i.e. exactly which feature of many is causing the observed problems).

And a final tip (for now) is Simply Avoid Unnecessary Work.

Sounds logical for teams looking for speed. But how does a team do this without sacrificing quality?  Our answer is to focus engineering impact on outcomes rather than output.  

In the push to accelerate releases, sometimes DevOps participants forget that features are built to achieve business outcomes, and online experimentation can measure whether customers exposed to a new feature actually advance those outcomes.

So, instead of checking the “done” box when code ships, it’s a more powerful feedback loop to only call it “done” if the impact is achieved. If multiple iterations still aren’t achieving the goal, then teams can refocus on more fertile areas.

Dave Karow is a continuous development evangelist at Split.io. He regularly speaks and writes on aligning progressive delivery (i.e. gradual rollouts of new code) with observability of system health, user experience, and user behavior.  He has three decades of experience with developer tools, communities and ways to make software delivery more sustainable – most recently with CI/CD and DevOps. Before Split.io, he democratized "shift left" performance testing at BlazeMeter.