Your app is finally in production, and your team continues to work on new features. But wait! There’s a critical bug that needs immediate attention, or you’ll face an instant flood of negative reviews from users.
You quickly Slack your team and hear not-so-great news: they are in the middle of the sprint, and there already are new, untested features on their main branch that can’t be released to production yet.
Take a look at the following image, which presents this frequent project situation in the GIT repository:
The obvious solution is to just wait for the next stable release, with new features and the bugfix C tested together with features B and D. In this scenario the wait time could be days, which as you might imagine, can have dire consequences: lost users, negative reviews, lost revenue. Overall – not good.
The second option is to create a release branch and copy the commit from the main one to it and release a new production version out of it.
Unfortunately, this solution doesn’t come without drawbacks. Often the bugfix "C" uses some piece of code from feature "B" so the dev needs to somehow extract it to create the bugfix "C" on the release branch.
What if you wanted to release feature "D" because the QA team managed to test it quickly enough? You’re probably getting a headache already at this point! Software development is hard, isn’t it?
As you can see, there are a lot of moving parts, risks, and points of failure which all make your developers a little less sure about the whole process and a little bit more cautious.
You may also like: Python 3.11. New features and improvements you should know
This in turn leads to significantly less frequent releases. It’s also difficult to maintain a process like this so in the majority of teams there is a single god-level developer who does all of that and if they disappeared one day, everything would just fall apart.
According to Accelerate State of DevOps 2022 Report high performing organizations: deploy multiple times a day; have a significantly lower lead time for changes and failure rate.
But seeing stuff like this, you tend to ask yourself: how could we, mere mortals, achieve that? We’re not Github, we don’t want to be part of the cargo cult. We need quick and easy implementations to spend time producing actual features.
The solution
Fortunately, there’s a seemingly very simple solution to all of this (of course there is, why would I write an article about it otherwise?). Its name is "feature flagging"!
Feature flagging is a very simple technique that allows one to enable or disable a feature without the need to modify the code. But in reality, it’s much more than that.
It’s virtually a tool to decouple code deployments from feature releases. You as a developer focus on writing code and remove yourself from the discussion on the order of which features should be released next to your users.
In the end, you can think of feature flags as variables that are either true or false, and you use them in your code to write various 'if' statements that either render something or not. That’s it.
Step 1: Select a feature flagging system
Remember, this is the only condition that we need to meet: "enable or disable a feature without the need to modify the code".
Environmental variables: quick implementation but imperfect
This would be the easiest way to approach it! Just define environment variables in whatever deployment environment you use and use them in your code base.
If you are developing a web app or a mobile app, it’s probably a good idea to create a /api/feature-flags endpoint on your backend and return values of those flags so that the frontend can also render proper UI.
3rd party feature flagging service
A much better approach is to use a specialized tool that allows everyone in the team to change the values of feature flags and your app just connects to it and downloads current values when necessary.
This way you don’t have to redeploy your app every time you want to enable a feature. It also creates a single source of truth, which is very hard to achieve with env variables.
If you want to use a cheap approach, you could for example use a Firebase Remote Config; or you could also use something more advanced, like Launchdarkly, which has a huge set of features that make the job even easier.
Might be interesting: How to manage AWS ECS environment variables with Chamber?
Step 2: Modify your development process, so everyone is aware of feature flags
Making your whole team aware of the existence of feature flags in your codebase and ensuring they are maintained properly by everyone is crucial. Everyone needs to understand the flow of the ticket: from the day it is put into the backlog to the day its related feature flag is enabled in production (and later archived!).
Here’s an example of the process that you could follow, based on Jira, but to be honest, it should work in every issue tracking product out there.
Plan
During refinement or planning meetings decide if the ticket should be feature flagged. Usually, all new features or major modifications to existing features should be feature flagged.
Develop
During the sprint, a developer picks up the ticket and creates a feature branch in GIT and a feature flag in relevant software. At this point, the feature flag is enabled for the development environment and is off for both staging and production.
Merge
Developers finish their work on the feature, it goes through regular review and is merged into the master branch. People reviewing and testing the feature should check both scenarios: with the flag enabled and disabled.
The status of the ticket is changed to “Merged to master”.
Assign version number
Now comes the fun part! Tag your commit with a version number (for example according to the semantic versioning system) and assign it to all tickets that currently have the “Merged to master” status. In Jira, it’s possible to use the “fix version” field for example.
Deploy to staging
At this point either manually or through CI/CD, the master branch should be deployed to all relevant staging environments. And nobody should observe any real change. This is what is called a “dark launch”! You just deployed the code, which didn’t change the behavior of your application.
Change the status of the ticket to “Deployed to Staging”.
Test
Next, manually enable the feature flag on your first staging environment and perform your usual QA process. Whether it’s automatic or manual is really up to you and your team.
When your QA process is done, change the status of the ticket to “Release candidate”. This informs everyone that the feature is tested, and the flag can be enabled as soon as the assigned version has been deployed to production.
Deploy to production
Just deploy the new version to production as soon as you finish sanity checks on your staging environments. If you didn’t find any critical issues, there’s no reason to wait. Remember, this version can contain various bug fixes that didn’t feature flagged, and you want them available in production as quickly as possible.
Change the status of the ticket to “Done”. Sit down with your team and debate the definition of done to be certain that this is the right time for you to do that. Each team is different!
Worth checking: AWS X-Ray - A Distributed Tracing System for Debugging Applications
Step 3: Remove old feature flags from your codebase
I can’t stress enough how important it is to remove feature flags from your code as soon as they are enabled on production and you’re sure they are working fine. If you don’t, you’ll end up with a bunch of old screens and components that you practically don’t have in your app but need to maintain. Don’t let the tech debt build up! I like to create small tickets in Jira that remind me to go through them and clean up every sprint.
Step 4: Just keep deploying to production
Don’t be afraid to deploy frequently! Enable features you are certain are ready and keep the ones that are not disabled. Now your deployments and releases are decoupled.
Just don’t forget to introduce new feature flags for major changes to existing, even unreleased, features if you’re developing a mobile app. Your users can have different versions of the app! You wouldn’t like to enable an incomplete feature for someone who forgot to update their app, wouldn’t you?