A case for replacing environment-based feature flags with user bucket management in principle. For the longest time I think I’ve been solving the wrong problem ; progressing features through environments when I should have been progressing features through user buckets. For anyone unaware a feature flag is a way of hiding feature development work from users until its ready while preserving your deployment capability.
I cannot say how many feature flags I have under development but its more than 30; and in a world of agentic development this number seems likely to increase. That means a lot of conditional blocks, each independently maintained, with no awareness of the others while humans focus increasingly on the verification tests. When you layer on regional concerns, test versus production configuration, and customer segmentation, the complexity compounds exponentially. We (my wider team) tried the trendy thing, using Launch Darkly but it didn’t address the root problem in my view. Now I’ve come to see a new possibility; we shouldn’t be advancing features through a system. We should be advancing users through our features.
The Current State: Where It Breaks
Here’s my current setup both in reality but also in my brain. We advance features through environments. Miscommunication about what should be turned on where is frequent because it relies on multiple teams interacting. Basically this setup helps develop lots of things in parallel but it doesn’t help the deployment process.
Here’s a visual if it helps.
If we want to deploy a rebrand and make sure it integrates well, we currently put that behind a feature flag. I test it in QA, advance to UAT for others to test. When we are ready for all users in PROD environments to see the feature we turn it on. However within our setup we never do a canary releases. This would be turning the feature on in the USA but not in Australia. I don’t like doing this for several reasons:
- Because customers who travel between regions; for example a USA management company of an Australian mining operation would see different things.
- Because unless you really need independent regions you typically have reader DB’s and just one writer instance. So having potentially differing DB schemas between environments is at least potentially dangerous for data integrity.
- With so many things being developed in parelle its difficult to manage.
So while my above picture is accurate, in practice its more like this:
There are also other issues.
Spaghetti Code
A single feature requires conditional logic in multiple places across the codebase. With an example of 60 active flags, each adding two code paths, you get 120 separate decision trees. At some point the flags become so many that even asking for clear feedback about cleanup is challenging.
State Management Complexity
Test infrastructure has to be duplicated to account for flag states. We double-run test suites and annotate the test framework to be flag-aware. This doesn’t just make testing harder, it makes tests less accurate. We actually cannot test the positive state because feature flags are added to the system far in advance of them being ready. This also means that if a feature is under development for a substantial time then it can go stale.
Communication Overhead
Product, Engineering, and QA don’t have a shared source of truth about what’s in flight. Which flags are supposed to match across regions? Which ones are ready to flip? The answer typically requires pulling someone into a meeting. Release notes are manually assembled. Coordination becomes a bottleneck.
Then finally as as a part of communication issues you have customer success friction. Giving early access to a feature requires either waiting for engineering to change a flag or working around the system in ways that aren’t auditable.
User Buckets
My hypothetical solution to this problem is to adopt user buckets. This is interesting to me because Staging environments cannot ever fully match the config of PROD. For example I’ve once seen a prod outage because the cookie loading was exceeding a threashold we were unaware of. To duplicate this error in a QA env you’d need to have two google analytics accounts. This is often outside of the Dev teams control. Therefore, new cookies setup passes QA quickly and then takes out PROD. A user bucket setup allows for testing in PROD with specific users by adding them to a bucket. For example here geotags pt2 would be visible to TEST users. Such an approach would save quite a bit of operational cash (rendering STGing environments less valuable) and give a little more options for testing with real user traffic before features hit the big customers.
- TEST — Internal engineering teams and nominated test customers
- EARLY_ACCESS — LOW_TOUCH customers who want to validate new capabilities
- ENTERPRISE — All other customers, always on the stable code path
So here rather than a 5 enviornment sequence you could go to two environments with more testing stages built into the user configuration.
Testing in PROD has become quite a trendy idea of over the last 10 years in a few different flavours. Be it Meta’s turn off Italy and ensure traffic falls over in PROD to Switzerland’s infra or Netflix practice of chaos engineering.
Not accepted
Alas in this particular case my suggestions were not accepted; but it was considered. This is fine, and part of progressing in the engineering track is making technical proposals. Instead we will continue with a more vanilla outlook on feature flags. However that does leave me with a conundrum. Testing feature configurations are challenging. I do not currently have the resources to test a State A vs State B setup.
That said I have had a few evening hacking sessions to see if I could do a State A vs State B setup to at least make it possible. Luckily I did manage that and so now its funadmentally possible to swtich the test envs to mirror production config at midnight, run testing and then switch to State B in the late afternoon to allow some manual investigations on new features. There is some testing overhead here but hopefully I’ll figure something out. I’m going to take this idea of user bucket based testing and move that to my side projects.
Why this is significant as a discussion
One of the ways you can support a technical team is by adopting the DORA framework as a means of measurement. Ofc you can have no measurement method but that’s probably not healthy when pressure arrives. There have been a series of very dumb measurements in the past such as total lines of code written, number of bugs resolved etc. These are all bad. DORA proposes 4 core metrics which are complimentatry. When you adopt all 4 together you aren’t so likely to over optimize one thing and cause problems in another place; such as measuring number of bugs resolved and thus incrasing the total number of bugs written in order to resolve more. https://newsletter.pragmaticengineer.com/p/developer-productivity-with-dr-nicole
- Lead time – time to PROD with new features
- Deployment Frequency – how often you move things into prod
- Change failure rate – how often it goes wrong in PROD
- Change failure reversion time – how quickly you fix that
Since I’ve worked on this subject I’ve mostly been able to affect Deployment Frequency and I’ve taken the team from 20day average down to 5. This is not perfect, but it is a lot better. And feature flags have been one of the most important tools to acheive this.
Anything Mapping 
Leave a comment