This guide outlines the process Tai teams should follow to assess, document, and escalate issues related to a production feature flag that may need to be turned off.
Step 1: Triage the Issue
Before escalating, you should gather basic evidence and evaluate impact.
Checklist:
-
Identify the affected feature.
-
Confirm that the issue is directly tied to the feature flag.
-
Reproduce the issue, if possible, in Beta.
-
Determine if there’s a workaround.
Step 2: Gather Key Impact Metrics
Fill out or submit the following key data points (can be logged in a ticket or Slack message):
Field |
Description |
---|---|
Feature Flag ID |
What is the ID of the feature flag? |
Date enabled |
When was the feature turned on in production? |
Affected users/accounts |
How many users and accounts are affected? Any strategic customers? |
Support ticket volume |
How many tickets or reports have been logged? |
Severity |
How disruptive is the issue? |
Workaround |
Is there any temporary fix available for users? |
Screenshots or logs |
Attach if available for faster debugging. |
Step 3: Score the Situation Using the Triage Matrix
Use this matrix to objectively evaluate whether a feature flag should be turned off. Each row is scored independently. Sum the values to guide the decision.
Criteria |
Question |
Scoring |
Notes |
---|---|---|---|
User Impact |
How many end users are affected? (Out of ~6,000) |
0 = <10 users |
Based on feature usage |
Account Impact |
How many customer accounts are affected? (Out of ~400) |
0 = 0 accounts |
Based on feature usage |
Severity |
How disruptive is the issue? |
0 = Cosmetic, no impact |
|
Time Since Launch |
How long has the feature been enabled in production? |
0 = 3+ weeks |
Check in TMS Admin > Application Feature Flags and sort by "enabled" and last toggled. |
Core Feature? |
Is this tied to a Top 25 page or essential workflow? |
0 = None |
Check Top 25 Pages here |
Mitigation Available? |
Can the issue be resolved with a fix-forward approach? |
0 = Patch already deployed |
Disabling may be necessary if no fix is feasible in short term. |
-
If the score is 13 or higher: escalate immediately as High Priority and turn the feature flag off.
-
If the score is 8–12: escalate with full context for product team to decide.
-
If the score is 7 or lower: continue monitoring; no immediate action.
Step 4: Escalate to Product Team
If escalation is needed, notify the Product Team in #Product:
Support has flagged a potential issue with [Feature Name and ID] (enabled on [Date]).
Issue Score: [XX] / 18
Affected users: [summary]
Affected accounts: [summary]
Ticket count: [X]Summary: [brief description of issue and impact]
Screenshots/logs: [link or upload]
Requesting evaluation for potential feature flag deactivation.
Step 5: Stay Aligned
After escalation:
-
Wait for product team to confirm next steps (disable, patch, monitor).
-
Update support tickets with the status and expected resolution.
-
Notify customers if needed.
Do Not:
- Turn off a feature flag without product confirmation if score is 8–12.
- Escalate without completing impact assessment.