Build a Progressive Feature Rollout System

A progressive rollout reduces risk by exposing new features to a small percentage of users first, catching problems before they affect everyone. Instead of a full launch, you increase traffic gradually — from 1% to 5% to 25% and beyond — monitoring application health at each stage and rolling back if metrics degrade.

What you will build

Shipping a feature to 100% of users on day one is risky. A progressive rollout lets you release a feature to a small percentage of users, monitor for problems, and gradually increase exposure until you reach full availability.

By the end of this tutorial, you will have:

• A feature flag with variables that control the new feature’s behavior
• Rollout rules that target specific audiences at controlled traffic percentages
• A monitoring strategy that gates each traffic increase on application health metrics
• Code that handles rollback gracefully when metrics degrade

Before you start

This tutorial builds on the concepts and SDK setup from the Implement Feature Flags tutorial. Complete that tutorial first if you have not yet created a feature flag or integrated the SDK into your application.

Ensure you have:

• Feature Experimentation access — Log into app.optimizely.com
• Your SDK key — Found in Settings > Environments
• A running application — With the Optimizely SDK already initialized
• Monitoring in place — Error tracking (Sentry, Datadog, etc.) and application metrics you can observe during the rollout

Step 1: Define the feature and its variables

Start by identifying what the feature flag will control. A good rollout flag is more than a boolean toggle — it includes variables that let you adjust the feature’s behavior without deploying code.

In the Optimizely app:

1. Navigate to Features > Feature Flags
2. Click Create New Feature
3. Set the key to new_search_engine
4. Add variables that control the feature:

Variable key	Type	Default value	Purpose
max_results	Integer	20	Number of search results to display
enable_typo_correction	Boolean	true	Whether to auto-correct typos in queries
ranking_algorithm	String	"relevance_v2"	Which ranking model to use
cache_ttl_seconds	Integer	300	How long to cache search results

5. Click Save

Variables give you levers to turn during the rollout. If the new ranking algorithm causes problems, you can switch to a different value without turning off the entire feature.

Step 2: Create the flag check in your code

Integrate the feature flag into your application. The flag check should wrap the entire new code path.

Feature flag integration

JavaScript C# (.NET 8+) Python

javascript Copy

import { createInstance } from '@optimizely/optimizely-sdk';

const optimizely = createInstance({ sdkKey: process.env.OPTIMIZELY_SDK_KEY });
await optimizely.onReady();

async function handleSearchRequest(userId, query, userAttributes) {
  const user = optimizely.createUserContext(userId, userAttributes);
  const decision = user.decide('new_search_engine');

  if (decision.enabled) {
    const config = {
      maxResults: decision.variables.max_results,
      typoCorrection: decision.variables.enable_typo_correction,
      rankingAlgorithm: decision.variables.ranking_algorithm,
      cacheTtl: decision.variables.cache_ttl_seconds,
    };
    return await newSearchEngine(query, config);
  }

  return await legacySearchEngine(query);
}

csharp Copy

using OptimizelySDK;

public class SearchService
{
    private readonly Optimizely _optimizely;

    public SearchService(Optimizely optimizely)
    {
        _optimizely = optimizely;
    }

    public async Task<SearchResults> HandleSearchAsync(
        string userId, string query, Dictionary<string, object> attributes)
    {
        var user = _optimizely.CreateUserContext(userId, attributes);
        var decision = user.Decide("new_search_engine");

        if (decision.Enabled)
        {
            var variables = decision.Variables.ToDictionary();
            var config = new SearchConfig
            {
                MaxResults = decision.Variables.GetValue<int>("max_results"),
                TypoCorrection = decision.Variables.GetValue<bool>("enable_typo_correction"),
                RankingAlgorithm = decision.Variables.GetValue<string>("ranking_algorithm"),
                CacheTtl = decision.Variables.GetValue<int>("cache_ttl_seconds"),
            };
            return await NewSearchEngineAsync(query, config);
        }

        return await LegacySearchEngineAsync(query);
    }
}

python Copy

from optimizely import optimizely

client = optimizely.Optimizely(sdk_key='YOUR_SDK_KEY')

def handle_search_request(user_id, query, user_attributes):
    user = client.create_user_context(user_id, user_attributes)
    decision = user.decide('new_search_engine')

    if decision.enabled:
        config = {
            'max_results': decision.variables['max_results'],
            'typo_correction': decision.variables['enable_typo_correction'],
            'ranking_algorithm': decision.variables['ranking_algorithm'],
            'cache_ttl': decision.variables['cache_ttl_seconds'],
        }
        return new_search_engine(query, config)

    return legacy_search_engine(query)

Step 3: Add telemetry around the flag

Before rolling out, instrument both code paths so you can compare their performance side by side.

Telemetry instrumentation

JavaScript

javascript Copy

async function handleSearchRequest(userId, query, userAttributes) {
  const user = optimizely.createUserContext(userId, userAttributes);
  const decision = user.decide('new_search_engine');
  const searchPath = decision.enabled ? 'new' : 'legacy';

  const startTime = performance.now();

  try {
    const results = decision.enabled
      ? await newSearchEngine(query, extractConfig(decision))
      : await legacySearchEngine(query);

    const duration = performance.now() - startTime;

    // Emit metrics for comparison
    metrics.histogram('search.duration_ms', duration, { path: searchPath });
    metrics.counter('search.success', 1, { path: searchPath });
    metrics.histogram('search.result_count', results.length, { path: searchPath });

    // Track in Optimizely for experiment analysis
    user.trackEvent('search_completed', {
      value: duration,
    });

    return results;
  } catch (error) {
    const duration = performance.now() - startTime;
    metrics.counter('search.error', 1, { path: searchPath, error: error.code });
    metrics.histogram('search.duration_ms', duration, { path: searchPath });

    // Fall back to legacy on error
    if (decision.enabled) {
      return await legacySearchEngine(query);
    }
    throw error;
  }
}

Key metrics to track at every stage:

• Latency — p50, p95, p99 response times for both code paths
• Error rate — Percentage of requests that fail
• Business metrics — Search result click-through rate, conversion rate
• Resource usage — CPU, memory, database connection pool utilization

Step 4: Configure the initial rollout rule (1%)

In the Optimizely app:

1. Open your new_search_engine feature flag
2. Select the Production environment
3. Click Add Rule > Targeted Delivery
4. Name it: Progressive Rollout — Phase 1
5. Set the audience to Everyone (or a specific internal audience for the first phase)
6. Set traffic allocation to 1%
7. Click Save, then Publish

Starting at 1% lets you validate the integration in production with real traffic while limiting blast radius if something breaks.

Step 5: Validate at 1%

Wait at least 1 hour at 1% traffic. Check:

• No increase in error rates compared to the baseline
• Latency p95 is within 10% of the legacy path
• No new exceptions in your error tracker
• Search results are returning valid data

If any check fails, pause the rollout and investigate. Do not increase traffic until all checks pass.

Step 6: Increase to 5%

In the Optimizely app:

1. Open the Progressive Rollout — Phase 1 rule
2. Change traffic allocation from 1% to 5%
3. Click Save, then Publish

At 5%, you have enough traffic to spot statistical patterns. Monitor for 4-8 hours.

Validation checklist at 5%:

• Error rate delta is less than 0.1% compared to baseline
• Latency p95 is within 5% of the legacy path
• Business metrics (click-through, conversion) are stable or improving
• No user-reported issues

Step 7: Add audience-based targeting (10%)

For this phase, target a specific audience to validate the feature with a representative user segment.

1. Edit the rollout rule
2. Change the audience to: country is "US" AND plan is "pro" OR plan is "enterprise"
3. Set traffic to 10% within that audience
4. Save and publish

This approach validates the feature with your highest-value users at a controlled level before broadening.

Step 8: Increase to 25%

After passing all checks at 10%, increase to 25%. At this traffic level, start monitoring resource consumption more closely.

1. Change traffic allocation to 25%
2. Save and publish
3. Monitor for 12-24 hours

At 25%, watch for:

• Database connection pool pressure from the new code path
• Cache hit rates changing under higher load
• Any latency degradation under sustained traffic

Step 9: Increase to 50%

The 50% mark is where you gain real statistical confidence in the feature’s impact on business metrics.

1. Change traffic allocation to 50%
2. Broaden the audience to Everyone (remove the segment restriction)
3. Save and publish
4. Monitor for 24-48 hours

At 50%, your telemetry data becomes a reliable A/B comparison. Compare the new path against legacy across all metrics.

Step 10: Increase to 100%

When all metrics are stable or improving at 50%:

1. Change traffic allocation to 100%
2. Save and publish

At 100%, every user sees the new feature. Keep monitoring for 1-2 weeks before cleaning up.

Step 11: Set up automated rollback

Build a rollback mechanism that reduces traffic automatically if key metrics degrade. This code uses the Optimizely REST API to update the rollout rule.

Automated rollback monitor

JavaScript

javascript Copy

// Rollback monitor — runs on a schedule (e.g., every 5 minutes)
async function checkRolloutHealth() {
  const errorRate = await metrics.query('search.error.rate', { path: 'new' });
  const p95Latency = await metrics.query('search.duration_ms.p95', { path: 'new' });
  const baselineP95 = await metrics.query('search.duration_ms.p95', { path: 'legacy' });

  const thresholds = {
    maxErrorRate: 0.02,         // 2% error rate
    maxLatencyRatio: 1.5,       // 50% slower than baseline
  };

  const shouldRollback =
    errorRate > thresholds.maxErrorRate ||
    p95Latency > baselineP95 * thresholds.maxLatencyRatio;

  if (shouldRollback) {
    console.error('Rollout health check failed — triggering rollback');
    console.error(`Error rate: ${errorRate}, P95 latency: ${p95Latency}ms`);

    // Option 1: Alert the team
    await alerting.send('pagerduty', {
      severity: 'critical',
      message: `new_search_engine rollout health check failed. Error rate: ${errorRate}`,
    });

    // Option 2: Reduce traffic via the Optimizely REST API
    // See https://docs.developers.optimizely.com/feature-experimentation/reference
    // for API details on updating feature flag rules
  }
}

Step 12: Clean up after full release

Once the feature has been at 100% for 1-2 weeks with stable metrics, remove the flag from your code.

1. Remove the flag check — Replace the branching logic with direct calls to the new code path
2. Delete the old code — Remove the legacy search engine
3. Archive the flag — In the Optimizely app, archive the new_search_engine feature flag
4. Update documentation — Note the completion date and final metrics

Before and after cleanup

JavaScript

javascript Copy

// BEFORE: Flag-controlled code path
async function handleSearchRequest(userId, query, userAttributes) {
  const user = optimizely.createUserContext(userId, userAttributes);
  const decision = user.decide('new_search_engine');

  if (decision.enabled) {
    return await newSearchEngine(query, extractConfig(decision));
  }
  return await legacySearchEngine(query);
}

// AFTER: Clean code, flag removed
async function handleSearchRequest(query) {
  return await searchEngine(query, {
    maxResults: 20,
    typoCorrection: true,
    rankingAlgorithm: 'relevance_v2',
    cacheTtl: 300,
  });
}

Do not leave stale flags in your codebase. Each flag adds a conditional branch that developers must understand and maintain. Set a calendar reminder to clean up flags 2 weeks after reaching 100%.

Rollout timeline summary

Phase	Traffic	Duration	Gate criteria
Phase 1	1%	1+ hours	No errors, latency within 10%
Phase 2	5%	4-8 hours	Error delta < 0.1%, latency within 5%
Phase 3	10%	8-12 hours	Targeted audience validation
Phase 4	25%	12-24 hours	Resource utilization stable
Phase 5	50%	24-48 hours	Business metrics stable or improving
Phase 6	100%	1-2 weeks	Full stability, then clean up

What to do next

• Add experiments — Run A/B tests on feature variables (e.g., ranking algorithm variants) before the full rollout
• Automate gates — Build CI/CD pipeline integration that blocks traffic increases until health checks pass
• Create a rollout playbook — Document your team’s standard rollout phases, gate criteria, and escalation procedures

Related content

Continue your journey with these related pages.

📖Understand the concepts

• Feature Experimentation
• Experimentation in Optimizely

🔧Follow a guide

• Experimentation guides