Implement CMAB-Driven Content Optimization

What you will build

A standard A/B test finds one winning variation for all visitors. A Contextual Multi-Armed Bandit (CMAB) finds the best variation for each visitor segment by learning from contextual attributes in real time.

By the end of this tutorial, you will have:

• A CMAB experiment with multiple content variations
• Contextual attributes that the bandit uses to personalize decisions
• Monitoring in place to observe exploration vs. exploitation behavior
• An understanding of when the bandit has converged and how to interpret results

How CMAB differs from A/B testing

In a traditional A/B test, traffic splits evenly across variations for the entire test duration. You wait for statistical significance, pick a winner, and apply it globally.

A CMAB works differently:

1. Exploration phase — The bandit distributes traffic roughly evenly to learn which variations perform well.
2. Exploitation phase — As the bandit learns, it shifts traffic toward the best-performing variation for each context. A visitor with attribute country=US might see Variation A, while a visitor with country=DE might see Variation C.
3. Continuous adaptation — The bandit keeps exploring at a low rate to detect changes in performance over time.

The key advantage: CMAB reduces the opportunity cost of a test. Instead of showing a losing variation to 50% of visitors for weeks, the bandit automatically reduces traffic to underperforming variations within days.

Before you start

Ensure you have:

• Optimizely account access — Feature Experimentation or Web Experimentation
• Sufficient traffic — CMAB needs at least 500 visitors per day to learn effectively
• Clear success metric — One primary metric (click-through rate, conversion, revenue) that the bandit optimizes toward
• Contextual data — User attributes that might influence which variation performs best (location, device type, plan tier, referral source)

Step 1: Form your hypothesis

A CMAB hypothesis is broader than a standard A/B test hypothesis because you are testing whether different segments respond differently to variations.

Write your hypothesis in this format:

“Different visitor segments will respond best to different [content element]. Specifically, I expect that [attribute] influences which [variation] drives the most [metric], because [reasoning].”

Example:

“Different visitor segments will respond best to different hero banner designs. Specifically, I expect that visitor plan tier and referral source influence which headline drives the most trial sign-ups, because enterprise prospects respond to ROI messaging while organic visitors respond to product capabilities.”

Step 2: Create the CMAB experiment

Available in Both Products

Experimentation

CMAB experiments are available in both Feature Experimentation (server-side) and Web Experimentation (client-side). The setup in the Optimizely app is the same. The code integration differs by product.

In the Optimizely app:

1. Navigate to Experiments
2. Click Create New > Multi-Armed Bandit
3. Name it: Hero Banner CMAB — Q1 2026
4. Set the primary metric to your conversion event (e.g., trial_signup)
5. Under Bandit Settings, select Contextual (not Basic)
6. Click Create

The contextual setting tells the bandit to learn which variations work best for different attribute combinations, rather than finding one global winner.

Step 3: Define your variations

Create 3-5 variations that represent meaningfully different content approaches. More variations give the bandit more options but require more traffic to learn.

Variation	Headline	Approach
Control	”Build better digital experiences”	Generic value proposition
ROI Focus	”Increase conversion rates by 30%“	Data-driven, results-oriented
Speed Focus	”Launch experiments in minutes”	Ease of use, speed to value
Social Proof	”Trusted by 9,000+ brands worldwide”	Authority and trust

Experimentation

In Web Experimentation, create each variation using the visual editor. Click on the hero banner element and edit the text, images, or layout for each variation.

Experimentation

In Feature Experimentation, create a feature flag with a string variable for the headline content. Each variation sets the variable to a different value.

Feature Experimentation setup

If you are using Feature Experimentation, create the flag and variations in code:

CMAB flag setup (Feature Experimentation)

JavaScript C# (.NET 8+) Python

javascript Copy

const user = optimizely.createUserContext(userId, {
  plan: userPlan,             // 'free', 'pro', 'enterprise'
  referral_source: utmSource, // 'organic', 'paid', 'referral'
  device_type: deviceType,    // 'mobile', 'desktop', 'tablet'
  country: geoCountry,        // ISO country code
});

const decision = user.decide('hero_banner_cmab');

if (decision.enabled) {
  const headline = decision.variables.headline;
  const subheadline = decision.variables.subheadline;
  const ctaText = decision.variables.cta_text;

  renderHeroBanner({ headline, subheadline, ctaText });
}

csharp Copy

var user = _optimizely.CreateUserContext(userId,
    new UserAttributes
    {
        { "plan", userPlan },
        { "referral_source", utmSource },
        { "device_type", deviceType },
        { "country", geoCountry },
    });

var decision = user.Decide("hero_banner_cmab");

if (decision.Enabled)
{
    var headline = (string)decision.Variables["headline"];
    var subheadline = (string)decision.Variables["subheadline"];
    var ctaText = (string)decision.Variables["cta_text"];

    RenderHeroBanner(headline, subheadline, ctaText);
}

python Copy

user = client.create_user_context(user_id, {
    'plan': user_plan,
    'referral_source': utm_source,
    'device_type': device_type,
    'country': geo_country,
})

decision = user.decide('hero_banner_cmab')

if decision.enabled:
    headline = decision.variables['headline']
    subheadline = decision.variables['subheadline']
    cta_text = decision.variables['cta_text']

    render_hero_banner(headline, subheadline, cta_text)

Step 4: Configure contextual attributes

Contextual attributes are the signals the bandit uses to learn which variation works best for different visitor types. Choose attributes that you believe influence visitor preferences.

In the Optimizely app:

1. Open your CMAB experiment
2. Navigate to Audiences > Attributes
3. Add the attributes the bandit should consider:

Attribute key	Type	Values	Why it matters
plan	String	free, pro, enterprise	Different plan tiers have different motivations
referral_source	String	organic, paid, referral, direct	Traffic source reflects visitor intent
device_type	String	mobile, desktop, tablet	Content consumption differs by device
country	String	ISO codes	Regional preferences for messaging tone

4. Save the configuration

The bandit uses these attributes to build a model. It learns, for example, that enterprise visitors from paid campaigns respond best to the ROI-focused headline, while free-tier organic visitors prefer the speed-focused headline.

Attribute selection guidelines

• Start with 3-5 attributes. Too many attributes fragment the data and slow learning.
• Choose attributes with clear values. Categorical attributes (plan tier, device type) work better than continuous ones (time on site).
• Include attributes you can act on. The value of a CMAB is that you learn which content works for which segments. Choose attributes that define actionable segments.

Step 5: Set up conversion tracking

The bandit needs a clear signal of what “success” means for each visitor interaction.

Web Experimentation

Track conversions (Web Experimentation)

JavaScript

javascript Copy

// Track when a visitor signs up for a trial
window.optimizely = window.optimizely || [];
window.optimizely.push({
  type: 'event',
  eventName: 'trial_signup',
});

// Track with revenue
window.optimizely.push({
  type: 'event',
  eventName: 'purchase_completed',
  tags: {
    revenue: 2999,  // $29.99 in cents
  },
});

Feature Experimentation

Track conversions (Feature Experimentation)

JavaScript C# (.NET 8+) Python

javascript Copy

// Track the primary conversion event
user.trackEvent('trial_signup');

// Track secondary metrics
user.trackEvent('hero_banner_clicked');
user.trackEvent('demo_requested');

csharp Copy

user.TrackEvent("trial_signup");
user.TrackEvent("hero_banner_clicked");
user.TrackEvent("demo_requested");

python Copy

user.track_event('trial_signup')
user.track_event('hero_banner_clicked')
user.track_event('demo_requested')

Create these events in the Optimizely app under Events before deploying the tracking code.

Step 6: Launch the experiment

Before launching, verify:

• All variations render correctly across devices and browsers
• Conversion tracking fires on the success action
• Contextual attributes are being passed with correct values
• The primary metric is set in the experiment configuration

In the Optimizely app:

1. Open your CMAB experiment
2. Review the experiment summary
3. Click Start Experiment

The bandit starts in exploration mode. During the first 24-48 hours, traffic is distributed roughly evenly across all variations to gather initial data.

Step 7: Monitor the exploration phase

During the first 2-3 days, the bandit is exploring. Do not make changes during this phase. The bandit needs time to collect data across all attribute combinations.

What to observe:

• Traffic distribution — Should be roughly even across variations. If one variation has significantly less traffic after 48 hours, check that it is rendering correctly.
• Conversion rates by variation — Early differences are noise. Do not draw conclusions until the bandit has processed at least 1,000 visitors per variation.
• Attribute coverage — Verify that visitors are arriving with varied attribute values. If 95% of visitors have the same plan tier, that attribute will not help the bandit differentiate.

Monitor bandit state (Feature Experimentation)

JavaScript

javascript Copy

import { OptimizelyDecideOption } from '@optimizely/optimizely-sdk';

// Log decisions to understand bandit behavior
optimizely.notificationCenter.addNotificationListener(
  'DECISION',
  (notification) => {
    if (notification.decisionInfo.flagKey === 'hero_banner_cmab') {
      console.log('CMAB Decision:', {
        userId: notification.userId,
        variation: notification.decisionInfo.variationKey,
        enabled: notification.decisionInfo.enabled,
      });
    }
  }
);

Step 8: Observe exploitation behavior

After the exploration phase (typically 3-7 days depending on traffic volume), the bandit shifts toward exploitation. Traffic moves toward better-performing variations, and you begin seeing differentiation by context.

Signs the bandit is exploiting:

• Uneven traffic split — The best-performing variation receives more traffic. This is expected and desirable.
• Context-dependent allocation — The variation distribution differs across attribute segments. For example, enterprise visitors might see the ROI headline 60% of the time, while free-tier visitors see the speed headline 70% of the time.
• Rising overall conversion rate — The aggregate conversion rate should exceed what you would see with an even A/B split, because the bandit is directing visitors to their best-performing variation.

In the Optimizely results dashboard:

1. Open your experiment results
2. Look at the Bandit Allocation chart — it shows traffic distribution over time
3. Check Performance by Segment — filter by each contextual attribute to see how the bandit personalizes

Step 9: Analyze convergence

A CMAB experiment does not “end” with a single winner the way an A/B test does. Instead, you analyze whether the bandit has converged — meaning its allocation strategy has stabilized.

Signs of convergence:

Indicator	What it means
Traffic allocation is stable for 5+ days	The bandit is confident in its model.
Overall conversion rate has plateaued	The bandit has found near-optimal allocations.
Variation performance by segment is consistent	The per-segment winners are stable, not fluctuating.
Exploration rate is minimal	The bandit is sending less than 10% of traffic to exploration.

When the bandit converges, you have two options:

Option A: Keep the bandit running. The bandit continues to optimize and adapts if visitor behavior changes. This is the recommended approach for high-traffic pages where visitor preferences shift over time.

Option B: Extract the winners and hard-code. For each attribute segment, note which variation performed best. Implement that personalization logic directly in your application code and stop the experiment.

Step 10: Extract insights and iterate

The most valuable output of a CMAB experiment is not a single winner — it is a personalization model. Analyze what the bandit learned.

Build a segment-variation matrix from your results:

Segment	Best variation	Conversion rate	Lift vs. control
Enterprise + Paid	ROI Focus	8.2%	+34%
Enterprise + Organic	Social Proof	6.9%	+18%
Pro + Paid	Speed Focus	5.4%	+22%
Free + Organic	Speed Focus	3.1%	+41%
Free + Direct	Control	2.3%	baseline

This matrix tells you:

• Which messaging resonates with which audience
• Whether personalization is worth the complexity (large lifts justify the investment)
• Which segments to prioritize in future experiments

Next experiments to run

Based on your CMAB insights:

1. Deep-dive A/B tests — For the highest-lift segments, run focused A/B tests with more variations of the winning approach
2. Expand to other page elements — Apply the same contextual personalization to subheadlines, images, or CTAs
3. Test new attributes — Add attributes like time of day, session count, or previous page viewed
4. Layer with Optimizely Data Platform — Use behavioral segments from ODP as CMAB attributes for richer personalization

Deeper Personalization with ODP

ExperimentationODP

Connect Optimizely Data Platform segments as targeting attributes in your CMAB experiments. Behavioral attributes like “visited pricing page 3 times” or “downloaded whitepaper” provide stronger signals than demographic attributes alone.

What to do next

• Run longer — CMAB experiments improve over time. Consider running for 4-6 weeks to capture weekly traffic patterns.
• Add more variations — If one messaging approach dominates, create 2-3 variants of that approach to refine further.
• Apply to other surfaces — Replicate the CMAB approach on email subject lines, in-app messages, or product recommendation ordering.
• Build a personalization strategy — Use CMAB insights to inform your broader content personalization roadmap.

Related content

Continue your journey with these related pages.

📖Understand the concepts

• Feature Experimentation
• Web Experimentation
• Experimentation in Optimizely

🔧Follow a guide

• Experimentation guides