CMAB: Contextual Multi-Armed Bandit

The problem: A/B tests do not adapt

In a traditional A/B test, traffic is split evenly across variations for the entire duration of the experiment. If variation A is clearly outperforming variation B after a few days, the experiment continues sending 50% of traffic to the losing variation until statistical significance is reached. Every visitor sent to the underperforming variation represents lost revenue, fewer conversions, or a worse experience.

Beyond that, traditional A/B tests treat all visitors the same. Variation A might work better for new visitors while variation B resonates with returning customers, but a standard A/B test cannot detect or act on that difference. It picks one winner for everyone.

Contextual Multi-Armed Bandit (CMAB) solves both problems. It automatically shifts traffic toward winning variations while simultaneously learning which variations perform best for different user segments.

How CMAB works

CMAB combines two strategies:

Explore vs. exploit

The bandit algorithm balances two competing goals:

• Explore — Try different variations to gather data and reduce uncertainty about their performance.
• Exploit — Send more traffic to variations that are already performing well.

Early in the experiment, CMAB explores more aggressively, distributing traffic broadly to learn which variations work. As data accumulates and confidence grows, it shifts toward exploitation, sending the majority of traffic to top-performing variations. This transition happens automatically — you do not need to monitor results or manually adjust traffic.

Contextual intelligence

The “contextual” part is what distinguishes CMAB from a simple bandit. CMAB uses user attributes (the same attributes you pass for audience targeting) as contextual signals to personalize variation assignment.

For example, CMAB might learn that:

• Visitors from the US convert better with variation A (bold pricing display)
• Visitors from Europe convert better with variation B (detailed comparison table)
• Mobile users perform best with variation C (simplified single-column layout)

Instead of picking one global winner, CMAB selects the best variation for each visitor based on their attributes. This effectively runs a personalized experiment where every user segment gets its optimal experience.

CMAB vs. traditional A/B testing

Aspect	A/B Test	CMAB
Traffic split	Fixed (e.g., 50/50) for the duration	Dynamic, shifts toward winners
Winner selection	One winner for all users	Can select different winners per segment
Optimization	Manual (you declare a winner)	Automatic and continuous
Opportunity cost	High (traffic to losing variations)	Low (traffic shifts away from losers)
Best for	Learning which variation is objectively better	Maximizing conversions during the experiment
Statistical rigor	Fixed-horizon or sequential testing	Bayesian reward modeling

CMAB is not a replacement for A/B testing in every situation. A/B tests are better when you need clean causal evidence for a product decision (“Does feature X improve retention?”). CMAB is better when you want to maximize a metric during the experiment while still learning which variations work.

Contextual attributes

CMAB uses the same user attributes available in audience targeting. You do not need to define separate attributes for CMAB — the model automatically identifies which attributes are predictive of variation performance.

Shared Attribute Model

Experimentation

Both Feature Experimentation and Web Experimentation can run CMAB experiments. The contextual attributes come from the user context (SDK) or visitor attributes (snippet). The same underlying machine learning model drives optimization in both products.

Attributes that tend to be useful for CMAB:

• Geographic — Country, region, language
• Behavioral — New vs. returning, pages visited, session depth
• Demographic — Plan tier, customer segment, lifetime value
• Technical — Device type, browser, operating system

The model evaluates attribute predictiveness automatically. Attributes that do not correlate with variation performance are effectively ignored. You do not need to guess which attributes matter — provide the ones you have, and the model determines relevance.

When to use CMAB

Use CMAB when:

• Revenue is at stake during the experiment. If sending traffic to a losing variation has real cost (lost sales, reduced engagement), CMAB minimizes that cost by shifting traffic to winners faster.
• Your audience is diverse. If different user segments respond differently to variations, CMAB captures that heterogeneity and serves each segment its best variation.
• You have many variations. Testing 5 or more variations with an A/B test requires splitting traffic thinly. CMAB efficiently explores many variations without requiring massive traffic.
• You want continuous optimization. CMAB can run indefinitely, continuously adapting as user behavior changes over time.

Stick with traditional A/B testing when:

• You need a definitive answer. Product decisions (“Should we launch this feature?”) benefit from the clear statistical framework of A/B testing.
• Sample size is very small. CMAB needs enough data to learn; with very low traffic, a simple A/B test converges faster.
• You need to report a single lift number. Stakeholders sometimes need a straightforward “Variation B improved conversion by 12%.” CMAB’s personalized assignments make that kind of summary less meaningful.

Tier availability

CMAB is not available on all Optimizely plans:

Tier	CMAB availability
Essential	Not available
Enhanced	1 active CMAB experiment at a time
Advanced	Unlimited active CMAB experiments
Ultimate	Unlimited active CMAB experiments with priority model training

On the Enhanced tier, the single-experiment limit means you should choose your highest-impact experiment for CMAB and run other experiments as standard A/B tests.

How CMAB operates at runtime

When a user enters a CMAB experiment, the flow differs from standard bucketing:

1. Attribute collection — The SDK or snippet collects the user’s contextual attributes.
2. Model query — The attributes are sent to Optimizely’s CMAB service, which runs the trained model to determine the optimal variation for this user’s context.
3. Variation assignment — The user receives the variation selected by the model.
4. Event tracking — Conversion events are tracked normally and fed back into the model to improve future predictions.

Experimentation

In Feature Experimentation, CMAB decisions require a network call to Optimizely’s CMAB service at decision time. This is different from standard decisions, which are evaluated locally from the datafile. Plan for the additional latency (typically under 100ms) in your application architecture.

Experimentation

In Web Experimentation, the CMAB model query happens during snippet execution. The snippet handles the network call and caches the result so subsequent page loads do not repeat the query.

Getting started with CMAB

To run a CMAB experiment:

1. Create an experiment and select the CMAB optimization mode in the Optimizely app.
2. Define your variations as you would for a standard experiment.
3. Ensure your user context includes relevant attributes (the more contextual signals, the better the model can personalize).
4. Set a primary conversion metric that the model will optimize toward.
5. Launch the experiment. The model begins in exploration mode and gradually shifts to exploitation as it learns.

No changes to your SDK integration or snippet installation are required beyond what you already have for standard experiments.

📝 Check your understanding 3 questions

Related content

Continue your journey with these related pages.

📖Understand the concepts

• User Context and Bucketing
• Event Dispatching and Analytics
• Experimentation in Optimizely