Experiment Types: When to Use What
The problem: choosing the wrong experiment type wastes traffic
Section titled “The problem: choosing the wrong experiment type wastes traffic”Every visitor routed into an experiment is a resource. Send them into an A/B test when a bandit would converge faster, and you lose conversions. Run a multivariate test without enough traffic, and the experiment never reaches significance. Choose a bandit when you actually need clean causal evidence, and you cannot make a confident product decision.
Optimizely supports four experiment types. Each exists because it solves a different problem. Understanding when to use each one prevents wasted traffic, inconclusive results, and misguided decisions.
The four experiment types
Section titled “The four experiment types”A/B test
Section titled “A/B test”The standard experiment. You create two or more variations of a single element (or a full page) and split traffic evenly between them. After the experiment reaches statistical significance, you identify the winner.
How it works: Traffic is allocated to variations randomly and stays fixed throughout the experiment. Stats Engine evaluates whether the observed difference is real or noise.
Strengths:
- Clean causal inference — the result directly answers “Is variation B better than A?”
- Simple to design and interpret
- Works with any traffic level (you just need to wait longer at low volumes)
Limitations:
- Traffic goes to losing variations until the experiment ends
- One global winner — does not account for audience heterogeneity
Multivariate test (MVT)
Section titled “Multivariate test (MVT)”Tests multiple elements simultaneously to find the best combination. Instead of testing one headline, you test headline, image, and CTA button together. The experiment evaluates all possible combinations.
How it works: If you have 3 headlines and 2 images, MVT creates 6 combinations (3 x 2). Traffic is split across all of them. Stats Engine evaluates both the individual element effects and their interactions.
Strengths:
- Detects interaction effects (e.g., headline A works better with image B but worse with image C)
- Efficient when you suspect elements influence each other
Limitations:
- Requires significantly more traffic than an A/B test. With 6 combinations, you need roughly 6x the traffic of a two-variation A/B test to reach significance.
- Results are harder to interpret — more combinations mean more statistical comparisons
Traffic rule of thumb: Multiply your typical A/B test duration by the number of combinations. If an A/B test takes 2 weeks with your traffic, a 6-combination MVT takes approximately 12 weeks.
Multi-armed bandit (MAB)
Section titled “Multi-armed bandit (MAB)”Automatically shifts traffic toward the best-performing variation during the experiment. The algorithm explores broadly at first, then exploits winners as confidence grows. MAB reduces opportunity cost but provides weaker causal evidence than an A/B test because of the unequal traffic split.
Contextual multi-armed bandit (CMAB)
Section titled “Contextual multi-armed bandit (CMAB)”Extends MAB by using visitor attributes to personalize variation assignment. CMAB uses machine learning to model how visitor context (geography, device, behavior) affects variation performance, then assigns each visitor to their predicted best variation. It requires sufficient traffic and attribute diversity to train the model. See the CMAB deep dive for details.
Comparison table
Section titled “Comparison table”| Factor | A/B test | MVT | MAB | CMAB |
|---|---|---|---|---|
| Primary goal | Learn which is best | Find best combination | Maximize metric | Maximize + personalize |
| Traffic split | Fixed, even | Fixed, even | Dynamic, shifts to winners | Dynamic + personalized |
| Traffic requirement | Moderate | High (multiplicative) | Moderate | Moderate-high |
| Causal evidence | Strong | Strong | Moderate | Moderate |
| Winner type | One global winner | One best combination | One global winner | Per-segment winners |
| Best for | Product decisions | Element interaction testing | Short campaigns | Diverse audiences |
| Complexity | Low | Medium | Low | Medium |
Decision framework
Section titled “Decision framework”Start with these questions:
-
Do you need a definitive product decision? Use an A/B test. The fixed traffic split provides the cleanest causal evidence.
-
Are you testing multiple elements that might interact? Use an MVT — but only if you have enough traffic. If traffic is limited, test elements sequentially with A/B tests instead.
-
Is maximizing a metric more important than learning? Use a MAB. Ideal for promotional campaigns, seasonal offers, or any scenario where you care more about performance during the test than about a clean conclusion afterward.
-
Do different audience segments respond differently? Use CMAB. It automatically discovers and exploits segment-level differences that a standard A/B test or MAB would miss.
Tier availability
Section titled “Tier availability”| Experiment type | Essential | Enhanced | Advanced | Ultimate |
|---|---|---|---|---|
| A/B test | Yes | Yes | Yes | Yes |
| MVT | Yes | Yes | Yes | Yes |
| MAB | Yes | Yes | Yes | Yes |
| CMAB | No | 1 active experiment | Unlimited | Unlimited (priority training) |
Common mistakes
Section titled “Common mistakes”- Running MVT with insufficient traffic. If your site gets fewer than 50,000 visitors per week, stick to sequential A/B tests. MVT with 8+ combinations on low traffic will run for months.
- Using MAB for product decisions. The dynamic traffic allocation means the “losing” variation did not get a fair trial. If you need to convince stakeholders with clean data, use an A/B test.
- Defaulting to A/B tests when optimization matters more than learning. During a holiday sale, a MAB or CMAB can recover revenue that would be lost to underperforming variations in a fixed-split A/B test.
1. Your e-commerce site gets 10,000 visitors per week. You want to test 3 headlines, 2 hero images, and 2 CTA button colors simultaneously. What is the best approach?
With 12 combinations you would need roughly 12x the traffic of a standard A/B test. At 10,000 visitors per week, an MVT could take months to reach significance. Running sequential A/B tests is the practical alternative when traffic is limited.
With 12 combinations you would need roughly 12x the traffic of a standard A/B test. At 10,000 visitors per week, an MVT could take months to reach significance. Running sequential A/B tests is the practical alternative when traffic is limited.
Review this topic →2. Your team is running a holiday promotion for the next two weeks. You have 4 banner variations and want to maximize revenue during the campaign rather than learn which banner is objectively best. Which experiment type should you choose?
A multi-armed bandit automatically shifts traffic toward the best-performing variation during the experiment, reducing opportunity cost. This is ideal for time-limited campaigns where maximizing a metric matters more than clean causal evidence.
A multi-armed bandit automatically shifts traffic toward the best-performing variation during the experiment, reducing opportunity cost. This is ideal for time-limited campaigns where maximizing a metric matters more than clean causal evidence.
Review this topic →3. You suspect that US visitors prefer a bold pricing display while European visitors prefer a detailed comparison table. You want the experiment to automatically discover and act on these segment differences. Which experiment type fits?
CMAB uses visitor attributes like geography to personalize variation assignment. It automatically discovers which variations perform best for different segments and serves each visitor their predicted best variation, rather than picking one global winner.
CMAB uses visitor attributes like geography to personalize variation assignment. It automatically discovers which variations perform best for different segments and serves each visitor their predicted best variation, rather than picking one global winner.
Review this topic →