Interpret Results
Why interpretation matters
Section titled “Why interpretation matters”A green arrow on the results page does not mean “ship it.” Results require interpretation. Understanding what the numbers mean — and what they do not — prevents you from shipping a false positive, missing a real winner, or drawing conclusions from insufficient data.
Navigate to the results page
Section titled “Navigate to the results page”- Open your experiment in the Optimizely application
- Click the Results tab
- The page shows a summary card for each metric, updated as data flows in
You can view results at any time during a running experiment. However, avoid making ship decisions until the experiment reaches statistical significance.
Understand the results summary
Section titled “Understand the results summary”Each metric card displays several key fields.
| Field | What it tells you |
|---|---|
| Baseline rate | The control group’s conversion rate or average value |
| Variation rate | Each variation’s conversion rate or average value |
| Improvement | Percentage change compared to the baseline — positive means the variation outperforms control |
| Statistical significance | The probability that the observed difference is real and not due to random chance |
| Confidence interval | The range of values where the true improvement likely falls |
| Visitors | Number of unique visitors bucketed into each variation |
What statistical significance means
Section titled “What statistical significance means”Optimizely uses Stats Engine, a sequential testing methodology that lets you check results as data accumulates without inflating your false positive rate.
Key thresholds:
- Below 90% — Insufficient evidence. The result could easily be random noise. Keep the experiment running.
- 90% — Moderate confidence. Acceptable for low-risk, easily reversible changes.
- 95% — Strong confidence. The standard threshold for most business decisions.
- 99% — Very strong confidence. Use for high-revenue or irreversible changes.
When the results page shows “Significant,” it means the observed difference has reached the significance level you configured before launch.
What significance does NOT mean:
- It does not guarantee the improvement is large enough to matter. A statistically significant 0.1% lift may not justify the engineering effort to implement.
- It does not mean the result will hold forever. External factors (seasonality, marketing campaigns) can shift behavior over time.
Read confidence intervals
Section titled “Read confidence intervals”The improvement percentage is a point estimate. The confidence interval shows the plausible range.
Example: Improvement = +7%, 95% confidence interval = [+2%, +12%]
This means you can be 95% confident the true improvement is somewhere between 2% and 12%. The variation is almost certainly better than control, but the magnitude is uncertain.
Watch for intervals that cross zero. If the interval is [-1%, +8%], the true improvement might be negative. The result is not yet significant — the experiment needs more data.
Narrow vs. wide intervals:
- Narrow interval (e.g., [+5%, +9%]) — High precision. You have a strong estimate.
- Wide interval (e.g., [-3%, +15%]) — Low precision. More data is needed.
When to call a winner
Section titled “When to call a winner”Use this decision framework:
| Situation | What to do |
|---|---|
| Primary metric is significant and positive, no negative secondary metrics | Ship the variation |
| Primary metric is significant and negative | Stop the experiment and keep control |
| Primary metric not significant after 2+ weeks with adequate traffic | Inconclusive — stop and iterate |
| Primary positive but secondary metric is negative | Investigate the tradeoff before deciding |
| Different results across segments | Run a follow-up experiment on the winning segment |
Avoid common timing mistakes
Section titled “Avoid common timing mistakes”- Do not stop early on a positive result. Early results are volatile. A +20% lift on day two often regresses to +3% by week two.
- Do not extend indefinitely. If the experiment has been running for four weeks with no trend toward significance, the effect is likely too small to detect with your traffic. Stop and test a bolder change.
- Run for at least two full business cycles. Weekday and weekend behavior differ. A one-week test captures both; a five-day test does not.
Use segmentation carefully
Section titled “Use segmentation carefully”Break down results by visitor attributes to discover whether the effect varies across groups.
- Click Segment above the results table
- Select an attribute (device type, country, traffic source, etc.)
- Review the results for each segment
Important: Segmented results are exploratory, not confirmatory. If you look at enough segments, you will find one that appears significant by chance alone. Use segment insights to generate hypotheses for future experiments, not as proof.
Distinguish practical from statistical significance
Section titled “Distinguish practical from statistical significance”A result can be statistically significant but practically meaningless. Ask these questions:
- Is the lift large enough to matter? A 0.3% improvement in conversion may not justify the development cost.
- What is the revenue impact? Translate the lift into dollars, orders, or leads per month.
- Is the improvement sustained? Check whether the lift is stable across the experiment duration or driven by a single spike.
Share results with stakeholders
Section titled “Share results with stakeholders”When reporting results to your team:
- Lead with the business question the experiment answered
- State the primary metric result with the confidence interval
- Note any secondary metric tradeoffs
- Provide a clear recommendation: ship, revert, or iterate
- Link to the results page so stakeholders can explore details
Troubleshooting
Section titled “Troubleshooting”| Issue | Cause | Fix |
|---|---|---|
| Results show zero visitors | Experiment not active or snippet not installed | Verify experiment status and snippet installation |
| Significance fluctuates up and down | Normal behavior with sequential testing | Wait for the result to stabilize — do not react to daily swings |
| Improvement seems impossibly large | Very small sample size inflates early estimates | Wait for more visitors before interpreting |
| Secondary metric declining | Variation improves one behavior at the expense of another | Evaluate whether the tradeoff is acceptable for the business |