Analyze Results

Why analyze results carefully

A green arrow does not always mean you should ship. Experiment results require interpretation. Understanding confidence intervals, sample sizes, and segment behavior prevents you from shipping changes that looked good in a test but hurt performance at scale.

Open the results page

1. Navigate to Experiments in the Optimizely application
2. Click on your experiment
3. Select the Results tab

The results page updates automatically as data flows in. You can view results at any time, but avoid making decisions until the experiment reaches statistical significance.

Read the results summary

The results page shows a summary card for each metric.

Key fields

Field	What it means
Baseline conversion rate	The control group’s performance
Variation conversion rate	Each variation’s performance
Improvement	Percentage change from baseline (positive = better for “increase” metrics)
Statistical significance	Confidence that the observed difference is real, not random
Confidence interval	The range where the true improvement likely falls
Visitors	Number of unique visitors bucketed into each variation

Understand statistical significance

Optimizely uses a sequential testing methodology (Stats Engine) that lets you check results without inflating false positive rates. The key thresholds:

• Below 90% — Not enough evidence. Keep the experiment running.
• 90% significance — Moderate confidence. Acceptable for low-risk changes.
• 95% significance — Strong confidence. Standard threshold for most decisions.
• 99% significance — Very strong confidence. Use for high-impact or irreversible changes.

The significance level you set before starting the experiment (see Set Up Metrics) determines when Optimizely marks a result as conclusive.

Interpret confidence intervals

The improvement percentage is a point estimate. The confidence interval shows the range of plausible values.

Example: Improvement = +8%, 95% confidence interval = [+2%, +14%]

This means you can be 95% confident the true improvement is between 2% and 14%. The narrower the interval, the more precise the estimate. Wide intervals suggest you need more data.

Watch for intervals crossing zero. If the confidence interval includes zero (e.g., [-1%, +8%]), the result is not statistically significant. The improvement might be positive, negative, or zero.

Segment results

Break down results by user attributes to uncover hidden patterns.

1. Click Segment above the results table
2. Select an attribute to segment by (e.g., device type, country, plan tier)
3. Review performance across segments

Common findings:

• A variation wins overall but loses on mobile
• A variation loses overall but wins for premium users
• A variation shows no effect on average but has strong segment-specific effects

Segment analysis is exploratory. If a segment shows a strong signal, validate it with a follow-up experiment targeted to that segment.

Make a ship decision

Use this framework:

Scenario	Recommendation
Primary metric significant, positive improvement, no negative secondary metrics	Ship the variation
Primary metric significant, negative improvement	Revert to control
Primary metric not significant after sufficient sample	No winner — revert to control or iterate
Primary metric positive but secondary metric negative	Investigate further — check if the tradeoff is acceptable
Segment-specific win	Run a follow-up experiment targeting that segment

Export results

For offline analysis or stakeholder reporting:

1. Click the Export button on the results page
2. Select the export format (CSV or PDF)
3. The export includes all metric data, confidence intervals, and visitor counts

Access results via the API

For automated reporting or integration with data warehouses, use the Results API.

Fetch experiment results via API

JavaScript Python

javascript Copy

const response = await fetch(
  'https://api.optimizely.com/v2/experiments/{experiment_id}/results',
  {
    headers: {
      'Authorization': 'Bearer YOUR_API_TOKEN',
      'Content-Type': 'application/json',
    },
  }
);

const results = await response.json();

// Access metric results
results.metrics.forEach(metric => {
  console.log(`${metric.name}: ${metric.results.lift.value}% lift`);
  console.log(`Significance: ${metric.results.significance}`);
});

python Copy

import requests

response = requests.get(
    f'https://api.optimizely.com/v2/experiments/{experiment_id}/results',
    headers={
        'Authorization': 'Bearer YOUR_API_TOKEN',
        'Content-Type': 'application/json',
    }
)

results = response.json()

# Access metric results
for metric in results['metrics']:
    print(f"{metric['name']}: {metric['results']['lift']['value']}% lift")
    print(f"Significance: {metric['results']['significance']}")

Common pitfalls

Pitfall	Why it matters	What to do instead
Stopping early on a positive result	Early results are noisy and unreliable	Wait for the experiment to reach the pre-set significance level
Ignoring secondary metrics	A conversion lift that increases support tickets is not a win	Review all metrics before deciding
Cherry-picking segments	Looking at enough segments guarantees a false positive	Treat segment analysis as hypothesis generation, not proof
Running too many variations	Each variation reduces per-variation sample size	Limit to 2-4 variations for most experiments
Not accounting for novelty effects	New designs get attention that fades over time	Run experiments for at least two full business cycles

Related content

Continue your journey with these related pages.

📖Understand the concepts

• How Experimentation Works

🔧Follow a guide

• Create an Experiment
• Set Up Metrics
• Configure Targeting and Audiences