Skip to content
Optimizely CertificationOptimizely Certification Optimizely Certification

Automated Experiment Analysis with Opal

⏱ 35 minutes intermediate
📜CoreOpalexperimentation

Why experiment results go unread

Section titled “Why experiment results go unread”

Experimentation programs generate data. The problem is not running experiments — it is communicating results. Stats Engine produces statistically rigorous outputs, but most stakeholders cannot interpret confidence intervals, p-values, and lift calculations. The results page sits in a dashboard. Decisions are made without consulting it.

This recipe solves the communication gap. When an experiment completes, an Opal agent reads the results, translates them into plain language, and delivers a summary that any stakeholder can act on — no statistics degree required.

Architecture overview

Section titled “Architecture overview”
┌────────────────────────────────────────────────────┐
│     Experimentation: Experiment Completes           │
│  Stats Engine reaches statistical significance      │
│  on the primary metric, or the experiment           │
│  reaches its scheduled end date.                    │
└────────────────────────┬───────────────────────────┘
                         │ results data
                         ▼
┌────────────────────────────────────────────────────┐
│     Opal: Experiment Analysis Agent                 │
│  1. Retrieves experiment configuration and results  │
│  2. Analyzes primary and secondary metrics          │
│  3. Generates plain-language summary                │
│  4. Produces recommendations (implement, iterate,   │
│     or abandon)                                     │
│  5. Formats stakeholder-ready report                │
└────────────────────────┬───────────────────────────┘
                         │ analysis report
                         ▼
┌────────────────────────────────────────────────────┐
│     Stakeholders: Decision and Action               │
│  Product managers, marketers, and executives        │
│  receive a summary they can understand and          │
│  act on without interpreting raw statistics.         │
└────────────────────────────────────────────────────┘

Prerequisites

Section titled “Prerequisites”
  • Opal with Agent Builder access
  • Web Experimentation or Feature Experimentation with at least one completed experiment
  • Opti ID roles that grant access to experiment data

Step 1: Define your analysis framework

Section titled “Step 1: Define your analysis framework”

Before building the agent, decide what a good experiment summary includes. A useful analysis answers five questions:

  1. What did we test? — The hypothesis, variations, and audience
  2. What happened? — Primary metric result with clear win/loss/inconclusive verdict
  3. How confident are we? — Statistical significance in plain language
  4. What else did we learn? — Secondary metrics, segments, and unexpected findings
  5. What should we do next? — Concrete recommendation with reasoning

Document these five sections as the template for your agent’s output.

Step 2: Create the experiment analysis agent

Section titled “Step 2: Create the experiment analysis agent”
  1. Navigate to Opal > Agents
  2. Click Create Agent > Specialized Agent
  3. Name it “Experiment Analysis Reporter”
  4. Write the instructions:
## Objective
You analyze completed experiment results and produce a plain-language
summary that non-technical stakeholders can understand and act on.


## Process
1. Retrieve the experiment configuration: name, hypothesis,
   variations, audience, primary and secondary metrics
2. Retrieve the results: metric values for each variation,
   statistical significance, confidence intervals
3. Determine the verdict:
   - WIN: Primary metric shows statistically significant improvement
   - LOSS: Primary metric shows statistically significant decline
   - INCONCLUSIVE: No statistically significant difference detected
4. Translate statistics into plain language
5. Analyze secondary metrics for additional insights
6. Formulate a recommendation


## Output Format
### Experiment Summary: [Experiment Name]


**Verdict:** WIN | LOSS | INCONCLUSIVE
**Test Duration:** [start date] to [end date]
**Traffic:** [number of visitors/users in the experiment]


#### What We Tested
[1-2 sentences describing the hypothesis and what changed]


#### Results
- **Primary metric ([metric name]):**
  [variation name] performed [X%] better/worse than control.
  This result is [statistically significant / not yet significant].
- **What this means:** [Plain-language interpretation]


#### Secondary Metrics
| Metric | Control | Variation | Change | Significant? |
|---|---|---|---|---|


#### Recommendation
[IMPLEMENT / ITERATE / ABANDON] — [2-3 sentences explaining why
and what specific action to take next]


#### Caveats
[Any limitations: small sample size, seasonal factors, external
events that may have influenced results]


## Constraints
- Never overstate confidence. If results are not significant,
  say so clearly.
- Never recommend implementing a losing variation.
- Use percentages and plain comparisons, not p-values or
  confidence intervals.
- If sample size is below 1,000 per variation, flag this as a
  limitation.
  1. Enable the experimentation tool so the agent can access experiment data
  2. Add an input variable: experiment_id (required, text)
  3. Save the agent

Step 3: Test with a completed experiment

Section titled “Step 3: Test with a completed experiment”

Run the agent against a real completed experiment to validate the output.

  1. Find a completed experiment in Web Experimentation or Feature Experimentation
  2. Note the experiment ID
  3. Navigate to Opal > Chat
  4. Invoke the agent:
@ExperimentAnalysisReporter
experiment_id: EXP-12345

  1. Compare the agent’s summary against the actual results page:

    • Does the verdict match the statistical outcome?
    • Are the metric values accurate?
    • Is the recommendation reasonable?
    • Would a non-technical stakeholder understand the summary?

  2. Test with different experiment types:

    • A clear winner (statistically significant improvement)
    • A clear loser (statistically significant decline)
    • An inconclusive result (no significant difference)

Step 4: Configure automated triggers

Section titled “Step 4: Configure automated triggers”

Set up the agent to run automatically when experiments complete.

  1. In Opal > Agents, select the Experiment Analysis Reporter
  2. Configure a trigger based on experiment status:
    • Trigger event: Experiment status changes to “Completed” or reaches statistical significance
    • Input mapping: The experiment ID is passed automatically from the completed experiment
  3. Configure the output destination:
    • Attach the summary as a comment on the experiment
    • Send the summary to a designated notification channel or email distribution list

Step 5: Customize for your organization

Section titled “Step 5: Customize for your organization”

Adapt the agent to your team’s specific needs:

Add business context

Section titled “Add business context”

Include instructions that reference your organization’s KPIs:

## Business Context
Our primary business KPI is revenue per visitor. When analyzing
experiments, always relate the primary metric result back to
estimated revenue impact. Use an average order value of $85 for
revenue estimates.

Configure stakeholder-specific formats

Section titled “Configure stakeholder-specific formats”

Create variations of the agent for different audiences:

  • Executive summary — 3 sentences maximum, verdict and revenue impact only
  • Product team summary — Full analysis with secondary metrics and segment breakdowns
  • Engineering summary — Technical details including confidence intervals and sample sizes

Add historical context

Section titled “Add historical context”

If RAG is enabled, instruct the agent to reference past experiments:

## Historical Context
Search for previous experiments on the same page or feature.
Reference past results to show trends (e.g., "This is the third
test on the checkout page; previous tests showed 5% and 8%
improvements with CTA changes").

Step 6: Verify the end-to-end workflow

Section titled “Step 6: Verify the end-to-end workflow”

Run through the complete automated flow.

  1. Start an experiment in Web or Feature Experimentation (or use a recently completed one)
  2. Wait for the experiment to reach completion or statistical significance
  3. Confirm the Opal agent triggers automatically
  4. Review the generated summary for accuracy and clarity
  5. Verify the summary reaches stakeholders through your configured notification channel
  6. Collect feedback from stakeholders on summary usefulness

When to use this pattern

Section titled “When to use this pattern”

This recipe is most valuable when:

  • Your experimentation program runs multiple experiments per month
  • Stakeholders need results communicated in non-technical language
  • Experiment results frequently go unread in dashboards
  • You want to build an organizational knowledge base of experiment learnings

For high-stakes experiments (pricing changes, major UX redesigns), supplement the automated summary with a human-led results review meeting. The agent summary serves as the starting point for discussion, not the final word.

Extending this recipe

Section titled “Extending this recipe”
  • Experiment knowledge base — Store all summaries in a searchable repository so teams can reference past learnings
  • Trend analysis — Build a monthly reporting agent that aggregates experiment summaries and identifies patterns across your testing program
  • Automated follow-ups — Chain the analysis agent with a planning agent that generates the next experiment hypothesis based on results