Certification Prep: Experimentation

Prerequisites

Foundation certification required. You must hold the Optimizely Foundation certification before attempting this Core exam. If you have not completed it yet, start with the Foundation Learning Path.

Exam format

Detail	Value
Certification tier	Core
Tracks	Web Experimentation, Feature Experimentation
Number of questions	60 multiple-choice
Time limit	90 minutes
Passing score	70%
Delivery	Online, proctored
Validity	2 years

Who this path is for

You are a developer, marketer, or product manager who uses Optimizely Experimentation to run A/B tests and manage feature flags. You want to validate your skills through the official Experimentation certification. This path covers every exam domain and links to the documentation that explains each topic.

How to use this path

Work through each exam domain in order. Study the linked material, then test your understanding with the self-check questions. The exam tests both conceptual understanding and practical application, so pay attention to decision-making scenarios, not just facts.

Estimated total time: 4-5 hours (spread across sessions as needed).

---

Exam domain 1: Experiment design (25%)

Strong experiment design determines whether results are trustworthy. This is the most conceptually demanding section of the exam.

Topics to study

• Hypothesis formulation — Structuring testable hypotheses with a clear metric, direction, and rationale
• Metric selection — Primary metrics, secondary metrics, and guardrail metrics. Why each matters.
• Sample size planning — How traffic volume, baseline conversion rate, and minimum detectable effect determine how long to run an experiment
• Audience targeting — Targeting experiments to specific user segments for relevant results
• Mutual exclusion groups — Preventing experiment interactions when running multiple tests on the same page

Key concepts

• A good hypothesis follows the format: “If we [change], then [metric] will [direction] because [rationale].”
• Guardrail metrics protect against winning on the primary metric while harming the business elsewhere.
• Running an experiment with insufficient sample size produces unreliable results, regardless of what the numbers show.

Self-check

• What makes a hypothesis testable versus aspirational?
• When would you use a guardrail metric?
• How does minimum detectable effect influence required sample size?
• When should you use mutual exclusion groups?

---

Exam domain 2: Web Experimentation (20%)

The exam covers the mechanics of creating and running experiments with Web Experimentation (the visual/client-side product).

Topics to study

• Experiment creation — Using the visual editor and coded variations
• Variation design — Single-variable versus multivariate tests
• Traffic allocation — Splitting traffic and the implications of changing allocation mid-experiment
• URL targeting — Page targeting rules and their interaction with SPA routing
• Custom events and tags — Tracking interactions beyond page views

Key skills

• Creating an A/B test with the visual editor
• Setting up URL targeting for specific pages or page patterns
• Configuring custom event goals
• Understanding the impact of traffic allocation changes on statistical validity

Self-check

• What happens to statistical validity if you change traffic allocation mid-experiment?
• How does Web Experimentation handle single-page applications?
• When would you choose a coded variation over a visual editor change?

---

Exam domain 3: Feature Experimentation (20%)

Feature flags and server-side experimentation enable product teams to test changes without client-side code injection.

Topics to study

• Feature flags — Creating, managing, and targeting feature flags
• Flag variations — Boolean flags versus multi-variate flags with JSON payloads
• Server-side SDK — Initialization, decision methods, and event tracking
• Environments — Development, staging, and production flag management
• Rollouts — Progressive rollouts with percentage-based targeting

Key skills

• Creating a feature flag with multiple variations
• Implementing flag checks in server-side code
• Configuring targeted delivery rules (audience conditions)
• Running a progressive rollout: 10% to 50% to 100%

Self-check

• What is the difference between a feature flag and an experiment?
• How do you run an experiment on a feature flag?
• What SDK method do you call to get a flag decision?
• How does the SDK handle decisions when the CDN datafile is unavailable?

---

Exam domain 4: Statistical analysis (20%)

The exam tests your ability to interpret results and make sound decisions based on data.

Topics to study

• Statistics Engine — How Optimizely’s sequential testing differs from fixed-horizon tests
• Statistical significance — What it means and what it does not mean
• Confidence intervals — Interpreting the range of plausible effects
• False discovery rate — Why running many metrics increases the chance of false positives
• Segments — Post-hoc analysis by user segments and its limitations

Key concepts

• Optimizely uses sequential testing, which means you can check results at any time without inflating false positive rates.
• Statistical significance alone does not prove a result is meaningful. Consider practical significance (effect size).
• Segment analysis is exploratory, not confirmatory. Segment-level wins should be validated in follow-up experiments.

Self-check

• Why can you check results at any time with Optimizely’s Stats Engine?
• What is the difference between statistical significance and practical significance?
• When should you trust segment-level results versus running a follow-up experiment?
• How does the false discovery rate correction affect multi-metric experiments?

---

Exam domain 5: Program management (15%)

The certification also covers experimentation best practices at the organizational level.

Topics to study

• Experiment prioritization — Frameworks for deciding what to test next (impact, confidence, ease)
• Testing culture — Building organizational buy-in for experimentation
• Documentation and knowledge sharing — Recording experiment learnings for institutional memory
• Collaboration — Roles in an experimentation team (designer, developer, analyst, PM)
• Experimentation maturity — Evolving from ad-hoc tests to a systematic program

Self-check

• How do you prioritize experiments when you have more ideas than traffic?
• What information should you record for every completed experiment?
• What distinguishes a mature experimentation program from an ad-hoc one?

---

Study Plan

Web Experimentation Certification

estimated remaining

0 Ready

0 Review

0 Deep Study

0 Not Started

🤖

AI Certification Coach

Web Experimentation Exam Preparation

Analyzing your preparation progress...

Final preparation

1. Run an experiment end-to-end — Design a hypothesis, create the experiment, let it run, and interpret the results. Nothing replaces hands-on practice.
2. Review weak areas — Revisit any self-check questions you could not answer confidently.
3. Study the Stats Engine — The statistical analysis section trips up many candidates. Make sure you understand sequential testing.
4. Take practice tests — Visit Optimizely Academy for official practice exams.
5. Schedule the exam — Book your certification through Optimizely Academy.

---

What to do next

After passing the Experimentation certification:

• Combine with CMS — Learn how to test CMS content with the CMS Developer Certification Prep path
• Learn the platform — Take the Platform Certification Prep path
• Explore the Experimentation learning path — Follow the Experimentation Practitioner path for deeper operational knowledge