Experimentation in Optimizely
The problem experimentation solves
Section titled “The problem experimentation solves”Every digital decision is a bet — which headline converts better, whether a new checkout flow reduces abandonment, whether showing recommendations increases revenue. Without experimentation, these decisions are based on opinions, past experience, or competitor imitation. Sometimes those instincts are right. Often they are not.
Experimentation replaces guesswork with evidence. You show different variations to different visitors, measure the outcomes, and let statistical analysis determine which variation performs better. The result is decisions backed by data from your actual users.
Two products, two approaches
Section titled “Two products, two approaches”Optimizely offers two experimentation products. They solve different problems and serve different workflows.
| Feature Experimentation | Web Experimentation | |
|---|---|---|
| What it tests | Code-level features (backend and frontend) | Visual changes on web pages |
| Who uses it | Developers and product teams | Marketers and optimization teams |
| How changes are made | SDK in your application code | Visual editor or client-side JavaScript |
| Deployment model | Server-side (SDKs for .NET, JS, Python, etc.) | Client-side (JavaScript snippet) |
| Best for | Feature flags, progressive rollouts, server-side logic | A/B tests on copy, images, layout, CTAs |
| Requires code deploy | Yes (SDK integration) | No (changes applied via snippet) |
Feature Experimentation
Section titled “Feature Experimentation”Feature Experimentation is a developer-centric platform for controlling feature releases and running experiments in application code. It operates through SDKs embedded in your application.
Core capabilities:
- Feature flags — Toggle features on or off without deploying code
- Progressive rollouts — Release features to 1%, then 10%, then 50%, then 100% of users
- Targeted delivery — Show features to specific user segments based on attributes
- Server-side experiments — Test backend logic, algorithms, pricing, and API responses
- Kill switches — Instantly disable a broken feature without a rollback deploy
- Multi-platform — SDKs for JavaScript, React, .NET, Python, Java, Go, Ruby, PHP, Swift, Android
Why it matters: Feature flags fundamentally change how teams deploy software. Instead of big-bang releases where everything goes live at once, teams deploy code continuously and control visibility through flags. This reduces deployment risk, enables faster iteration, and lets product teams make release decisions independently of engineering deployment schedules.
Web Experimentation
Section titled “Web Experimentation”Web Experimentation is a marketer-centric platform for testing visual changes on websites without code changes. It operates through a JavaScript snippet added to your site.
Core capabilities:
- Visual editor — Point-and-click interface to modify page elements
- A/B testing — Compare two or more variations of a page element
- Multivariate testing — Test multiple elements simultaneously to find optimal combinations
- Personalization campaigns — Deliver targeted experiences based on audience rules
- Stats engine — Sequential testing with always-valid statistical results
- Audience targeting — Define who sees experiments based on behavior, geography, technology, and custom attributes
Why it matters: Marketing teams need to optimize continuously, and waiting for developer resources creates bottlenecks. Web Experimentation lets marketers test hypotheses directly — changing headlines, images, CTAs, and layouts — without engineering involvement.
How experimentation connects to Optimizely One
Section titled “How experimentation connects to Optimizely One”Experimentation becomes more powerful when connected to other Optimizely products:
| Integration | What it enables |
|---|---|
| CMS + Web Experimentation | A/B test CMS page variations directly |
| CMS + Feature Experimentation | Test content delivery logic server-side |
| ODP + Experimentation | Target experiments to ODP audience segments |
| Analytics + Experimentation | Measure experiment impact on business metrics |
| Graph + Feature Experimentation | Test headless content variations across channels |
| Opal + Experimentation | AI-suggested experiment hypotheses and variations |
The experimentation workflow
Section titled “The experimentation workflow”Regardless of which product you use, effective experimentation follows the same discipline:
- Hypothesize — Define what you expect to happen and why. “Showing social proof on the pricing page will increase conversions because visitors trust peer validation.”
- Design — Create variations that test the hypothesis. Keep changes focused — test one variable at a time when possible.
- Target — Define who should see the experiment. Not every test needs 100% of traffic.
- Run — Launch the experiment and let it collect data. Do not stop early based on gut feeling.
- Analyze — Use statistical analysis to determine if the result is significant. Optimizely’s Stats Engine handles this automatically.
- Act — If the variation wins, implement it permanently. If it loses, learn from the data and form a new hypothesis.
Statistical rigor: why it matters
Section titled “Statistical rigor: why it matters”A common mistake in experimentation is declaring winners too early. If you check results after 100 visitors and see a 15% lift, that is almost certainly noise, not signal.
Optimizely addresses this with Stats Engine, which uses sequential testing. Unlike traditional A/B testing that requires a fixed sample size, sequential testing provides always-valid results — you can check at any time and the statistical confidence is accurate.
Key concepts:
- Statistical significance — The probability that the observed difference is real, not random chance. Optimizely defaults to 90% confidence.
- Minimum detectable effect — The smallest improvement worth detecting. Larger effects need fewer visitors; subtle effects need more traffic.
- False discovery rate — The expected proportion of winning experiments that are actually false positives. Optimizely controls this automatically.
Decision framework: which product to use
Section titled “Decision framework: which product to use”| Scenario | Recommended product |
|---|---|
| Test a new checkout flow involving backend logic | Feature Experimentation |
| Test different hero banner copy and images | Web Experimentation |
| Gradually roll out a new feature to reduce risk | Feature Experimentation |
| A marketer wants to test CTAs without developers | Web Experimentation |
| Test a new recommendation algorithm | Feature Experimentation |
| Run a multivariate test on a landing page | Web Experimentation |
| Test across web, mobile app, and API simultaneously | Feature Experimentation |
| Run a quick copy test before a campaign launch | Web Experimentation |
Many organizations use both. Feature Experimentation handles backend logic and cross-platform tests. Web Experimentation handles visual optimization that marketers manage independently.
Experimentation tiers
Section titled “Experimentation tiers”Optimizely Experimentation is available in tiered plans with increasing capabilities:
| Capability | Essential | Enhanced | Advanced | Ultimate |
|---|---|---|---|---|
| A/B testing | ✓ | ✓ | ✓ | ✓ |
| Feature flags | ✓ | ✓ | ✓ | ✓ |
| Visual editor | — | ✓ | ✓ | ✓ |
| Multivariate testing | — | — | ✓ | ✓ |
| Mutual exclusion groups | — | — | ✓ | ✓ |
| Advanced audience targeting | — | — | ✓ | ✓ |
| Multi-armed bandit | — | — | — | ✓ |
| Program management | — | — | — | ✓ |
Getting started
Section titled “Getting started”- Developers: Read Feature Experimentation to understand SDK integration and feature flags
- Marketers: Read Web Experimentation to understand visual testing and campaign setup
- Both: Follow the tutorials to run your first experiment
1. Your marketing team wants to test three different hero images on a landing page without involving developers. Your engineering team also wants to test a new pricing algorithm on the backend. Which approach best fits both needs?
Web Experimentation is designed for visual changes that marketers manage without code deploys, while Feature Experimentation handles backend logic through SDKs. Using each product for its intended use case avoids bottlenecks and leverages the right tooling.
Web Experimentation is designed for visual changes that marketers manage without code deploys, while Feature Experimentation handles backend logic through SDKs. Using each product for its intended use case avoids bottlenecks and leverages the right tooling.
Review this topic →2. You check your A/B test results after 100 visitors and see a 15% lift with what looks like strong numbers. What should you do?
Early results with small sample sizes are almost certainly noise, not signal. Optimizely's Stats Engine uses sequential testing to provide always-valid results, so you should wait for it to confirm statistical significance rather than acting on premature data.
Early results with small sample sizes are almost certainly noise, not signal. Optimizely's Stats Engine uses sequential testing to provide always-valid results, so you should wait for it to confirm statistical significance rather than acting on premature data.
Review this topic →3. A product manager wants to gradually release a new checkout flow to reduce deployment risk, starting with 1% of users and scaling up. Which Optimizely capability should they use?
Progressive rollouts are a core Feature Experimentation capability. Feature flags let you release to 1%, then 10%, then 50%, then 100% of users, reducing deployment risk. This is a code-level change that requires SDK integration, not a visual test.
Progressive rollouts are a core Feature Experimentation capability. Feature flags let you release to 1%, then 10%, then 50%, then 100% of users, reducing deployment risk. This is a code-level change that requires SDK integration, not a visual test.
Review this topic →