Skip to content
Optimizely CertificationOptimizely Certification Optimizely Certification

Create an Experiment

⏱ 20 minutes beginner
📜Foundationexperimentation

Why create an experiment

Section titled “Why create an experiment”

Shipping changes without measurement is guessing. An experiment lets you compare variations of a page, feature, or flow against a control, using real user behavior as the judge. By structuring changes as experiments, you get statistical proof of what works before committing to a permanent rollout.

What you will do

Section titled “What you will do”
  1. Define a hypothesis and choose an experiment type
  2. Create the experiment in the Optimizely application
  3. Add variations
  4. Set traffic allocation
  5. Start the experiment

Choose an experiment type

Section titled “Choose an experiment type”
TypeBest forHow variations are delivered
A/B test (Web)Visual changes — headlines, layouts, CTAsOptimizely snippet modifies the page in the browser
Feature experimentBackend logic, algorithms, pricingYour application code reads flag values via an SDK
Multi-armed banditQuickly converging on a winnerOptimizely dynamically shifts traffic toward the best variation

Create an A/B test (Web Experimentation)

Section titled “Create an A/B test (Web Experimentation)”
  1. In the Optimizely application, navigate to Experiments and click Create New Experiment
  2. Select A/B Test
  3. Enter the experiment name — use a clear, descriptive name like “Homepage hero CTA test Q1 2026”
  4. Set the URL targeting — enter the page URL where the experiment runs
  5. Click Create Experiment

Add variations

Section titled “Add variations”
  1. In the experiment editor, click Create Variation
  2. Name the variation (e.g., “Green CTA button”)
  3. Use the visual editor to modify elements on the page — click any element to change text, styles, or layout
  4. For advanced changes, switch to code mode and add custom JavaScript or CSS
  5. Repeat for additional variations

Set traffic allocation

Section titled “Set traffic allocation”
  1. Navigate to the Traffic Allocation section
  2. Set the percentage of visitors included in the experiment (start with 100% for most tests)
  3. Distribute traffic evenly across variations or set custom splits
  4. The Original (control) always receives its share of traffic

Create a feature experiment (Feature Experimentation)

Section titled “Create a feature experiment (Feature Experimentation)”
  1. Navigate to Feature Flags and create or select a flag
  2. Add variables to the flag for each value you want to test (e.g., button_color, algorithm_version)
  3. Navigate to Experiments and click Create New Experiment
  4. Select Feature Experiment
  5. Choose the flag you created
  6. Add variations and set different variable values for each

Implement the flag in code

Section titled “Implement the flag in code”

Your application reads the flag decision at runtime. The SDK handles variation assignment and bucketing.

Read a feature flag decision
javascript 
import { createInstance } from '@optimizely/optimizely-sdk';

const optimizely = createInstance({
  sdkKey: 'YOUR_SDK_KEY',
});

await optimizely.onReady();

const user = optimizely.createUserContext('user-123', {
  country: 'US',
  plan: 'premium',
});

const decision = user.decide('checkout_flow');

if (decision.enabled) {
  const buttonColor = decision.variables.button_color;
  const algorithm = decision.variables.algorithm_version;
  renderCheckout({ buttonColor, algorithm });
} else {
  renderCheckout({ buttonColor: 'blue', algorithm: 'v1' });
}
python 
from optimizely import optimizely

client = optimizely.Optimizely(sdk_key='YOUR_SDK_KEY')

user = client.create_user_context('user-123', {
    'country': 'US',
    'plan': 'premium',
})

decision = user.decide('checkout_flow')

if decision.enabled:
    button_color = decision.variables['button_color']
    algorithm = decision.variables['algorithm_version']
    render_checkout(button_color, algorithm)
else:
    render_checkout('blue', 'v1')
csharp 
using OptimizelySDK;

var optimizely = OptimizelyFactory.NewDefaultInstance("YOUR_SDK_KEY");

var user = optimizely.CreateUserContext("user-123", new UserAttributes
{
    { "country", "US" },
    { "plan", "premium" },
});

var decision = user.Decide("checkout_flow");

if (decision.Enabled)
{
    var buttonColor = decision.Variables.GetValue<string>("button_color");
    var algorithm = decision.Variables.GetValue<string>("algorithm_version");
    RenderCheckout(buttonColor, algorithm);
}
else
{
    RenderCheckout("blue", "v1");
}

Start the experiment

Section titled “Start the experiment”

Before starting, verify:

  1. Metrics are attached — At least one primary metric is configured (see Set Up Metrics)
  2. Audience is defined — Target the right users (see Configure Audiences)
  3. QA testing — Preview each variation to confirm it renders correctly
  4. Click Start Experiment

Optimizely begins assigning visitors to variations and collecting data. Results appear in the Results tab within hours, but wait for statistical significance before drawing conclusions.

Troubleshooting

Section titled “Troubleshooting”
IssueCauseFix
Experiment not runningSnippet not installed or SDK not initializedVerify installation on the target page or check SDK logs
Variations look identicalVisual editor changes not savedRe-open the editor and confirm changes are saved
No visitors in resultsURL targeting mismatch or audience too narrowCheck URL pattern matches actual page URLs
Uneven traffic distributionSticky bucketing from a previous testVerify traffic allocation percentages and check for mutual exclusion groups