Run Your First A/B Test

What you will do

By the end of this tutorial, you will have:

• Created a hypothesis for your test
• Built a variation using the visual editor
• Defined a goal to measure success
• Set audience targeting
• Launched the experiment
• Understood how to read the results

No coding is required. This tutorial uses Web Experimentation’s visual editor.

Before you start

Ensure you have:

• Optimizely Web Experimentation access — Your admin should have invited you
• The snippet installed — The Optimizely JavaScript snippet must be on your website. Ask your developer if unsure.
• A test page — Pick a page with enough traffic (at least 1,000 visitors per week) to get results in a reasonable time

Step 1: Form your hypothesis

Before touching the tool, write down your hypothesis. A good hypothesis has three parts:

1. What you will change — The specific element you are modifying
2. What you expect to happen — The metric you expect to improve
3. Why you expect it — The reasoning behind your prediction

Example:

“Changing the CTA button text from ‘Learn More’ to ‘Start Free Trial’ will increase button clicks by 10% because direct action language creates urgency and clarity.”

Write your hypothesis down. You will refer back to it when setting up the experiment and interpreting results.

Step 2: Create a new experiment

1. Log into Optimizely at app.optimizely.com
2. Navigate to Experiments in the left sidebar
3. Click “Create New” → “A/B Test”
4. Name your experiment descriptively (e.g., “Homepage CTA Button Text Test — March 2026”)
5. Enter the URL of the page you want to test
6. Click “Create Experiment”

The visual editor opens, showing your live page.

Step 3: Create your variation

You are now in the visual editor. The page you see is the Original (Control). You need to create a variation.

1. In the variation list at the top, click “Create Variation”
2. Name it (e.g., “Direct CTA”)
3. You are now editing the variation — changes you make here only affect visitors who see this variation

To change text:

1. Click on the text element you want to change (e.g., the CTA button)
2. A toolbar appears — click “Edit Text”
3. Type your new text (e.g., “Start Free Trial”)
4. Click outside the element to confirm

To change an image:

1. Click on the image
2. Click “Change Image” in the toolbar
3. Enter the URL of your new image or upload one

To change styles:

1. Click on any element

2. Click “Edit Styles”

3. Modify colors, fonts, sizes, or spacing

4. When done, click “Save” in the top right

Step 4: Set your goal (metric)

A goal defines what you are measuring. For your first test, keep it simple — one primary goal.

1. In the experiment editor, click “Metrics” in the left panel
2. Click “Add Metric”
3. Choose a metric type:
   • Click — Track clicks on a specific element (best for CTA tests)
   • Pageview — Track visits to a specific page (best for conversion funnels)
   • Custom event — Track a JavaScript event (ask your developer to set this up)
4. Configure the metric:
   • For a click goal: click the element you want to track, or enter a CSS selector
   • For a pageview goal: enter the URL of the success page (e.g., /thank-you)
5. Name the metric (e.g., “CTA Button Clicks”)
6. Mark it as the primary metric

Step 5: Target your audience

Decide who should see the experiment. For your first test, start broad.

1. Click “Audiences” in the left panel
2. Choose targeting:
   • Everyone — All visitors see the experiment (simplest)
   • Custom audience — Target specific visitors (e.g., returning visitors, US only)
3. If using a custom audience, add conditions:
   • Example: “Country is United States AND Device is Desktop”

Tip: Start with “Everyone” for your first test. Narrower audiences require more traffic to reach significance.

Step 6: Set traffic allocation

Traffic allocation controls what percentage of qualifying visitors enter the experiment.

1. Click “Traffic Allocation” in the left panel
2. Set the percentage of visitors who should be in the experiment (default: 100%)
3. The traffic is split evenly between your variations (e.g., 50% Original, 50% Variation)

Tip: For your first test, keep 100% allocation with a 50/50 split. This gets you results fastest.

Step 7: Launch the experiment

Before launching, review your setup:

• ✅ Variation created with a clear, testable change
• ✅ Primary metric defined
• ✅ Audience targeting set
• ✅ Traffic allocation configured

When ready:

1. Click “Start Experiment”
2. Confirm in the dialog
3. The experiment is now live — visitors start being split between Original and Variation

Important: Once launched, do not change the variations. Modifying a running experiment invalidates the results. If you need to make changes, pause the experiment first.

Step 8: Read your results

Give the experiment time to collect data. Optimizely’s Stats Engine will tell you when results are statistically significant — do not declare a winner prematurely.

1. Navigate to “Results” in the experiment
2. You will see:
   • Improvement — The percentage difference between Original and Variation
   • Statistical significance — Whether the result is likely real (not random chance)
   • Confidence interval — The range of possible true improvement values

How to interpret:

What you see	What it means	What to do
Positive improvement, significant	The variation is winning	Consider implementing the change permanently
Positive improvement, not yet significant	Too early to tell	Keep the experiment running
Negative improvement, significant	The variation is losing	Stop the experiment, keep the original
Near-zero improvement	No meaningful difference	Stop the experiment, try a bolder change

When to stop:

• Stats Engine shows “significant” for your primary metric
• OR the experiment has run for at least 2 weeks with no trend
• OR business needs require a decision

What to do next

Congratulations — you have run your first A/B test! Here is how to build on this:

• Run another test — Use the losing variation’s insights to form a new hypothesis
• Try multivariate testing — Test multiple elements simultaneously
• Add personalization — Deliver the winning variation only to specific audiences
• Connect to ODP — Target experiments to behavioral segments for higher impact

Related content

Continue your journey with these related pages.

📖Understand the concepts

• Web Experimentation
• Experimentation in Optimizely
• Personalization

🔧Follow a guide

• Experimentation guides