Skip to content
Optimizely CertificationOptimizely Certification Optimizely Certification

Set Up A/B Testing on CMS Content

⏱ 15 minutes intermediate

Why use CMS A/B testing

Section titled “Why use CMS A/B testing”

A/B testing lets you compare two versions of a page to see which performs better. The CMS-native testing feature is simpler than full Web Experimentation. It works directly in the editor without requiring external snippet installation. Use it for straightforward content comparisons like headline tests or layout changes.

CMS A/B testing is a built-in feature for quick content comparisons. For advanced multivariate tests, audience targeting, or cross-page experiments, use the full Web Experimentation product.

Enable A/B testing

Section titled “Enable A/B testing”
  1. Confirm that A/B testing is enabled in Settings > Feature Configuration.
  2. Check that your user role has permission to create tests.
  3. Verify by opening any page in the editor. You should see the A/B Test option in the toolbar or context menu.

Create an A/B test

Section titled “Create an A/B test”
  1. Open the page you want to test in the editor.
  2. Click A/B Test in the toolbar (or right-click and select Create A/B Test).
  3. The system creates a draft copy of the page as Variation B.
  4. Edit Variation B with your alternative content (different headline, image, or layout).
  5. Verify that both variations appear in the test panel side by side.

Define a goal

Section titled “Define a goal”

Every test needs a measurable goal to determine the winner.

  1. In the A/B test setup panel, click Set Goal.
  2. Choose a goal type:
    • Page visit — visitor reaches a target page (thank-you page, checkout).
    • Click — visitor clicks a specific link or button.
    • Form submission — visitor submits a form on the page.
  3. Configure the goal details (target URL, element selector, or form name).
  4. Save the goal.
  5. Verify by checking the test summary. The goal should appear with its type.

Configure test settings

Section titled “Configure test settings”
  1. Set the traffic split percentage (default is 50/50).
  2. Set the test duration or minimum sample size.
  3. Choose the confidence level (typically 95%).
  4. Set whether the winner publishes automatically or requires manual approval.
  5. Save all settings. Verify the test summary shows your configuration.

Start the test

Section titled “Start the test”
  1. Review both variations and the goal one final time.
  2. Click Start Test.
  3. The system begins splitting visitor traffic between the two variations.
  4. Verify by visiting the page in an incognito browser multiple times. You should see both variations.

Monitor and conclude

Section titled “Monitor and conclude”
  1. Open the A/B Test panel to view real-time results.
  2. Monitor impressions, conversions, and conversion rate for each variation.
  3. Wait until the test reaches statistical significance (shown as a confidence indicator).
  4. Once a winner is determined, click Pick Winner or let auto-publish take effect.
  5. Verify that the winning variation is now the published page.

Common issues

Section titled “Common issues”
IssueCauseFix
Test not splitting trafficTest not startedClick Start Test
No conversions recordedGoal misconfiguredVerify the goal target URL or selector
Low confidence after many visitsVariations too similarMake a more distinct change
Test option not visibleFeature not enabled or missing permissionsCheck settings and user role