A/B Test Personalized Content
Why personalization needs experimentation
Section titled “Why personalization needs experimentation”Personalization without measurement is assumption. You create visitor groups, build targeted content variations, and deploy them to segments you believe will respond — but without experimentation, you have no proof that the personalized version outperforms the default. Worse, a poorly targeted personalization can reduce engagement.
Experimentation closes this gap. By wrapping personalized content in an A/B test, you compare the personalized experience against the control for each audience segment. You get statistical evidence that your personalization strategy works before committing to it permanently.
This recipe connects three Optimizely capabilities: CMS content management (where content lives), visitor groups (which define audience segments), and experimentation (which measures impact).
Architecture overview
Section titled “Architecture overview”┌──────────────────────────────────────────────────────┐│ Visitor arrives │└─────────────────────────┬────────────────────────────┘ │ ▼┌──────────────────────────────────────────────────────┐│ Visitor Group Evaluation ││ (CMS evaluates criteria: location, behavior, ││ device, referral source, ODP segments) │└─────────────────────────┬────────────────────────────┘ │ ┌─────────┴─────────┐ │ │ ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ Matches VG │ │ No VG match │ │ "Returning │ │ (default │ │ Customer" │ │ audience) │ └───────┬──────┘ └──────┬───────┘ │ │ ▼ ▼┌──────────────────────────────────────────────────────┐│ Experimentation Layer ││ Assigns visitor to experiment variation: ││ • Control (original content) ││ • Variation A (personalized hero + CTA) ││ • Variation B (personalized hero, default CTA) │└─────────────────────────┬────────────────────────────┘ │ ▼┌──────────────────────────────────────────────────────┐│ CMS Renders Content ││ Selected variation content served from CMS ││ via Graph (SaaS) or direct rendering (PaaS) │└─────────────────────────┬────────────────────────────┘ │ ▼┌──────────────────────────────────────────────────────┐│ Results Collection ││ Conversion events tracked back to Experimentation ││ Results segmented by visitor group membership │└──────────────────────────────────────────────────────┘How the products connect
Section titled “How the products connect”- CMS manages the content variations — the default page, the personalized hero banner, the alternative call-to-action. Content authors create these variations in the editor.
- Visitor Groups define the audience segments — returning customers, first-time visitors, users from a specific region. These groups evaluate visitor attributes at request time.
- Experimentation controls which variation each visitor sees and measures the outcome. The experiment uses visitor group membership as a targeting criterion, so only visitors in the target segment enter the test.
Step 1: Define visitor groups in CMS
Section titled “Step 1: Define visitor groups in CMS”Create visitor groups that represent the audience segments you want to personalize for.
Example visitor groups for this recipe:
| Group name | Criteria | Purpose |
|---|---|---|
| Returning Customers | Visit count > 3 AND has purchase history | Users who already know the product |
| High-Intent Browsers | Viewed pricing page AND session time > 2 min | Users close to a decision |
| New Visitors | First visit | Users who need education, not upsell |
Step 2: Create content variations in CMS
Section titled “Step 2: Create content variations in CMS”For each visitor group, create the content variations you want to test.
Example for the “Returning Customers” group:
- Control: The default homepage hero with a generic value proposition
- Variation A: A personalized hero that acknowledges returning status (“Welcome back”) with a direct link to their account or last-viewed product category
- Variation B: The personalized hero from Variation A but with the default CTA instead of the personalized one (to isolate the effect of CTA personalization)
On CMS PaaS, create these as separate content blocks referenced by a ContentArea. On CMS SaaS, create component variations in Visual Builder.
Step 3: Set up the experiment
Section titled “Step 3: Set up the experiment”In Optimizely Experimentation, create an A/B test that targets only visitors in the relevant visitor group.
// In the Optimizely Web Experimentation UI:
// 1. Create a new A/B test
// 2. Under Audiences, select "Returning Customers" visitor group
// 3. Define variations:
// - Control: Show default hero content
// - Variation A: Show personalized hero + personalized CTA
// - Variation B: Show personalized hero + default CTA
// The experiment code swaps the hero component:
var heroContainer = document.querySelector('[data-component="hero"]');
// Variation A: Personalized hero with personalized CTA
if (variationId === 'personalized_full') {
heroContainer.setAttribute('data-personalized', 'returning-customer');
heroContainer.setAttribute('data-cta', 'account-link');
}
// Variation B: Personalized hero with default CTA
if (variationId === 'personalized_hero_only') {
heroContainer.setAttribute('data-personalized', 'returning-customer');
// CTA stays default
}// Server-side experiment using Feature Experimentation SDK
var optimizelyClient = OptimizelyFactory.NewDefaultInstance(sdkKey);
var user = optimizelyClient.CreateUserContext(visitorId, new UserAttributes
{
{ "visitor_group", "returning_customer" },
{ "visit_count", visitCount },
{ "has_purchase_history", hasPurchaseHistory }
});
var decision = user.Decide("hero_personalization_test");
string heroVariant = decision.VariationKey;
// "control" | "personalized_full" | "personalized_hero_only"
// Pass the variant to your CMS content query
// to select the matching content variationStep 4: Configure metrics
Section titled “Step 4: Configure metrics”Define the metrics that measure whether personalization works.
Primary metric: Conversion rate (the action you want the visitor to take — purchase, signup, demo request)
Secondary metrics:
- Click-through rate on the hero CTA
- Time on page (do personalized visitors engage longer?)
- Bounce rate (does personalization reduce bounces?)
Configure these metrics in the Experimentation dashboard before starting the test.
Step 5: Analyze results
Section titled “Step 5: Analyze results”After the experiment reaches statistical significance:
- Open the Experimentation results dashboard
- Compare conversion rates across variations, filtered by the visitor group
- Look for interaction effects — does the personalized CTA add value beyond the personalized hero alone? (Variation A vs Variation B answers this)
- Check that the personalized experience does not harm other segments — if the experiment leaks to non-targeted visitors, verify no negative impact
Decision framework:
| Result | Action |
|---|---|
| Variation A wins significantly | Deploy full personalization (hero + CTA) for returning customers |
| Variation B wins | Personalized hero adds value but the CTA does not — deploy hero only |
| Control wins | The personalization hypothesis was wrong — revisit the visitor group criteria or content strategy |
| No significant difference | The sample size may be too small, or the change is not impactful enough — extend the test or try a bolder variation |
When to use this pattern
Section titled “When to use this pattern”This recipe is most valuable when:
- You have defined audience segments but no evidence that personalized content converts better
- Stakeholders disagree about what personalization means for different audiences
- You are expanding personalization to new segments and want to validate before scaling
- Regulatory or brand concerns require evidence before changing the default experience