A/B Test Recommendation Strategies

Why test recommendation strategies

Your first recommendation configuration is an educated guess. You chose an algorithm, set business rules, and picked a widget layout based on reasonable assumptions. But assumptions are not data. Does collaborative filtering outperform content-based matching for your audience? Do visitors engage more with a carousel or a grid? Does showing 3 items perform better than 6?

A/B testing answers these questions with visitor behavior data instead of opinions.

What you will do

1. Define what to test and how to measure success
2. Create test variations
3. Configure the A/B test
4. Run the test and monitor results
5. Apply the winning strategy

Define what to test

Recommendation tests fall into three categories. Pick one variable per test to get clean results.

Test category	What varies	Example hypothesis
Algorithm	The recommendation model	”Hybrid algorithm drives 15% more clicks than collaborative filtering”
Presentation	Widget type, layout, count	”A 3-item grid outperforms a 6-item carousel on mobile”
Rules	Business rules, filters, fallback	”Excluding previously viewed content increases click diversity”

Choose your success metric

Define a primary metric before starting the test:

Metric	What it measures	Best for
Click-through rate (CTR)	Percentage of visitors who click a recommendation	General engagement testing
Revenue per session	Revenue attributed to recommendation clicks	E-commerce optimization
Pages per session	Average pages viewed after recommendation click	Content engagement depth
Time on site	Additional time spent after clicking	Content quality validation
Conversion rate	Downstream conversions from recommendation clicks	Lead generation and sales

Create test variations

Test different models

To test algorithms, create two recommendation models with different configurations:

1. Navigate to Content Recommendations > Models
2. Ensure you have two models that differ only in the variable you are testing:
   • Control: Your current model (e.g., collaborative filtering, 4 items)
   • Variation: The challenger model (e.g., hybrid algorithm, 4 items)
3. Note the model IDs for both

Test different presentations

To test widget configurations, you will use the same model but render it differently:

Define widget variations

JavaScript

javascript Copy

// Control: grid layout with 4 items
const controlConfig = {
modelId: 'MODEL_ID',
widgetType: 'grid',
options: { count: 4, showImage: true, showDescription: true },
};

// Variation: carousel layout with 6 items
const variationConfig = {
modelId: 'MODEL_ID',
widgetType: 'carousel',
options: { count: 6, showImage: true, showDescription: false },
};

Configure the A/B test

Using Content Recommendations built-in testing

Content Recommendations includes a testing feature that handles traffic splitting automatically.

1. Navigate to Content Recommendations > Tests
2. Click Create Test
3. Enter a test name and description
4. Set the control — select the current model and widget configuration
5. Set the variation — select the challenger model or widget configuration
6. Configure traffic split (50/50 is standard; adjust if you need to limit exposure)
7. Set the primary metric
8. Set the minimum sample size or test duration:
   • Sample size — How many visitors each variation needs before results are reliable (minimum 1,000 per variation recommended)
   • Duration — Minimum run time to account for day-of-week and time-of-day effects (at least 7 days recommended)
9. Click Start Test

Cross-Product Integration

Recommendations

The built-in testing feature uses cookie-based visitor assignment to ensure the same visitor sees the same variation throughout the test. This prevents contamination from visitors seeing both variations.

Using Optimizely Experimentation

For more advanced test designs, use Optimizely Experimentation to control which recommendation widget renders:

Load widget variation based on experiment

JavaScript

javascript Copy

// After Optimizely Experimentation assigns a variation
function renderRecommendations(variationKey) {
const configs = {
  control: {
    modelId: 'MODEL_A',
    widgetType: 'grid',
    options: { count: 4 },
  },
  variation_1: {
    modelId: 'MODEL_B',
    widgetType: 'grid',
    options: { count: 4 },
  },
};

const config = configs[variationKey] || configs.control;

window.optimizelyContentRecs = window.optimizelyContentRecs || [];
window.optimizelyContentRecs.push({
  action: 'renderWidget',
  selector: '#recommendations',
  ...config,
});
}

This approach is useful when you want to combine recommendation testing with other page experiments or use Optimizely’s statistical engine for analysis.

Run the test and monitor results

Once the test is live:

1. Do not change anything during the test. Modifying models, rules, or widgets invalidates results.
2. Monitor the test dashboard daily for anomalies (sharp drops in traffic, technical errors)
3. Wait for the test to reach statistical significance — do not call a winner early based on small samples
4. Check both the primary metric and guardrail metrics (metrics that should not get worse, like page load time)

Reading results

Indicator	Meaning
Statistical significance > 95%	The observed difference is unlikely due to chance
Lift percentage	How much better the variation performs vs control
Confidence interval	The range of likely true improvement
Sample size met	Enough visitors have been tested for reliable conclusions

Apply the winning strategy

When the test concludes:

1. If the variation wins: update your production recommendation widget to use the winning configuration
2. If the control wins: keep the current configuration and test a different variable
3. If results are inconclusive: extend the test duration or increase traffic allocation

Document what you learned. Even losing tests provide insight into your audience’s preferences.

Iterate

Recommendation optimization is ongoing. After applying a winner, plan the next test:

• Test the winning algorithm with different business rules
• Test the winning widget layout with different item counts
• Test on different page types (homepage vs article page vs category page)

Each test builds a deeper understanding of what drives engagement for your specific audience and content.

Related content

Continue your journey with these related pages.

📖Understand the concepts

• Content Recommendations

🔧Follow a guide

• Set Up Recommendation Tracking
• Configure Recommendation Widgets
• Configure Recommendation Models