Run A/B tests
This guide shows you how to easily run A/B tests in Prepr CMS.
This guide is focused on the typical A/B testing activities of a marketer. Check out the set up A/B testing guide for developers for the one-time setup in the front-end app.
Introduction
Let's look at an everyday use case. You want to increase the clickthrough of visitors who end up on a specific landing page and have prepared two potentially successful headers but are unsure which one will appeal to customers the most. In this case, you can run an A/B test.
A/B testing is a simple and efficient way to compare two versions of something to figure out which performs better. In Prepr, you can create A and B variants of a section header, then show them to your website visitors randomly. Check out a demo A/B test in action (opens in a new tab).
The image below is an A/B test in the Acme Lease website. It shows the A and B variants of the section header on the Used Car Lease landing page.
Prerequisites
To run A/B tests based on this guide, you need to have the following set up in Prepr:
We will make use of the Landing page Occasion Lease content item from the Acme Lease demo data.
At the end of this guide, your content item should look like the example below. After adding an A/B test to your content, you won't see any metrics data initially until a developer adds the one-time setup and web app visitors start interacting with your A/B test content.
Create an A/B test in Prepr
Add the A/B test to a stack element
Go to the Content tab, click the Landing page Occasion Lease content item and add the A/B test as follows:
- To start an A/B test, simply click the icon on the section header. When you do this, a B variant will be created, which is a copy of the A variant.
- Update the images and text for each variant.
That’s it. You’ve just set up the A/B test in Prepr.
Once you’re ready, click Publish and close.
At the moment, it’s not possible to set which visitor percentage will see a certain variant. Variant A and Variant B both get shown to 50% of all customers. You can also run an A/B test with a single item (Variant A). In that case, 50% of your users will see the selected item and the other 50% won’t see it.
Evaluate the A/B test
After a developer has done the one-time setup for A/B testing and the A/B testing has been active for some time, you'll see some metrics data for your A/B test.
When the metrics for the A/B test are available and already significant for both variants, you can choose the winner and end the experiment.
Go to your A/B test in the content item and click the Metrics link to open the metrics report. If your A/B test has been running for a while, you should see significant results like in the image below.
The metrics data can be filtered by by a Date range (Maximum 90 days, defaulted to the start of the experiment) and the Segment. The color marker at the beginning of each variant indicates if the metrics are significant (green) or not (orange). Whether the metrics are significant is based on the confidence level and a minimum sample size.
See the list below for the metrics available in Prepr.
- Impressions (IMPR): The number of times that the element is viewed by visitors. The developer sets this element during the A/B testing setup.
- Conversions (CONV): The number of times that a desired action is performed on the element by visitors, such as a click. The developer sets this element during the A/B testing setup.
- Conversion rate: A percentage based on the conversions and the impressions. It is calculated as
(Impressions/Conversions)×100. - Standard deviation: You'll notice a +/- percentage value next to the conversion rate. This value tells you how much the conversion rates vary from the mean.
- Uplift - The increase in the conversion rate for the variant (B) compared to the control (A). It is calculated as: (Conversion Rate (B) − Conversion Rate (A)) / Conversion Rate (A) × 100
- Confidence level (CL) - This is the confidence level. In other words, the probability that this variant will outperform the control.
End the A/B test
If you don't have a significant result, then we recommend to continue the A/B test until the results are significant.
When you get a significant result, you can then choose the winning variant by clicking the Select A or the Select B button. Prepr automatically highlights the best performing variant based on the data collected.
After you selected A or B, Prepr will deliver the chosen variant to your website visitors. The other variant will be removed from the Stack.
Congratulations, you have successfully implemented an A/B test and used it to improve the customer experience on your web app.
Other settings
In the steps above, we guided you through a standard process to run A/B tests. In addition to these standard actions, click the icon in your A/B test group to make the following options visible:
- Reset data
After some time, your live A/B test will accumulate some data and these give you insights into how the variants perform against each other. It's possible that from these insights you want to make adjustments to the variants and re-evaluate the adjusted variants instead of creating a new A/B test. In this case you can reset the metrics data. When you do this Prepr starts tracking and collecting data from scratch as if it's a new A/B test.
Note that if you reset the metrics data, you can no longer view the previously tracked metrics.
- Remove A/B test
You can also remove an A/B test at any time by clicking the Remove A/B test button. If you choose to do this, both variants will be available as separate elements in the Stack field and it's up to you to decide if you want to keep both, delete one or delete both of them.
Want to learn more?
Check out the following guides:
Schedule a free consultation
Do you want to get started with A/B testing but still have questions or want a demo?
Schedule a free call (opens in a new tab) with a Prepr solution engineer.
Was this article helpful?
We’d love to learn from your feedback