Running A/B tests
This guide shows you how to easily run A/B tests in Prepr CMS.
This guide is focused on the typical A/B testing activities of a marketer. Check out the set up A/B testing guide for developers for the one-time setup in the front-end app.
Introduction
Let's look at an everyday use case. You want to increase the clickthrough of visitors who end up on a specific landing page and have prepared two potentially successful headers but are unsure which one will appeal to customers the most. In this case, you can run an A/B test.
A/B testing is a simple and efficient way to compare two versions of something to figure out which performs better. In Prepr, you can create A and B variants of a section header, then show them to your website visitors randomly. Check out a demo A/B test in action (opens in a new tab).
The image below is an A/B test in the Acme Lease website. It shows the A and B variants of the section header on the Used Car Lease landing page.
Prerequisites
To run A/B tests based on this guide, you need to have the following set up in Prepr:
We will make use of the Landing page Occasion Lease content item from the Acme Lease demo data.
At the end of this guide, your content item should look like the example below.
After initally adding an A/B test to your content, you won't see any metrics data initially (Awaiting data) until a developer adds the one-time setup and web app visitors start interacting with your A/B test content.
Create an A/B test in Prepr
Add the A/B test to a stack element
Go to the Content tab, click the Landing page Occasion Lease content item and add the A/B test as follows:
- To start an A/B test, simply click the icon on the section header. When you do this, a B variant will be created, which is a copy of the A variant.
- Update the images and text for each variant.
That’s it. You’ve just set up the A/B test in Prepr.
Once you’re ready, click Publish and close.
Manage the A/B test
You can make some changes to your A/B test, for example, if you want to change the default traffic distribution where Variant A and Variant B both get shown to 50% of all customers. The options listed below are available to you to make changes to your running A/B test. Click the icon in your A/B test group to see these options.
- Manage settings
Choose Manage settings to change the name of the A/B test group or to change the traffic distribution of Variant A and Variant B.
- Reset data
After some time, your live A/B test will accumulate some data and these give you insights into how the variants perform against each other. It's possible that from these insights you want to make adjustments to the variants and re-evaluate the adjusted variants instead of creating a new A/B test. In this case you can reset the metrics data. When you do this Prepr starts tracking and collecting data from scratch as if it's a new A/B test.
Note that if you reset the metrics data, you can no longer view the previously tracked metrics.
- Remove A/B test
You can also remove an A/B test at any time by clicking the Remove A/B test option. If you choose to do this, both variants will be available as separate elements in the Stack field and it's up to you to decide if you want to keep both, delete one or delete both of them.
You can also run an A/B test with a single variant by deleting one of them. In that case, by default 50% of your users will see the selected item and the other 50% won’t see it. Go to the Manage settings option to change this traffic distribution.
Evaluate the A/B test
After a developer has done the one-time setup for A/B testing and your A/B test has been active for some time, you'll see some metrics data for your A/B test.
When the metrics for the A/B test are available and already significant for both variants, you can choose the winner and end the experiment.
Go to your A/B test in the content item and click the Metrics link to open the metrics report. If your A/B test has been running for a while, you should see some results like in the image below.
The metrics data can be filtered by a Date range (Maximum 90 days, defaulted to the start of the experiment) and the Segment. You can also choose to view the metrics data by a specific Conversion event. The color marker at the beginning of each variant indicates if the metrics are significant (green) or not (orange). Whether the metrics are significant is based on the probability and a minimum sample size.
See the list below for the metrics available in Prepr.
- Dwell time - The average amount of time in seconds that customers view this element.
- Impressions (IMPR) - The number of times that the element is viewed by visitors. The developer sets this element during the A/B testing setup.
- Conversions (CONV) - The number of times that a desired action is performed on the element by visitors, such as a click. The developer sets this element during the A/B testing setup.
- Conversion rate - A percentage based on the conversions and the impressions. It is calculated as
(Impressions/Conversions)×100. - Standard error - You'll notice a +/- percentage value next to the conversion rate. This value tells you how much the conversion rates vary from the mean.
- Uplift - The increase in the conversion rate for the variant (B) compared to the control (A). It is calculated as: (Conversion Rate (B) − Conversion Rate (A)) / Conversion Rate (A) × 100
- Probability (PROB) - This is the probability that this variant will outperform the control.
End the A/B test
If you don't have a significant result, then we recommend to continue the A/B test until the results are significant.
When you get a significant result, you can then choose the winning variant by clicking the Select A or the Select B button. Prepr automatically highlights the best performing variant based on the data collected.
After you selected A or B, Prepr will deliver the chosen variant to your website visitors. The other variant will be removed from the Stack.
Congratulations, you have successfully implemented an A/B test and used it to improve the customer experience on your web app.
Want to learn more?
Check out the following guides:
Schedule a free consultation
Do you want to get started with A/B testing but still have questions or want a demo?
Schedule a free call (opens in a new tab) with a Prepr solution engineer.
Was this article helpful?
We’d love to learn from your feedback