Run A/B tests

Run A/B tests

Easily add A/B testing to your web application using Prepr. This guide shows you how to run A/B tests in Prepr CMS.

This guide is focused on the typical A/B testing activities of a marketer. Check out the set up A/B testing guide for developers for the one-time setup in the front-end app.

Introduction

Let's look at an everyday use case: you are about to run a new promotion on your website and have prepared two potentially successful promotions but are unsure which one will appeal to your customers the most. In this case, you can run an A/B test.

A/B testing is a simple and efficient way to compare two versions of something to figure out which performs better. In Prepr, you can create A and B variants of a page header, one for each promotion, then show them to your website visitors randomly. Check out a demo A/B test in action (opens in a new tab) like in the image below.

A/B testing introduction (opens in a new tab)

After setting up an A/B test, you can use an analytics tool like Google Analytics to measure the click-through rate for each page header version and determine which is more effective.

To run A/B tests, you need to complete the following steps:

Step 1: Create an A/B test in Prepr. In this step, we show you how to create an A/B test with the variants that you want to show to your web app visitors.

Step 2: Get the HTML attributes for analytics. Get the HTML attributes of the A/B test elements from the front end.

Step 3: Set up analytics. Using your preferred analytics tool, you will measure and compare click-through-rates for each of the variants.

Step 4: Evaluate and close the A/B test. Once you know which variant performed better, we show you how to select it and end the A/B test.

Prerequisites

To run A/B tests based on this guide, you need to have the following set up in Prepr:

We will make use of the Home Page content item from the demo data which is based on the Page pattern.

Home page

Step 1: Create an A/B test in Prepr

Go to the Content tab, click the Home page content item and add the A/B test as follows:

  1. To start an A/B test, simply click the A/B test icon on the page header. When you do this, a B variant will be created, which is a copy of the A variant.
  2. Update the images and text for each variant of the promotion.

Add A/B test

That’s it. You’ve just set up the A/B test in Prepr.

Once you’re ready, click Publish and close.

At the moment, it’s not possible to set which visitor percentage will see a certain variant. Variant A and Variant B both get shown to 50% of all customers. You can also run an A/B test with a single item (Variant A). In that case, 50% of your users will see the selected item and the other 50% won’t see it.

Step 2: Get the HTML attributes for analytics

If not already done, a front developer needs to apply a one-time setup for A/B testing in the front end as explained in the A/B testing setup guide for developers. Once A/B testing is enabled in the front end, a correct variant will be displayed every time a customer visits the website according to your A/B test content. You can see either the A or B version by visiting the page from different devices or by switching to incognito mode when browsing.

From your browser, visit the page that was set up with the A/B test and right-click the A/B test element. Select Inspect to get the HTML attribute value for the id of the A/B test element and get the name of the variant HTML attribute like in the example below. Do the same for the id of the click element like a call to action link that you want to use to measure the CTR. You need to reference these attribute names accurately in the analytics tool in the next step to measure which variant performs best.

<section id="PageHeader_b997e121-5a8a-4527-a4f5-431029bdf" 
prepr-variant="A"  ...>
  <div>
    ...
    <a id="cta_link_PageHeader_b997e121-5a8a-4527-a4f5-431029bdf" href="#" ... >...</a>
  </div>
</section>

Step: 3 Set up analytics

Prepr A/B testing works with any analytics tool. If you already have your analytics tool set up to measure visitor interactions on your web app, please adapt the following explanation to your specific case. In this example we show you how to measure visitor interactions on the A and B variants using GA4 and GTM.

Set up GTM

  1. If not already done, work together with the front-end developer to connect GA4 and GTM to the web app by following the GA4 and GTM setup process (opens in a new tab).

  2. In GTM, create a custom variable (opens in a new tab) and set the Attribute name to the name of the variant HTML attribute that you found in the previous step.

prepr-variant variable

Create impression and click events in GTM

  1. Create tags in GTM (opens in a new tab) for the following events:
  • An element visibility event that fires when the page header section is 100% visible to the visitor.

Element visibility tag

Make sure that the visibility trigger as shown in the image below has the same Element ID value as the HTML attribute id value of the A/B test element that you found in the previous step.

Element visibility trigger

  • A click link event that fires when a user clicks the CTA link.

Click link tag Make sure that the click trigger as shown in the image below has the same Click ID value as the HTML attribute id value of the CTA element that you found in the previous step.

Click trigger

  1. Test your events by clicking on the Preview button and enter the URL of the web app that you want to test.

preview tag assistant

  1. If the tags are firing successfully and the variables are being populated as expected from the front end, click the Submit button to publish these tags to your web app.

submit -publish changes

Create reports in GA4

  1. Create a custom dimension in GA4 (opens in a new tab) so that the variant parameter is available for detailed reporting in Google Analytics.
Custom dimension
  1. Create a Funnel exploration in GA4. Click on the + button for DIMENSIONS to add the custom dimension A/B test variant. Add STEPS for the visibility and click events and specify a BREAKDOWN by the custom dimension. Funnel exploration

While the A/B test is active and users visit each variant, the reporting in your analytics tool will eventually show significant results on the variants.

Step 4: Evaluate and close the A/B test

The last step is to check the CTR for both variants and if the outcome is already significant, you can choose the winner and end the experiment.

Use this online A/B test calculator (opens in a new tab) to check if the collected data is significant enough to determine a winning variant as follows:

  1. From your report created in the previous step, copy the number of users from the visibility event from your report to fill in the Visitors A and Visitor B totals.
  2. Copy the number of users who completed the click event in the Conversions A and Conversions B fields.
  3. After you apply the changes, this calculator indicates if your result is significant or not.

online calculator

When you get a significant result, you can then choose the winning variant in Prepr by following the steps

  1. Go to the content item and click the End A/B test button.
  2. Click Keep A or Keep B to choose the best performing variant.

A/B testing select a winner

From now on, Prepr will deliver the chosen variant to your website visitors. The other variant will be removed from the Stack.

Congratulations, you have successfully implemented an A/B test and used it to improve the customer experience on your web app.

Want to learn more?

Check out the following guides:

Schedule a free consultation

Do you want to get started with A/B testing but still have questions or want a demo?

Schedule a free call (opens in a new tab) with a Prepr solution engineer.

Was this article helpful?

We’d love to learn from your feedback