Set up A/B testing

Estimated duration: 30-45 minutes

Easily add A/B testing to your web application using Prepr. This tutorial demonstrates how to create an A/B test for stack field elements using the Prepr GraphQL API. Follow the step-by-step guide below.

Introduction

Let's look at an everyday use case: you are about to run a new promotion on your website and have prepared two potentially successful promotions but are unsure which one will appeal to your customers the most. In this case, you can do A/B testing.

A/B testing is a simple and efficient way to compare two versions of something to figure out which performs better. In Prepr, you can create A and B variants of a page header, one for each promotion, then show them to your website visitors randomly. Click on the image below to see a demo A/B test in action:

After that, you can use an analytics tool like Google Analytics to measure the click-through rate for each page header version and determine which is more effective.

To set up an A/B test in Prepr, you need to complete the following steps:

Step 1: Create an A/B test in Prepr. In this step, we show you how to create an A/B test with the variants that you want to show to your web app visitors.

Step 2: Implement the A/B test in the front-end code. This is a one-time update to the code which means that any updates to this A/B test will automatically be applicable without further code changes.

Step 3: Set up analytics. Using your preferred analytics tool, you will measure and compare click-through-rates for each of the variants.

Step 4: Evaluate and close the A/B test. Once you know which variant performed better, we show you how to select it and end the A/B test.

Prerequisites

To set up the A/B test, you need to have the following set up in Prepr before you implement the A/B test:

• A Prepr CMS account
• An environment with demo data in Prepr

We will make use of the Home Page content item from the demo data which is based on the Page pattern.

Step 1: Create an A/B test in Prepr

Go to the Content tab, click the Home page content item and add the A/B test as follows:

1. To start an A/B test, simply click on the A/B test icon on the page header. When you do this, a B variant will be created, which is a copy of the A variant.
2. Update the images and text for each variant of the promotion.

That’s it. You’ve just set up the A/B test in Prepr.

Once you’re ready, click Publish and close.

Step 2: Add the A/B test to the front end

In this step we look at how to do the following:

• Get a customer ID to identify visitors
• Retrieve the variants using the API
• Add HTML attributes to the page header component

Get a customer ID to identify visitors

When running an A/B test we have to indicate who is visiting the website to make sure that visitors get the same variant each time they visit the page. Prepr uses a customer ID to match a visitor to an A or B variant. You can use your own customer ID or an existing Prepr customer ID.

Retrieve variants using the API

Now it's time to retrieve the Home page content item including the page header variants.

Copy the sample code below into your front-end application. Update the id string of the query with the Home page Content item ID that you used in Step 1.

query{
  Page (id: "8da86294-c15d-415a-9fa3-6ea289c9403c") { 
    _id
    title
    _slug
    stack { 
      __typename
      ... on PageHeader {
        _id
        heading
        text
        cta_label
        _context {
          kind
          variant_id
          group_id
        }
      }
    }
  }
}

curl --location --request POST 'https://graphql.prepr.io/<YOUR-ACCESS-TOKEN>' \
--header 'Prepr-Customer-Id: 409ad1af-d644-4d3f-8cb9-691c0318a980' \
--header 'Content-Type: application/json' \
--data '{"query":"query{\n  Page (id: \"8da86294-c15d-415a-9fa3-6ea289c9403c\") { \n    _id\n    title\n    _slug\n    stack { \n        __typename\n    ... on PageHeader {\n        _id\n        heading\n        text\n        cta_label\n        _context {\n                kind\n                variant_id\n            }\n        }\n    }\n  }\n}","variables":{}}'

var myHeaders = new Headers();
myHeaders.append("Prepr-Customer-Id", "409ad1af-d644-4d3f-8cb9-691c0318a980");
myHeaders.append("Content-Type", "application/json");

var graphql = JSON.stringify({
  query: "query{\n  Page (id: \"8da86294-c15d-415a-9fa3-6ea289c9403c\") { \n    _id\n    title\n    _slug\n    stack { \n        __typename\n    ... on PageHeader {\n        _id\n        heading\n        text\n        cta_label\n        _context {\n                kind\n                variant_id\n            }\n        }\n    }\n  }\n}",
  variables: {}
})
var requestOptions = {
  method: 'POST',
  headers: myHeaders,
  body: graphql,
  redirect: 'follow'
};

fetch("https://graphql.prepr.io/<YOUR-ACCESS-TOKEN>", requestOptions)
  .then(response => response.text())
  .then(result => console.log(result))
  .catch(error => console.log('error', error));

// WARNING: For POST requests, body is set to null by browsers.
var data = JSON.stringify({
query: "query{\n  Page (id: \"8da86294-c15d-415a-9fa3-6ea289c9403c\") { \n    _id\n    title\n    _slug\n    stack { \n        __typename\n    ... on PageHeader {\n        _id\n        heading\n        text\n        cta_label\n        _context {\n                kind\n                variant_id\n            }\n        }\n    }\n  }\n}",
variables: {}
});

var xhr = new XMLHttpRequest();
xhr.withCredentials = true;

xhr.addEventListener("readystatechange", function() {
if(this.readyState === 4) {
  console.log(this.responseText);
}
});

xhr.open("POST", "https://graphql.prepr.io/<YOUR-ACCESS-TOKEN>");
xhr.setRequestHeader("Prepr-Customer-Id", "409ad1af-d644-4d3f-8cb9-691c0318a980");
xhr.setRequestHeader("Content-Type", "application/json");

xhr.send(data);

<?php
$client = new Client();
$headers = [
'Prepr-Customer-Id' => '409ad1af-d644-4d3f-8cb9-691c0318a980',
'Content-Type' => 'application/json'
];
$body = '{"query":"query{\\n  Page (id: \\"8da86294-c15d-415a-9fa3-6ea289c9403c\\") { \\n    _id\\n    title\\n    _slug\\n    stack { \\n        __typename\\n    ... on PageHeader {\\n        _id\\n        heading\\n        text\\n        cta_label\\n     _context {\\n                kind\\n                variant_id\\n            }\\n        }\\n    }\\n  }\\n}","variables":{}}';
$request = new Request('POST', 'https://graphql.prepr.io/<YOUR-ACCESS-TOKEN>', $headers, $body);
$res = $client->sendAsync($request)->wait();
echo $res->getBody();

To retrieve the page content, call the API using the query above and pass a Prepr Customer ID as a header. The Prepr Customer ID is required to determine which variant a visitor gets. Get a customer ID as shown above and use this value to set the Prepr Customer ID header value. See an example in the below code snippet:

--header 'Prepr-Customer-Id: 409ad1af-d644-4d3f-8cb9-691c0318a980' 

As a response to your API request, Prepr sends the variant to be displayed. And your app then needs to build its response based on this variant. See an example of a response below:

{
  "data": {
    "Page": {
      "_id": "8da86294-c15d-415a-9fa3-6ea289c9403c",
      "title": "Home page",
      "_slug": "home-page",
      "stack": [
        {
          "__typename": "PageHeader",
          "heading": "Bring a friend to any baking course for free",
          "text": "Sign up for any baking course in our calendar together with a friend and they can join for free!",
          "cta_label": "Sign up now!",
          "_context": {
            "kind": "AB_TEST",
            "variant_id": "A"
            "group_id": "34022364-7ed3-46ff-9055-a208c5b0cccd"
          }
        }
      ]
    }
  }
}

The algorithm remembers which visitor gets which variant so that the same visitor always gets the same page header variant when interacting with your web app.

Add HTML attributes to the page header

To track visitor interactions for A/B testing and get accurate reporting results, we need to add attributes to the page header. In our example, we have a Page header with some specific text and a call to action link for each variant, so we want to report on the following types of visitor interactions:

• Impressions: How many visitors looked at the page header for each variant.
• Clicks: How many visitors clicked the CTA link for each variant.

We can determine the click-through rate based on these two events, which will show which variant performed best. Because the above interactions need to be per variant, we also need a custom variable to identify the variant.

See an example code snippet to include the HTML attributes below:

<!-- Page header -->
<div
  id="{{__typename}}_{{group_id}}"
  prepr-variant="{{ variant_id }}"
  ...
>
<!-- Call to action link -->
  <a
    href="#" 
    id="cta_link"
    prepr-variant="{{ variant_id }}"
    ...
  >Learn more</a>
</div>

In the above example, we set the IDs dynamically by using the following fields returned in the query response:

• __typename - The element type, for example, PageHeader.
• group_id - A unique ID to identify a specific A/B test group. This is useful to identify all elements that have an A/B test enabled.
• variant_id - This returns a value of A or B.

Using the query response above as an example the IDs get the following values:

• id = "PageHeader_34022364-7ed3-46ff-9055-a208c5b0cccd"
• prepr-variant = "A"

That’s it. You have integrated the A/B test into your website. A correct page header variant will be displayed every time a customer visits your website.

The next step is to set up your analytics tool to measure the click-through rate to see which variant performs best.

Step 3: Set up analytics

Prepr A/B testing works with any analytics tool. If you already have your analytics tool set up to measure visitor interactions on your web app, please adapt the following explanation to your specific case. In this example we show you how to measure visitor interactions on the A and B variants using GA4 and GTM.

Set up GTM

1. Connect GA4 and GTM to your web app by following the GA4 and GTM setup process.
2. In GTM, create a custom variable to get the variant from the variant attribute that you created in the front end.

Create impression and click events in GTM

1. Create tags in GTM for the following events:

• An element visibility event that fires when the page header section is 100% visible to the visitor.

Make sure that the visibility trigger as shown in the image below has the same Element ID value as the page header ID created in the front-end in the previous step.

• A click link event that fires when a user clicks the CTA link.

Make sure that the click trigger as shown in the image below has the same Click ID value for the page header CTA ID created in the front end in the previous step.

2. Test your events by clicking on the Preview button and enter the URL of the web app that you want to test.

3. If the tags are firing successfully and the variables are being populated as expected from the front end, click the Submit button to publish these tags to your web app.

Create reports in GA4

1. Create a custom dimension in GA4 so that the variant parameter is available for detailed reporting in Google Analytics.

2. Create a Funnel exploration in GA4. Click on the + button for DIMENSIONS to add the custom dimension A/B test variant. Add STEPS for the visibility and click events and specify a BREAKDOWN by the custom dimension.

While the A/B test is active and users visit each variant, the reporting in your analytics tool will eventually show significant results on the variants.

Step 4: Evaluate and close the A/B test

The last step is to check the CTR for both variants and if the outcome is already significant, you can choose the winner and end the experiment.

Use this online A/B test calculator to check if the collected data is significant enough to determine a winning variant as follows:

1. From your report created in the previous step, copy the number of users from the visibility event from your report to fill in the Visitors A and Visitor B totals.
2. Copy the number of users who completed the click event in the Conversions A and Conversions B fields.
3. After you apply the changes, this calculator indicates if your result is significant or not.

When you get a significant result, you can then choose the winning variant in Prepr by following the steps

1. Go to the content item and click the End A/B test button.
2. Click Keep A or Keep B to choose the best performing variant.

From now on, Prepr will deliver the chosen variant to your website visitors. The other variant will be removed from the Stack.

Congratulations, you have successfully implemented an A/B test and used it to improve the customer experience on your web app. Check out the next section if you are implementing a static website.

Implementing A/B testing for static (SSG) websites

Unlike dynamic sites, static sites serve pre-built content to a web browser without calls to a database. To perform A/B testing for static site rendering, you need to set up variants ahead of time (particularly before you build the web app) and then split traffic between different routes using the edge middleware (see an example tool). Let’s see it in detail.

The A/B testing for static site rendering looks like this:

1. On pages where an A/B test needs to be triggered, you call the API to pre-fetch variants. Your request must include the following fields:

• _context - system field that contains additional details about variants such as the following fields:
  • kind – returns a value of PERSONALIZATION or AB_TEST.
  • variant - returns a value of A or B if kind has a value of AB_TEST

Here’s an example code snippet. Update the id string of the query with your Content item ID (find the ID in the right-hand column on the Page content item in Prepr).

query{
    Page (id: "8da86294-c15d-415a-9fa3-6ea289c9403c") { 
        _id
        title
        _slug
        stack (personalize: false) { 
            __typename
        ... on PageHeader {
        _id
        heading
        text
        cta_label
                _context {
                    kind
                    variant_id
                }
            }
    }
    }
}

{
  "data": {
    "Page": {
      "_id": "8da86294-c15d-415a-9fa3-6ea289c9403c",
      "title": "Home page",
      "_slug": "home-page",
      "stack": [
        {
          "__typename": "PageHeader",
          "_id": "79e752ca-76e5-4b52-8bf9-4a5960d0ccee",
          "heading": "Bring a friend to any baking course for free",
          "text": "Sign up for any baking course in our calendar together with a friend and they can join for free!",
          "cta_label": "Sign up now!",
          "_context": {
            "kind": "AB_TEST",
            "variant_id": "A"
          }
        },
        {
          "__typename": "PageHeader",
          "_id": "695afdf3-6b3e-4c05-8e2e-39dae548969e",
          "heading": "20% off on all bread baking courses",
          "text": "Join our Spring promotion and save 20% on any baking course.",
          "cta_label": "Sign up today!",
          "_context": {
            "kind": "AB_TEST",
            "variant_id": "B"
          }
        }
      ]
    }
  }
}

2. When a user visits your web app, you make a one-time request to know if this user is assigned to Bucket A or Bucket B. Your request must include the following headers:

• Prepr-Customer-Id with a session ID of a visitor
• Prepr-Bucket-Customer = true

The bucketing runs at the edge location of the CDN and takes no longer than 7 ms. A response to your API request contains a header – either X-Prepr-Customer-Bucket A or X-Prepr-Customer-Bucket B.

See an example request below:

curl --location --globoff 'https://graphql.prepr.io/<YOUR-ACCESS-TOKEN>' \
--header 'Prepr-Customer-Id: 409ad1af-d644-4d3f-8cb9-691c0318a980' \
--header 'Prepr-Bucket-Customer: true'

3. With this information, the front-end middleware can redirect a visitor to a respective variant of your A/B test. For all the requests from the same visitor, the same variant will be shown.
   
   

Once you’ve retrieved variants using the API, complete Steps 4, 5 and 6, described in the standard flow instructions.

Want to learn more?

Check out the following guides:

• Capturing event data
• Connecting your front-end application
• Personalization

Schedule a free consultation

Do you want to get started with A/B testing but still have questions or want a demo? Book a free appointment with a Prepr solution engineer right away.