How to set up a split test and measure the results!

When you create a piece of content with Bunting, you run through a 3 step process, Visitor targeting > Content > Measure Impact.

To set up split testing, all you have to do is change the drop-down under “Run a split test to measure impact?” on the Measure Impact page to Yes and let Bunting do all the hard work for you. Once you have done this click Save Changes and you are ready to start collecting Results when your Campaign is Activated.

Running a split test demonstration
Running a split test within the Bunting platform.

Where can I see the Results?

In the campaign lists within Bunting when a split test has been set up for the campaign you can see the results on the right-hand site similar to the image below.

If the Split test has not yet reached its confidence target within Bunting then it will still display that the split test between variations is still active.

If you click on one of these split test result figures it will take you to the Split Test Results page for that campaign and will display all the available metrics:

  • Converted Visitors
  • Revenue
  • Cart Products per Visitor
  • Visits per Visitor
  • Average Visit Duration
  • Bounce Rate
  • Product Views per Visit
  • Page Views per Visit
Split Test Results Page
Image showing split test results page within Bunting.

You can even select a specific variation to see the results for that variation at a much more granular level. Showing a graph of results as well as a history of the variation’s performance.

There is an advanced option to alter the amount of traffic that is able to view the split tests. The default for this is 50/50, but this can be catered to the specific campaign you are testing.

Advanced Split Test Screen
Advanced Split Test Results Screen

 

In addition to this, the confidence target can be changed. This is the metric that is calculated by Bunting as the probability for that variation to be the most effective. Once this target is reached it will deactivate the other variations.

All of the traffic is then routed to the winning variation from the deactivated variations and the control group.

To ensure there are no false positives Bunting will only automatically end the split test and show this campaign to all visitors when the confidence target is reached with more than 200 conversions.

Note: Google analytics has been turned on for this Campaign to further measure the impact. If you want to read more about Google analytics event tracking and what information Bunting passes to it click here.

Server Side A/B Split Testing with Bunting

Server-side split testing with Bunting is quick and easy to set up with this guide. However, before we continue, it is important to define what a server side split test is in Bunting, and how it differs from a client side split test.

Definitions

Server side: Content is delivered from your servers before page load, not from Bunting after page load. Your server must define who to show content to, and informs Bunting via Javascript which variation/control group each visitor is in.

Client side: Content is delivered from Bunting’s servers via Javascript. Bunting handles the visitor targeting, and manages the distribution of visitors across each of the different variation/control groups for you.

Continue reading Server Side A/B Split Testing with Bunting

What is a Control Group?

What is a Control Group?

Control Groups are portions of your visitors who are not exposed to the sales material presented by Bunting. They represent your website as it would be performing without Bunting. Control Groups allow you to accurately measure improvements (or drops) in your bottom line sales.

The Site-wide control group

By default Bunting isolates 10% of your website’s total visitor base into a Control Group, to demonstrate how well Bunting is working for your business. We believe in being fair and transparent, and hence the sales improvement figures appear right on your desktop. If you want to increase the percentage of visitors who go into your control group then you can do via your website monitor settings page. However, please keep in mind that a higher the control group percentage will mean the less visitors will be exposed to personalised advertising material, and therefore you may unnecessarily miss out on sales. Your website monitor control group can range from 10% – 50%.

Action-specific control groups

In Bunting you can optionally set Control Groups for specific actions. Doing so allows you to split A/B test each action you put in place, in order to scientifically measure the effect on your bottom line sales. To do this, simply set a Control Group when you are creating or editing an action, using the ‘Control Group’ field. If you don’t want to set a control group for an action then simply set the Control Group to ‘None’. This is relevant if you’re setting a personalisation feature on your website that you don’t expect or intend to affect your bottom line sales.