Data Driven A/B testing

Written by Jake Waller

Jake is a technical author with a wealth of experience helping people from companies, charities and governmental departments understand new trends in technology.

August 3, 2021

The benefits of data-driven A/B testing

Google engineers ran their first A/B test in 2000 and they’ve used data driven A/B testing on a regular basis ever since.


Because data driven A/B testing allows you to compare two elements or versions of something against one another and see which version performs better. See which version delivers the best results.

Such testing eliminates the guesswork and provides results based on real responses and real data. Extra sales and customer conversions can be won or lost on the simplest of changes.

77 percent of companies perform A/B tests on their websites, but websites aren’t A/B testing’s only use.

The basic premise is one of logic, and so data driven A/B testing has a lot in common with managed scientific experiments which use a control group alongside the trial being carried out.

While the entity being tested is most often a webpage, it could equally be an ad campaign, a physical or digital product or prototype, or a system.  

How do A/B tests work?

In an A/B Tests you split your audience randomly, but equally. 50% see the original ‘control’ version and the other 50% see a variation which tests for improvement.

For a prototype, this could mean a new feature. In the case of web pages, the change being tested could be the wording or images used or maybe it’s a slight change in the layout.

Did you know that even changing the colour of a button can make a difference to how many people click on that button?

Red beats green according to Hubspot, though the debate rages—and it probably depends on what you are asking people to do as to which colour button works best. If you’re wanting someone to book an environmental retreat, then green might beat red in that scenario.

The benefits of data driven A/B testing

Regardless of what the change is, you should only ever test one change at a time and it’s important that audiences see the two variations within the same time period; this ensures variables such as seasonality or advertising don’t distort the result.

This is one of data driven A/B testing’s biggest benefits: by running the two experiments simultaneously, you remove several other variables that might otherwise affect the result.

Let your tests run long enough to collect enough data to make an informed decision. On average, a controlled experiment like A/B testing a website will need at least 25,000 visitors to reach a statistically significant sample.

You might not have as many visitors to your website as that, but you shouldn’t base your outcomes on just a few visitors either.

How to assess your results

The whole point of an A/B test is to statistically prove that one version of something is better than another based on data.

A common measure of “proof” is achieving a statistical significance of at least 95%, although other thresholds are used as well.

You can check for significance ratings using this calculator: A/B Testing Calculator for Statistical Significance | SurveyMonkey

You may not always get the obvious result you hoped for, though you should be wary of compromising by ‘going with your gut’ if the results do prove inconclusive.

If the results aren’t conclusive, you need to go back to the drawing board. Maybe you weren’t testing a change that mattered to your clients? Maybe you need to test bolder changes?

Make the best use of your A/B test results

Once you have some significant test results, you can then use these to improve your results over-all.

You can roll out the use of the winning formula to 100% of your audience and maximise on the improved audience response.

Maybe you can take the lessons you learned from your A/B experiment and apply them to other areas of your business? If changing a headline had such an impact in one area, maybe it will boost results in another area too?

Gain the competitive edge

There are lots of benefits to using A/B testing: the improved conversion rates, the evidenced boost to business, the overall confidence that what you are producing resonates with your audience…

Yet, perhaps the most significant benefit is the edge it gives you over your competition.

You are not copying anyone else or relying on instinct and instead you are finding new, proven ways to make a difference.

The evidenced successes that arise from A/B testing can get addictive, but that’s OK…

You can keep testing and keep improving all the time.

You May Also Like…

No Results Found

The page you requested could not be found. Try refining your search, or use the navigation above to locate the post.