The A-Z on A/B Testing: Why It Matters & What You Can Test

Think about the last time you clicked on a ‘learn more’ button or entered your email to subscribe to a company’s newsletter.

The content you interacted with was most likely the result of an A/B test. 

You were compelled to engage with it because it was tested against a similar, though slightly different version of the same content. The content that caught your eye was the winner: more users acted as a result of it than a version sent to a separate segment of users.

We may never know what happened to all of the content that came 2nd in an A/B test. Farewell, less engaging content. 

If you’re new to A/B testing, then welcome aboard. This article explains why you should care about A/B testing and what you can and should test to launch your click-through rates on a trajectory up and to the right!

What exactly is A/B testing?

An A/B test (also referred to as split tests) is simply testing multiple versions of something to see which one performs better. 

How did A/B testing come about? It started with beer over 100 years ago, if you can believe. A brewer at Guinness named William S. Gossett split-tested different barley to identify the best way of brewing the famous Irish stout.

Fast forward to the present day, and A/B testing is now one of the most popular methods used by companies to optimize everything they do online, from website design to app development to digital marketing.

What should you test? 

Everything you put online for people to see could technically be split-tested. If you had all the time in the world, you could A/B test every detail within every piece of content you produce - across every landing page, in each of your paid ads and, in all of your newsletters. The placement, color, font, and size of every element on all of your landing pages could be tested to see what comes out on top between 2 or more options.

We recommend you focus on testing elements with the greatest potential to increase conversions, including: 

• headlines,
• call to actions (CTAs,) 
• buttons,
• sales copy or a product description, and  
• supporting graphics and images.

What’s a conversion? It’s what happens when someone completes a desired goal online, set by your company, like purchasing your product or supplying their email address for your mailing list. Your conversion rate is the percentage of people you convert out of the total number of site visitors. Here are typical conversion goals: 

• Increase sales of your product or service (e-commerce) 
• Increase leads (business to business) 
• Increase signups to your mailing list 
• Increase form completions 

The higher your conversion rate, the better quality digital marketing and content design you have. The key is to provide people with what they want and need as seamlessly as possible.

How does testing work? 

An A/B test begins with a piece of content, known as a ‘control.’ 

You present your control to 50% of your users, and present the other 50% of users with a ‘variation’ of the control:

You can have multiple variations, for example, presenting 25% of traffic to 1 control and the other 75% with 3 different variations, as long as you have enough traffic to split your users into groups while still producing qualitative findings. This type of test is known as multivariate testing. 

Multivariate testing is the big sister of A/B testing. It follows the same premise but compares a larger number of variables, and you’re able to capture more advanced results. You can target specific elements within your variations to see which ones have the most significant positive or negative impact on engaging your customers. Multivariate tests save a lot of time, as you can run multiple variations with different elements concurrently.

Testing framework

To set you up for success, follow this framework for running your A/B tests:

Look at your existing data: This will help you decide what to optimize. Focus on high traffic areas to gather qualitative findings. Does one of your ads have a low conversion rate? Or are you getting low click-throughs to a particular landing page? 

Set goals: What do you want to do? Increase sales? Lower cost per acquisition? Decide what you want to achieve so that you can easily measure if a variation of your content does a better job.

Generate variations: Create different versions of your existing content, whether it’s an ad, an app, or a landing page. Try changing the copy of descriptions, tweaking headlines, and tightening up CTAs. Focus on positioning your product or service as a solution to a problem your customers face.

Perform your test: This is where the fun begins! Sit back and let your users interact with your control and variation to see which one is most engaging. Try not to rush this process, you want qualitative findings to produce the most robust results. 

Analyze findings: Your test is finished; now what? It’s time to review the results. Look at the statistical difference between your control and variation(s) and pay close attention to any margin of error. 

Understanding your results 

Interpreting A/B results takes skill. If you’re not an expert in conversion rate optimization, partner with a team that can comprehend the data for you. There’s also software that helps digest findings from A/B tests (like Optimizely) through rapid experimentation, enabling you to measure impact and make changes that give you extraordinary results.

Take the results from the graphic above, for example. The report would have looked something like this: Control: 4.5% (+/- 1.8%) Variation 8% (+/- 2.1%). This result means 8% of users clicked through on your variation, with a margin error of 2.1%. The lower the margin of error, the better. It means that the result you get for the A/B test (a sample of your website visitors) is close enough to the result you would get for all your website visitors.

Creative A/B testing gets everyone here at Galactic Fed pretty excited. We love using data-driven marketing tactics to help companies reach new customers online. Using inventive static tests, we built a stronger draw for the product of an early-stage telehealth startup. Our testing increased their conversions by a factor of 10 in just a week, lowering their cost per acquisition (CPA) by 68%. 

Benefits of testing 

A/B testing is a great way to address several challenges you may have online when engaging new and returning customers.

A/B testing can help you: 

• Address customer pain points 
• Get more conversions, with less money spent 
• Improve user engagement 
• Reduce bounce rates 
• Make low-risk modifications  
• Redesign your website
• Adjust the price of a product 
• Adjust a feature or service 

Whatever issues you may be grappling with in relation to your digital marketing, Galactic Fed can help. From your website wireframes to the color of your CTA button, we’ll take a magnifying glass to your online marketing funnel to improve every customer touchpoint, turning high-quality traffic into new customers.

A/B testing FAQ

What’s an A/B test?

An A/B test is the process of testing multiple versions of something to see which one performs best.

What’s a control?

A control is an original piece of content you put through an A/B test. It can be a paid ad, a landing page, or your website homepage, for example. 

What’s a variation? 

A variation is a different version of your control. It can include changes to the position of elements such as buttons, changes to the copy of headlines or descriptions, and adjustments to color and imagery.

What’s a conversion?

A conversion is the act of your users completing a goal you set online. Buying your product, submitting your form, and generating a lead are all examples of conversions. 

What’s a conversion rate?

Your conversion rate is the percentage of people you convert out of the total number of visitors.

What’s a multivariate test?

A multivariate test is similar to an A/B test, except with an A/B test, you can only test one variable. With multivariate tests, you can test multiple variations with contrasting elements at the same time. 

What’s statistical significance?

Statistical significance is the probability that the difference in a conversion rate between a specific variation and the control isn’t by chance. If your A/B test produces a statistical significance level of 98%, you can be 98% confident the results of the test are accurate and not a result of randomness (there’s a 2% chance that you’re incorrect.) 

Get to grips with customer pain points by A/B testing

Now that you’ve got the lowdown on A/B testing, why not give it a try? You could increase sales or generate a ton of leads for your business. Just make sure you focus on elements with the most potential to increase your conversion rate - like headlines, CTAs, images and copy. Get to the bottom of a customer pain point that’s stopping your business from growing or tackle that high bounce rate on one of your landing pages. The sky’s the limit! Curious about using multivariate tests to boost click-throughs? The team at Galactic Fed would love to assist. Contact us today.