A/B testing is an essential component of a conversion rate optimization (CRO) strategy. These experiments provide insights into user behavior and help validate website changes, enabling direct-to-consumer brands to keep up with evolving customer preferences and demands.
On average, ecommerce brands perform between 24 and 60 A/B tests every year. But how do you determine which are the best tests to run? And how can you make sure they’re effective?
This guide will explore how to implement A/B testing and what to test for your Shopify store. We’ll also provide five tips to help you get the most out of your A/B testing strategy.
A/B testing—also known as split testing—compares two or more versions of a web page or landing page. At minimum, you’d create a control version (Version A) and a variant version (Version B). You’d then show these variants to two randomized groups of customers simultaneously to determine which one works best and make website optimizations accordingly.
When it’s part of a comprehensive CRO strategy, A/B testing helps identify and validate site changes that have the following benefits:
A/B testing is not the only method for determining the impact of website changes, though.
While slightly more rare, and only possible for bigger brands with a lot of traffic, multivariate testing (MVT) is an alternative to split testing.
With A/B testing, you can test two or more versions of a web page or a specific element on a page to understand user behaviors. By extension, you can develop hypotheses about how to influence their behaviors to increase average order value or boost conversion rate, for example. These tests can get you insights faster than their multivariate counterparts. And, because they typically test significant website changes, they can often result in more impactful results.
Alternatively, multivariate testing compares different variants of a webpage or different combinations of changes to see which one performs best. For example, you might want to test combinations of different headlines, subheadlines, hero images, and buttons in the hero section of your site.
These two types of testing complement each other. A/B testing helps brands gather intel on their customers and develop overarching hypotheses, while MVT can help you to further optimize and find the right combination of elements.
It must be said, however, that MVT isn’t necessary for most brands doing minimal A/B testing. Plus, for statistical reasons, it requires a big website, so it’s not worthwhile for lower-traffic ecommerce sites.
For instance, while a simple A/B test might only have two variants, an MVT test might have a different variant for each of several changes to test all possible combinations. So, say you have two versions of a headline, a subheadline, and a button. You’d have the equivalent of nine A/B test variants. You’d need a high volume of traffic to get enough eyes on each variant to produce accurate results.
With A/B testing, brands can test nearly anything—header copy, layout, graphics, buttons, offers, shipping price, or entirely new elements. With so many options, it can be challenging to know where to start. Here’s a step-by-step guide to conducting A/B tests.
You may already have an idea of what you want to test. For example, you might have a landing page with a high bounce rate and think tweaking the design could reduce it. While this may be true, relying on intuition can waste time and resources. By starting with research, you can create a data-driven strategy for testing.
At SplitBase, we use our Testing Trifecta framework to identify opportunities to improve conversion rates and other metrics. Our approach starts by examining your current sales conversion funnels. This includes digging into Google Analytics, clickmaps, heatmaps, and scrollmaps to understand the customer journey. Where do they fall off?
Image description: SplitBase’s Testing Trifecta methodology
This quantitative research reveals what the problem is.
Qualitative research reveals the root cause of problem areas on your site. Here, brands conduct audience research to identify the desires and pain points affecting conversion.
There are a variety of methods to gather information about your customers, such as customer interviews, surveys, usability testing, and session recordings. Reviewing customer service chat logs is also helpful to understand customer obstacles, and Shopify stores can capture great qualitative feedback through post-purchase surveys, which can be automated using tools like KnoCommerce or Fairing.
By conducting research, brands identify areas for improvement and potential solutions. Next, it’s time to put those solutions to the test.
Start by using your research to inform your hypothesis. For example, we partnered with Dr. Squatch, a personal care and organic soap brand, to conduct a site-wide audit. During our research, we noticed many customers purchased several soaps at one time. However, the product page did not have a quantity field. Instead, shoppers had to click “add-to-cart” multiple times. We hypothesized that adding a quantity field would increase quantities purchased and, therefore, boost average order value.
Image description: Adding quantities to Dr. Squatch’s product pages.
A/B testing requires careful research and planning to ensure that the results are statistically significant. (Statistical significance is the probability that outcomes in a study are not due to random chance.) For example, a press mention or unexpected news event during your testing window may skew your results.
A/B testing is similar to a science experiment. Brands must form a hypothesis, create control and variant samples, and set testing parameters. Here are some elements to focus on when designing your test:
If you didn’t have the appropriate sample size or number of variations, or if you didn’t run your test long enough, it would make your results unreliable. Needless to say, making optimization decisions based on misleading data is more likely to hurt your conversion rate than help it.
Numerous A/B testing tools are available, including Convert, Optimizely, and VWO. These tools simplify creating, running, and analyzing A/B tests without advanced coding skills. Shopify stores looking to test things like shipping thresholds or subscription offers, will need specialized apps such as Shipscout and Rebuy.
For tests that involve more than changing text, do not use the visual editor provided by A/B testing tools, as they auto-generate code and often cause significant browser compatibility and code issues. Always use developers to code and validate your tests. This ensures your tests are compatible across devices and browsers for accurate results.
When you’re ready to run your test, split your audience into two or more randomized groups (A/B testing software automates this process). Group A will see the original version, and Group B will see the modified version. Make sure all key metrics are tracked, and collect data for each group.
Once your test is complete, study the results. Look at the metrics that you pre-determined were key to your experiment. These could include conversion rates, bounce rates, click-through rates, or average order value to name a handful. Was your hypothesis correct? If not, what else did you learn? Most A/B testing tools will have analytics and reporting features that will help you answer these questions.
A/B testing is an ongoing process. One experiment can lead to new questions and tests, so use your learnings to tweak and test different elements and maximize the effectiveness of your Shopify store.
With A/B testing, brands can evaluate a variety of elements, such as header copy, website layout, graphics, buttons, colors, or the checkout process. Here, we’ll explore how to perform specific A/B tests for your Shopify store.
Shipping thresholds are the minimum purchase amounts that customers must meet to qualify for free or discounted shipping. According to a study by Inmar Intelligence, 78% of shoppers prefer to buy more to receive free shipping. Adjusting shipping thresholds can reduce cart abandonment rates and increase AOV.
Your shipping threshold will vary by your industry, product type, margins, and shipping costs. Start by looking at your average order value and determine a minimum purchase amount just above it.
Use A/B testing to evaluate how your shipping thresholds will perform. For example, half your visitors receive free shipping on all orders while the other half receive free shipping on orders over $75 dollars.
Upselling and cross-selling are also valuable tools to increase AOV. Offering product bundling or subscriptions can motivate customers to buy more. For example, Curlsmith gives customers the option of a one-time purchase or a subscription at a discounted rate.
Image source: Curlsmith
Offering different quantities can also increase AOV. In the case with Dr. Squatch, we hypothesized that if the product page defaulted to two soaps, it would increase the revenue per user. Our A/B tests showed an increase of up to 54% in revenue per user with the proposed change.
Image description: Dr. Squatch’s product pages default to two bars of soap.
You can also A/B test where you cross-sell and upsell. Some places to consider adding cross-selling or upselling opportunities include:
Image description: Everlane provides a carousel of product suggestions in their cart pages.
Many brands use a mix of the above, which can provide some great inspiration for your own upsell and cross-sell flows. But, of course, testing is essential to understand what your target customers respond best to.
Customers can sometimes be overwhelmed by the number of options on a website. Personalized product recommendations combats this decision fatigue, motivating customers to buy.
A/B testing helps brands determine the best way to suggest products to visitors. This includes messaging, product placement, product recommendations quizzes, or using an AI-powered chat.
Image description: CurlSmith uses a pop-up to guide users toward a product quiz.
Image description: Haircare brand Amika includes an AI-powered chat on its homepage to suggest relevant product recommendations.
Product detail pages include a product’s description, specifications, color options, sizing options, images, price, customer reviews, and so on. These pages should help customers understand a product’s value and guide them to the next stage of the shopping process, providing clear calls to action (CTAs) that motivate customers to purchase.
A/B testing PDP layout, copy, or other elements is key for helping you optimize these pages to drive more conversions.
For example, SplitBase leveraged it to help hair extension brand INH increase conversions by 26%. Using our Testing Trifecta process—the foundation of which is deep customer research—we found that shoppers were confused about how to use the brand’s products. They weren’t reading the product descriptions on the INH’s product pages.
After testing various approaches to product demonstrations, including videos and images, we found that a combination of three GIFs worked best to boost conversion and return on ad spend (ROAS).
Image description: Including a product demonstration in INF’s product detail pages.
But this example is not to say that those three GIFs on the product page are a universal solution for brands who have the same problem INH did.
Ultimately, the brands that have the greatest success with testing and optimization develop test ideas by researching what does and doesn’t work for their specific brands and customers. Our Testing Trifecta, in particular, combines both quantitative and qualitative research to reveal what’s hindering conversions and why. Armed with that info, you can then develop strong hypotheses to address those elements.
A/B testing is deceptively simple. But if A/B tests yield unreliable data, you could make changes to your Shopify store that don’t align with customer preferences—threatening loyalty and revenue. Here are five best practices to follow when conducting A/B testing.
All of the above can make or break your conversion rate optimization strategy.
A/B testing is a powerful tool to help ecommerce brands keep up with evolving customers’ preferences, improving the customer experience, increasing ROI, and driving revenue. But creating a thoughtful A/B testing strategy is critical. That’s where SplitBase comes in.
SplitBase is the leading conversion optimization and landing page agency for ecommerce brands. We provide full-site A/B testing for your Shopify store, boosting customer acquisition, AOV, and conversions. Get a free ecommerce CRO proposal today to see how SplitBase can help optimize your Shopify store.
Unlock the secrets to CRO mobile optimization and e-commerce success. Learn proven strategies to optimize your mobile site or app in our complete guide.