In ecommerce, small changes can mean thousands in revenue gained or lost. Without testing, you’re gambling with real money. Suppose you tweak a headline, swap out one group of ads for another, or overhaul your entire home page because someone on your team has a good feeling about it. Operating on instinct may feel productive, but without data, you’re only making educated guesses.
A strong A/B testing strategy replaces guesswork. Instead of relying on opinions, you collect data on real customer behavior and validate what actually affects business goals. Learn more about why A/B testing is a powerful tactic for increasing ecommerce revenue, as well as how to run A/B tests designed to boost your conversion rates.
What is A/B testing?
A/B testing (also called split testing or bucket testing) is a controlled experiment comparing two versions of something to see which version performs better. In the context of ecommerce, this means testing an original version (Variation A) against an alternative (Variation B).
This could be as simple as sending the same newsletter with two different email subject lines, or as complex as testing multiple pages against each other. The simplest A/B testing method involves comparing one change at a time. With multivariate testing, you test multiple variations simultaneously to compare the effectiveness of different combinations of headers, body copy, and images.
Mastering the simpler style of experimentation is the foundation for exploring more complex approaches like multivariate testing. The key is to isolate specific testing elements, so you know exactly what differences influenced the outcome. The goal is to make changes simple enough that they’re easily traceable but substantial enough that users can notice the difference.
Common goals for A/B testing
A/B testing usually has goals tied to clear key performance indicators (KPIs). Some common A/B testing goals include:
-
Increasing product sales. You might test discount-oriented ad copy against ad copy that highlights your product’s benefits to see which drives more sales. You may even drill down to test specific words, like “moisturizing” versus “glow-enhancing.”
-
Improving add-to-cart rates. Test whether a “low-stock” or “just three left” tag on dwindling items convinces more people to pull the trigger on a purchase.
-
Reducing cart abandonment. See whether free shipping or a generous return policy keeps shoppers from abandoning their carts.
-
Growing email subscriptions or leads. A “10% off your first order” call to action (CTA) might be more compelling than a “Keep in touch” CTA on your newsletter pop-up.
-
Elevating click-through rates. Test different versions of CTA buttons at the end of your email newsletters to see which compels readers to visit your website.
One real-world example: The team behind the electric toothbrush company Suri A/B tested the copy on their pre-launch landing pages. The tests allowed them to refine their messaging and product feature copy before investing in inventory or more robust marketing tactics. “We were doing fake ads to emails to refine the concept further,” founder Gyve Safavi says on an episode of Shopify Masters. “As we started doing pre-sales, we were able to see what the communication message was that really worked to drive the sale.”
What is statistical significance in A/B testing?
Statistical significance tells you whether an A/B test’s results are meaningful enough to implement permanently. In simple terms, it answers the question: Did this variation truly perform better, or did we just get lucky? Sometimes, one variation might win over the other by a small amount, which could be attributed to chance or external factors. Customer behavior fluctuates day to day, so small conversion rate differences can happen randomly.
A statistically significant result means there’s a high probability that your result is reliable enough to act on, and is likely to happen consistently. Typically, you reach statistical significance when:
-
You’ve gathered enough total traffic to create a meaningful sample size.
-
The test has run long enough to account for normal behavioral variability (weekdays versus weekends, promotions, seasonality).
-
Your testing tool calculates a 95% or higher confidence level between variations.
You can also run A/A tests, in which you test identical versions, to determine baseline, normal fluctuations in engagement. In testing identical product pages, you might find that one group converts at 3.2% and the other at 3.4% due to random variation. So, when a future test shows a 5% lift from a new checkout button, you’ll know it’s a statistically significant result.
How do tools pick up on statistical significance?
Under the hood, A/B testing tools calculate statistical significance by comparing the performance of Variation A and Variation B against the total number of visitors each version received. The more traffic you send through the test, the more reliable the comparison becomes. When tools report a 95% confidence level, it means there’s only a 5% chance that the result occurred by chance. Most ecommerce teams use 95% as the standard threshold before declaring a winner.
For example, let’s say you test combinations of a benefit- versus feature-driven headline for a new pillow. The benefit-driven headline emphasizes how cool and comfortable the pillow stays all night, while the feature-driven one highlights the tech innovations that enable the cool comfort. When the benefit-driven headline converts at a rate 2% higher than the feature-driven one, statistical significance would tell you whether that 2% difference is real or just due to random chance.
Some reasons why a result may not be statistically significant include:
-
The sample size is too small. Each variation should have at least 1,000 visitors to achieve meaningful data.
-
The variation changes are too minor. A slight variation in adjectives or colors may be too subtle to make an impact on users.
-
User behavior variability is too high. Perhaps you have many micro audiences that all behave differently, making it difficult to test what changes resonate with which audience.
The more tests you run, the more data you’ll have to help determine what counts as a statistically significant change among your audience.
Marketing elements to consider A/B testing
Adam Davis, senior marketing manager at Magnolia Bakery, tells Shopify Masters that “test and learn” is a core philosophy of the brand. Adam’s team has tested everything from emojis in the navigation bar to upsells at checkout to promotion-specific landing pages. “We uncovered learnings where a banana pudding purchaser is more likely to add a cupcake to their cart, so we show that customer a cupcake on the sidebar once they have banana pudding already in their cart,” he says.
Here are just a few marketing elements you can A/B test:
-
Call-to-action buttons. Test CTA button sizes, colors, and copy to increase add-to-cart rates or button clicks.
-
Product images. Test lifestyle versus studio images, different angles, and moods to see which lead to more product page conversions.
-
Product descriptions. Test long versus short copy to improve conversion rates.
-
Pricing and discounts. Test $29.99 versus $30, or discount urgency to improve add-to-carts.
-
Homepage layout. Test different arrangements of sections, banners, and featured products to improve overall engagement and sales.
-
Email subject lines. Test personalization, emojis, and tone to improve open rates.
-
Checkout flow. Test upsell pop-ups, fewer steps, and trust signals like badges and reviews to improve average order value (AOV) and checkout completions.
How to run an A/B test
- Identify a goal and form a hypothesis
- Create your test variations
- Run the test
- Analyze the results
- Implement and iterate
The variables you plug into your A/B test are practically limitless, but the process for conducting one is relatively straightforward. These five steps take you through the testing process, from hypothesizing what will work to implementing the winning versions.
1. Identify a goal and form a hypothesis
Before you test anything, figure out what you want to improve. Identify the primary success metrics (bounce rate, checkout completions, ad click-throughs, etc.) you’ll be tracking. Use results from previous tests (if applicable), as well as existing data from tools like Google Analytics, to guide you.
Turn your goal into a hypothesis, or a measurable statement of what you think will happen if you change something. For example, “I believe adding a limited-time 20%-off banner will result in higher add-to-cart rates.” The key to developing a good hypothesis is that it should be measurable so that you can prove it after testing.
2. Create your test variations
Next, build two versions to compare. Limiting two versions of the same page in a given test helps isolate results. Otherwise, you’re veering into multivariate testing, which compares multiple versions but makes it harder to know what actually caused the results. In the example above, you would create:
-
Variation A: Control (no change)
-
Variation B: 20% off banner
Label each variation clearly in your content management system (CMS) when you set up the banners so that you can easily compare the results.
3. Run the test
Use an A/B testing tool like Intelligems, Shoplift, or Optimizely. Split traffic down the middle with 50% going to Variation A and 50% going to Variation B, and let the test run for two weeks. The key is to gather sufficient traffic to each variation to achieve meaningful results. As a general rule, each variation should have at least 1,000 visitors to achieve meaningful data.
4. Analyze the results
Once your current test has run, your A/B testing tool will tell you whether your result is statistically significant. Compare how each variation performed against your original goal and whether your hypothesis was proven correct.
Sometimes, your test results may not be statistically significant, but balancing them with direct customer feedback can still give you a solid direction. Stephanie Chen, founder of Anyday kitchen essentials, tells Shopify Masters the brand doesn’t A/B test everything, because it would take a long time. Plus, it may not always have the traffic to detect a statistical difference between A and B in some cases.
To find the middle ground and work efficiently, Anyday balances test results with qualitative feedback and firsthand experience. “We also are consumers and have a gut [feeling] of what will resonate based on our qualitative and quantitative customer surveys,” Stephanie says.
5. Implement and iterate
If your new variation passes the test with statistically significant results, you can confidently proceed with permanent implementation for all website visitors. If not, refine your hypothesis, implement a bigger change, or run the test for a longer period of time. A good A/B testing program is an ongoing process of learning and improving.
A/B testing FAQ
What is an example of A/B testing?
A simple example of A/B testing is testing two different headlines on a web page, such as a seasonal headline versus an evergreen headline, to see which one leads to more purchases.
What does A/B mean in a test?
In a test, A/B refers to the two versions being compared. Version A is the original, also called the control. Version B is the variation with one change, also called the variant. The goal is to see which version performs better based on a specific metric, such as conversion rate.
What are the types of A/B testing?
Three common types of A/B testing include:
- A/B testing (or split testing). One version versus another version.
- Multivariate testing. Testing multiple elements at once to find the best combination.
- Split URL testing. Sending 50% of traffic to one page and 50% of traffic to a completely different page URL. Often, these pages have different designs or messaging.
