Shopify Rollouts: Native A/B Testing on Shopify

https://kubixmedia.co.uk/blog/shopify-rollouts-native-ab-testing-on-shopify

Published

05.01.2026

Category

CRO, Shopify

Overview

Shopify Rollouts is a built-in system for preparing, scheduling, and testing changes to an online store before they go live permanently. Merchants can prepare seasonal updates, schedule launches at a specific time, and draft customisations directly on the published theme without spinning up duplicate themes. Launched as part of the Winter 2026 Editions update, it also introduces native split testing for stores on Advanced plans and above. This post covers how Rollouts works in practice, how to configure traffic splits, what can and cannot be tested, how to interpret the analytics, and how the related SimGym tool fits into the testing workflow.

Key Takeaways

Rollouts works from the published theme rather than a separate environment. Changes made only apply to the rollout experience, so shoppers outside the test continue seeing the live version until the scheduled launch or test begins.

Split testing and analytics require Shopify Advanced or higher. Basic plan merchants can use Rollouts for scheduling and controlled publishing but cannot run traffic-split experiments or access performance data.

Starting with a smaller traffic percentage is the safest approach, as it allows teams to spot issues early and build confidence before expanding the test. Shopify provides estimated visitor volumes to help judge whether the audience size is large enough to produce useful insight.

Rollouts can be used for market-level testing. Brands using Shopify Markets can test changes in one specific region without affecting the experience in others, allowing optimisation for regional preferences with less risk.

SimGym, released alongside Rollouts as part of Winter Editions 26, lets merchants simulate shopper behaviour using AI agents trained on data from billions of purchases. It is designed for pre-launch insight, while Rollouts handles live testing with real visitors.

FAQ

What is the difference between Shopify Rollouts and SimGym?

Rollouts tests changes with real store visitors in a live environment. SimGym uses AI-generated shopper simulations to predict how a change might perform before it goes live, making it a pre-launch planning tool rather than a live testing one.

Which Shopify plan do you need to run A/B tests with Rollouts?

Split testing and access to analytics require Shopify Advanced or higher. Merchants on Basic and above can use Rollouts for scheduling and managing theme changes, but the traffic-split experimentation feature is limited to Advanced plans and above.

How long should a Shopify Rollouts experiment run?

A store with 500 visits a day running a 50/50 split would generate roughly 3,500 visitors per variant over 14 days. Letting the experiment run long enough to capture normal buying patterns across the full week gives a much clearer and more dependable read than reacting to early signals.

What can Shopify Rollouts currently not test?

Rollouts does not currently cover product pricing, discounts, checkout journeys, post-purchase experiences, app embeds, or deeper Liquid template logic. Audience-level targeting is also limited, so tests cannot be narrowed to specific visitor types such as new customers, returning shoppers, or mobile users only.

Can you run more than one Rollout at the same time?

Yes, multiple Rollouts can run simultaneously. However, if they affect similar parts of the storefront at once, it becomes much harder to determine which specific change influenced the result, so overlapping tests need to be managed carefully.

Explore similar resources

Amazon PPC: Everything You Need To Know

Amazon PPC is pay-per-click advertising on the Amazon platform that helps sellers promote products by placing them in front of interested shoppers. Each time a potential buyer clicks on an ad, the seller pays Amazon a cost per click, which is a key factor in determining the overall cost-effectiveness of a campaign. This guide covers all three ad types, Sponsored Products, Sponsored Brands, and Sponsored Display, alongside the core metrics needed to evaluate performance, seven practical optimization techniques, and a glossary of 20 essential Amazon PPC terms. It is written for both new sellers getting started and experienced advertisers looking to improve campaign efficiency.

Thrive

9 min