A/B Testing with Low Traffic: Why Your Test Is Lying

On a site with a few hundred visitors a day, A/B testing almost always lies. To reliably detect a 20% lift in conversion rate, each variant needs about 400 conversions; at 200 visitors a day and a 3% conversion rate, that takes four and a half months. A test stopped early, or one you check every day, declares a winner where there is none in one case out of four or more. Below: test durations for typical small-business traffic, a simulation of how false winners appear, and what to do instead of a classic test.

Every figure marked "my calculation" comes from code: Python 3.13, numpy 2.5, scipy 1.18, two-sided two-proportion z-test, 5% significance, 80% power, 50/50 split. Simulations use 20,000 runs per scenario.

How much traffic does an A/B test need?

Sample size depends on three things: your current conversion rate, the effect you want to catch, and how much error you accept. The effect is set in advance as the MDE (minimum detectable effect) – the smallest lift the test should reliably detect. Industry defaults are 5% false positives and 80% power, the chance of finding an effect that really exists.

For conversion rates there is a rule of thumb from van Belle, quoted by both Evan Miller and Ron Kohavi and co-authors: each group needs roughly 16 · p · (1 − p) / δ² visitors, where p is the baseline conversion rate and δ the absolute difference to detect. Halve the effect and you need four times the traffic.

Here is what that means for a site converting at 3%.

My calculation: 3% baseline, two-sided test, α = 5%, 80% power.

Count conversions, not visitors

The table hides a handy pattern. For any baseline between 1% and 5%, each group needs roughly the same number of conversions:

• about 1,600 per group to detect a 10% lift;
• about 410 for 20%;
• about 190 for 30%;
• about 75 for 50%.

Lower conversion rates need more visitors to collect them, but conversions are the real bottleneck. A site with 30 leads a month needs over two years to test for a 20% lift (410 leads in each of two groups, 820 total). That answers "should I A/B test at all?" faster than any calculator.

Kohavi, Deng and Vermeer (KDD 2022) put it more bluntly: A/B tests are useful for reasonably sized effects when you have at least thousands of active users, preferably tens of thousands. A site with 200 visitors a day gets 6,000 visits a month – and those are not unique users.

Why low-traffic tests "find" winners

A small business owner runs a test, sees "+35%, significant" after two weeks and ships the winner. A month later, leads are flat. Two mechanisms cause this, and both work even when the test was set up correctly.

Peeking: a false winner in one test out of four

A 5% significance level means that if the variants are identical, the test is wrong 5% of the time – provided you check once, on the planned day. Evan Miller showed back in 2010 that almost nobody does: the dashboard updates daily, and the test gets stopped the moment it turns green. Every look is another lottery ticket.

I simulated what this looks like on small-business traffic. An A/A test: both variants identical, 3% conversion, 28 days, results checked every evening, stopped at the first "significant".

At 50 visitors a day, daily peeking gives 24% false winners; at 1,000 a day, 28%. Traffic barely matters – the number of looks does. The longer a test runs with daily checks, the higher the chance a random swing crosses the threshold.

This is in line with Miller's estimate: in his worst case, checking after every visitor, the false positive rate is 26.1% instead of 5%. Ramesh Johari, Leo Pekelis and David Walsh (Stanford and Optimizely, 2015) state plainly that standard p-values and confidence intervals become unreliable when decisions are made while continuously monitoring a test.

Bayesian testing does not fix peeking either

A popular tip is to switch to Bayesian A/B testing because it supposedly lets you look at results as often as you like. That claim has been refuted repeatedly.

• David Robinson (Stack Exchange, 2015) simulated the Bayesian stopping rule proposed by Miller and Chris Stucchio: without peeking the worse variant won 2.5% of the time, with daily peeking 11.8%.
• Markus Loecher (Berlin School of Economics and Law, 2018) showed that both posterior probability and "value remaining" are severely affected by repeated testing.
• Alex Molas (2025) ran simulations for 0.1%, 1% and 10% conversion rates at different peeking intervals and reached the same conclusion: a fixed posterior threshold does not control false winners when you stop on the first success.

My simulation shows the same on small traffic. The rule "probability that B beats A is above 95% (or below 5%)" gives about 10% false winners with a single check and 42% with daily checks. The posterior honestly answers "how likely is B better given today's data", but a stopping rule built on it still errs.

The winner's curse: even real effects come out inflated

The second trap is subtler. Suppose the new variant really is 10% better and nobody touches the test until the end. At 200 visitors a day and 3% conversion, a 28-day test detects that effect only one time in ten. Power is about 10% instead of 80%.

Worse, in the rare cases where the test does show "significant", the average reported lift is about +41% against a true +10%. A small effect on a small sample becomes significant only when noise happens to inflate it. Statisticians call this the winner's curse; Andrew Gelman and John Carlin (2014) named it a Type M (magnitude) error.

My calculation: 3% baseline, single check at the end.

In practice, the owner budgets for +40% more leads and gets +10% or nothing.

Famous results that did not hold up

Kohavi, Deng and Vermeer examined a test published in 2021 where a new landing page "increased conversions 337%". Control had 82 visitors and 3 conversions, treatment 75 visitors and 12 conversions, p-value around 0.01. They estimated the pre-experiment power to detect a 10% change at about 3%, so the result cannot be trusted, and invoked Twyman's law: any figure that looks interesting or different is usually wrong.

The same paper has a sobering table. If roughly one idea in ten succeeds (the rate the authors cite for Booking.com, Google Ads and Netflix) and tests run at 20% power, more than half of "significant" results are false: 52.9%. At 80% power, 22%.

A recent example: a Journal of Consumer Research study (2023) claimed rounding button corners lifted click-through by 55% (p = 0.037). In late 2025 Kohavi, Gelman, Imbens and co-authors published three replications, each with more than two thousand times as many users as the original. The effect was about two orders of magnitude smaller and not statistically different from zero.

What to do when traffic is low

Dropping classic A/B tests does not mean guessing. Several approaches are more honest on small traffic.

Test only big changes

The table suggests a rule: at 200 visitors a day, test only things you expect to move conversion by at least 30–50%. Button colors and headline wording do not. A new offer, a different price, a different lead form, a whole new first screen can. If an idea cannot plausibly deliver +30%, ship it without a test and watch overall numbers.

Pick a metric closer to the action

Clicks on "Request a quote" happen more often than submitted requests, and requests more often than payments. More events means shorter tests. The catch is that a click is a proxy: a variant can win clicks without winning sales. Use proxies to weed out clearly bad ideas, not for final money decisions. I covered which metrics actually tie to revenue in website metrics for owners.

Fix the end date, or use a sequential test

If you do run a test, the simplest cure for peeking is to compute the sample size up front, write down the end date and make no decisions before it. Looking is fine; stopping is not.

If waiting is unbearable, use a sequential test, where early stopping is built into the math. Evan Miller described a version in 2015 simple enough to run on paper:

1. Choose the maximum number of conversions N from Miller's table or calculator.
2. Count conversions in treatment (T) and control (C).
3. If T − C reaches about 2√N, treatment wins (two-sided: 2.25√N).
4. If T + C reaches N without crossing the threshold, there is no winner.

For a one-sided test of a 20% lift at α = 5% and 80% power, Miller gives N = 808 conversions and a threshold of 56. There is no free lunch: when there is no effect, the test runs longer than a fixed one, but a real blockbuster shows up noticeably sooner. Commercial platforms use the same idea (sequential testing, always-valid p-values) – Johari and co-authors' method was deployed at Optimizely.

Watch people instead of counting percentages

Qualitative research gives no percentages, but it shows why people do not convert. Jakob Nielsen and Tom Landauer's model says five usability-test participants uncover about 85% of interface problems; Nielsen recommends several small rounds of five rather than one big study. It is a 2000 finding about discovering problems, not measuring effects, but for a small business five sessions watching customers try to order often teach more than six months of A/B testing. Session recordings (Hotjar, Microsoft Clarity, Yandex Webvisor) do a similar job at scale.

Compare periods, with caveats

"A month before vs a month after" is not an experiment: seasonality, ad spend and competitor prices change in between. To make it more useful:

• change one thing at a time and log the date;
• compare the same weekdays and, where possible, year over year;
• look at channels separately: if only paid traffic grew, bids may explain it, not the page;
• track leads to sales, not just leads, and judge over several weeks rather than the first few days.

This evidence is weaker than an A/B test, and it is worth saying so. But it beats a test that is almost guaranteed to crown a random winner.

Key takeaways

• Before you A/B test, count conversions: about 400 per variant to detect a 20% lift, about 1,600 for 10%.
• At 200 visitors a day and 3% conversion, a +20% test takes four and a half months and a +10% test a year and a half. Small tweaks cannot be tested on that traffic.
• Daily checking and stopping "once it turns green" turns a 5% error rate into 25–35%. A Bayesian probability on the dashboard does not protect you.
• If a small test shows a huge lift, the real effect is most likely much smaller or zero.
• On low traffic, rely on big changes, a fixed end date or a sequential test, watching real users and careful period comparisons. More on how I approach this for businesses on the business page.