"How many creatives should I test at once?" gets answers from three to thirty, and each is delivered with the same confidence. The honest answer is that there is no fixed number. There is a number for your budget and your conversion volume, and you can work it out in about five minutes.
The short version: the more expensive the event you are judging on, the fewer variants you can afford.
The constraint is events, not ideas
Every creative needs enough results to tell it apart from its neighbours. Spread a small budget across too many ads and each one ends up with a handful of conversions, which is not enough to separate a good ad from a lucky one.
The scale of the problem is documented. Lewis and Rao examined 25 digital advertising experiments covering $2.8 million of spend and found that only 3 of the 25 had the power to distinguish a 50% return from zero, and none could reliably separate a 10% return from zero. That was measuring incremental return, which is harder than comparing two ads in one account, because the noise is partly shared. But the lesson carries: small differences on small volume are not signal. The full argument is in kill it or scale it.
Work out your own number
Step 1: events per week
Divide your weekly budget for the test by your typical cost per event. A budget of $1,050 a week at $30 per purchase buys about 35 purchases.
Step 2: events needed per variant
This is the part people skip. A common working floor, used in our decision framework, is 25 to 30 of the campaign's primary event before an ad has earned any verdict at all. That is a floor for saying anything, not enough to detect a modest difference.
Step 3: divide
35 purchases a week divided by 30 per variant is one variant. On that budget you cannot test five purchase-optimised ads in a week and learn anything from the purchase column. That is not a failure of discipline. It is arithmetic.
Apply it. Suppose a landing page converts at 2% and you want to detect a lift of 20%, from 2.0% to 2.4%. That is a difference of 0.4 points, so you need about 19,600 visitors per variant. At a 2% rate that is roughly 400 purchases per variant. To catch a bigger swing of 40% you need about a quarter of that.
Now the same logic for a hook rate near 30%. To detect a difference of 3 points you need about 3,700 impressions per variant, and for 5 points about 1,300. Impressions are plentiful and cheap compared with purchases.
That is the key insight for anyone with a modest budget.
Match the number of variants to the metric
| You are judging on | Volume you get | How many variants |
|---|---|---|
| Purchases or leads | Scarce | Few, two or three, and make them clearly different |
| Clicks or link CTR | Moderate | A handful |
| Hook rate or hold rate | Plentiful | Many, as long as they differ only in the opening |
So test many openings and few concepts. You can screen eight openings for the same ad on hook rate with a small budget, then put the two best through to a purchase test. The research backs the framing: the first moments of an ad matter disproportionately. 1.5 seconds of genuine attention was enough to encode memory in the VCCP and Nelson-Field work across 72 ads. See the 3-second hook, deconstructed and what is a good hook rate on Meta ads.
Change one thing at a time
Ten variants that differ in angle, talent, format, length and offer will teach you which ad won and nothing about why. Vary one dimension and the difference is attributable. Four ads with four different first two seconds beat four unrelated concepts as a source of learning, because everything downstream is held constant.
Delivery is not a random split. In an ordinary ad set the platform allocates spend unevenly towards ads it expects to do well, so ads do not get comparable samples. Meta's A/B test tool randomises, and ordinary ad sets do not. Read within ad set comparisons as directional.
The formula assumes a fixed test. It assumes independent observations and a sample size set in advance. Peeking daily and stopping when you like the result inflates false positives. Decide the volume before you start.
Knack will not rank an ad that has not yet earned a verdict. Creatives below the spend and volume floor are shown as too early to call rather than promoted, so a cheap winner on a few hundred impressions is held back. Verdicts are graded on the metric each campaign's objective implies, which is how a high volume metric like hook rate can carry the early read for upper funnel work.
Book a demo to see this run on your own account.
A sensible default
- Count the primary events you actually get in a week.
- Allow at least 25 to 30 per variant, more if you want to detect modest differences.
- If that allows fewer than three variants, judge on a higher volume metric first.
- Change one dimension per test.
- Set the sample size before you launch, and do not stop early because you like what you see.
Sources
- Lewis, R. A., and Rao, J. M. (2015). The Unfavorable Economics of Measuring the Returns to Advertising. The Quarterly Journal of Economics, 130(4), 1941 to 1973. 25 experiments, $2.8m of spend.
- Lehr, R. (1992). Sixteen S-squared over D-squared: a relation for crude sample size estimates. Statistics in Medicine, 11(8), 1099 to 1102.
- VCCP Media and Dr Karen Nelson-Field / Amplified (May 2025). Hacking the Attention Economy.
- The worked sample sizes use the Lehr rule at 80% power and 5% significance. Treat them as order of magnitude, not a replacement for a proper power calculation.