All notes

Measurement

A creative test is only fair if the loser had the same chance

Portrait of Leila Haddad

Leila Haddad

Head of paid social

·

6 min read

A hand reaching through a torn hole in a yellow paper backdrop holding a plain turquoise paper shopping bag with no logo

Most creative tests are decided before they begin. Two ads go into a campaign, the platform shows one of them to nine people in ten, and a week later somebody announces a winner. The other ad never had a chance to lose properly.

The problem with letting the platform choose

Ad platforms are built to find a favourite quickly. Within hours they pour budget into whichever ad gets the earliest clicks, and early clicks are mostly luck. In an audit of 60 tests we ran this way during 2025, the ad that led after 48 hours was the eventual best performer only 55% of the time. A coin does nearly as well.

That is fine if all you want is for the budget to be spent. It is useless if you want to learn something you can use next month.

What we do instead

Every new ad gets its own small budget, fixed for seven days, aimed at the same broad audience as every other ad in the test. No ad can borrow money from another. Seven days matters because people behave differently on a Tuesday and a Sunday, and a test that misses a weekend has only measured weekdays.

We decide the pass mark before launch and write it down. Usually it is a cost per order: at or below the account’s average, the ad graduates into the main campaign. Above it, the ad is switched off. Writing it first removes the temptation to find a kinder metric for an ad the team happens to love.

Decide what losing looks like before you launch. Afterwards, everyone is a lawyer.

How much is enough

The most common mistake is calling a result on too little. Our rule of thumb is that an ad needs to have spent at least three times the target cost per order, and the test as a whole needs at least 30 orders, before anyone is allowed an opinion. For a brand with a $25 target that means about $75 an ad at the very least, and more is better. Below that, the difference between two ads is noise wearing a suit.

This is also why we tell brands spending under $5,000 a month that they cannot test twelve ads fairly. Four, tested properly, will teach them more.

V

Volta Cycles

Feed

Skip the queue. All of them.

Book a test ride

1,840

test rides booked · Volta Cycles, Mar to May 2026

The Volta Cycles ad shown here lost its first two days badly. Under the old method it would have been starved by Wednesday. With a protected budget it ran its full week, finished with the lowest cost per booked test ride of the eight ads in that round, and went on to bring in a large share of the 1,840 rides booked between March and May 2026.

Reading the result

At the end of the week we ask three questions in order. Did it pass the mark? If it did, do we know why, in a sentence that could guide the next ad? And is the reason about the idea or only about the execution? A winning ad with no explanation is a lucky ticket. A winning ad with a reason is a method.

We also keep the losers. Every switched-off ad goes into a shared library with its numbers and one line on what we think went wrong. After two years that library is the most useful thing a new strategist can read, because it is a record of what a specific audience refused to care about.

None of this is sophisticated. It is closer to a school science lesson than to machine learning: change one thing, keep the rest the same, decide the pass mark first, and wait long enough. The platforms will always be better than us at spending a budget. They have no interest in telling you why it worked.

Ready to be heard?

Tell us what you sell and what you spend. You get a plan and a price within three working days.

Ready to be heard?

Tell us what you sell and what you spend. You get a plan and a price within three working days.

Create a free website with Framer, the website builder loved by startups, designers and agencies.