Your AI agent’s dashboard says it drove thousands of conversions. How many of those customers would have bought anyway? How many of those customers would have bought anyway? Without holdout testing for AI agents, nobody knows, and the vendor reporting the number has every incentive not to ask. A holdout group is the cheapest, oldest, most reliable way to separate real lift from credit-taking.
Why dashboards flatter agents
Agents are very good at finding people who were already going to act. They message the lapsed customer who was about to return. They pitch the creator who was already warm. They retarget the cart that was already full.
The dashboard counts all of it. Attribution models reward the last touch, and an autonomous agent can touch a lot of last moments.
That is not fraud. It is arithmetic. If the agent acts on high-intent audiences, its reported results will look strong whether or not it changed any behavior. The only way to find out is to leave a comparable group untouched and compare.
We have made similar arguments about other vendor claims, from opaque visibility scores to attribution vendors. The pattern repeats: a metric appears, nobody can see the mechanism, and budget follows the chart.
What a holdout actually is
Simple idea. Randomly split an eligible audience. One group gets the agent’s actions. The other, the holdout, gets whatever you did before: business-as-usual messaging, human-run outreach, or nothing. The gap in outcomes is your incremental lift.
Random assignment is the whole trick. It makes the two groups statistically alike, so differences are caused by the agent rather than by who the agent chose to target.
A real example helps. A recent arXiv paper on agentic personalisation analyzed 11 months of randomised controlled trial data. Users were randomly assigned either to the agentic system or to a holdout control group receiving the platform’s standard messaging. The audience was roughly 8.8 million users, split 90-10 between agentic and business-as-usual.
Note that split. Note that split: the holdout was the smaller share, not half the audience. You do not need to starve half your audience to learn something, though smaller holdouts need more volume to detect small effects.
Design the test before you sign the contract
Most holdout failures are design failures made in week one. Agree on the following in writing before the agent touches a single customer.
- The unit of randomization. Individual users, accounts, creators, or geographic regions. Pick the unit that matches where spillover could happen.
- The primary metric. One. Revenue, qualified leads, or repeat purchase. Not a dashboard of twelve.
- The holdout size and duration. Calculate the sample you need for the smallest lift worth paying for.
- The control experience. Say exactly what the holdout group receives.
- The kill criteria. If lift is below a stated threshold at a stated date, the agent is paused. Our piece on kill criteria covers how to set those without hindsight bias.
Who owns the randomization? You do. Never let the vendor assign its own control group and grade its own homework. If the tool cannot support an externally controlled holdout, that is itself a finding.
If the party being measured controls the holdout, you are not running a test. You are reading a testimonial.
Users, accounts or geos?
For agents that message individuals, such as CRM, creator outreach, or support, user-level randomization is usually cleanest. For agents that buy media or shift spend, geography often works better because people see ads across devices.
Google’s Meridian documentation notes that smaller geo units can improve statistical power, but that people travel across regions, which creates contamination bias in the incrementality metric. It also warns that participating campaigns should be isolated from overlapping brand lift, conversion lift, or holdout studies.
That last point catches teams constantly. Run an agent holdout while three other experiments hit the same audience, and you can no longer say what caused what.
The same guidance says to match local and regional media efforts between treatment and control, and to exclude geos that will deviate too much. Do not drop a poor-performing region mid-test to “improve efficiency,” and do not move a control region into treatment. Tempting. Wrong.
Reading the results without fooling yourself
Compute lift as the difference between treatment and holdout on your primary metric, then attach a confidence interval. A point estimate alone is a guess wearing a suit.
Then compute what we call the incrementality factor: incremental conversions divided by the conversions the dashboard claimed. If the agent reported 500 conversions and your test shows 300 were incremental, the factor is 0.6. Apply it to future reporting until the next test.
Watch for three traps:
- Novelty effects. Early lift often fades. Run long enough to see it.
- Secondary metrics gone fishing. If the primary metric is flat but engagement is up, that is a story, not a result.
- Cannibalization. Lift in one channel or product line that comes from another is not growth.
Keep the holdout running
Agents learn and drift, so a one-time test expires. The arXiv study is instructive here. It kept a persistent holdout across both an active human-management phase and a later passive phase in which agents ran autonomously. The authors reported that autonomous agents sustained a roughly 57% lift in notification clicks during the passive period, but also a relative dampening of lift when humans stepped back, about 15% for overall engagement and 26% for days with intent and conversion events per user.
Two lessons. Autonomous agents held up better than static rules would suggest. And human oversight still added measurable value. Without a persistent holdout, you would never see either effect. Keep in mind the study comes from one platform and one reactivation use case, so treat it as a pattern to test for, not a benchmark to import.
A holdout that stops after launch tells you whether the agent worked once. A persistent holdout tells you whether it still works.
Where holdouts fit in influencer and creator programs
Creator workflows are full of agents now: matching, outreach, briefing, payouts. Each one makes claims you can test.
- Creator matching. Hold out a slice of campaigns where humans pick creators the old way, then compare outcomes. We outline the logic in our guide to piloting creator matching.
- Outreach agents. Compare reply and conversion rates against manually sent pitches, and keep a human review gate in place on both arms so quality is not the confound.
- Trust and risk scores. Before a score cuts anyone’s budget, test whether flagged creators actually underperform. See this pilot approach.
The compliance angle you cannot skip
Holdouts also protect you legally. If an agent generates endorsement content or synthetic personas, your disclosure obligations do not shrink. One legal explainer says synthetic endorsers practically require a separate disclosure on top of the standard sponsorship one, so check the current FTC Endorsement Guides and confirm with counsel before you publish agent-generated endorsements. It also says synthetic endorsers practically require a separate disclosure on top of the standard sponsorship one. That is a law firm’s reading, not the FTC’s text, so confirm with counsel.
The practical point: a test that proves lift from a noncompliant tactic proves nothing you can use. Build disclosure checks into the treatment arm from day one.
FAQs
How big should the holdout group be?
Big enough to detect the smallest lift that would justify the agent’s cost. Ten percent is common for large audiences, as in the arXiv trial’s 90-10 split. For smaller audiences, you may need a larger holdout share, a longer test window, or both. Run a power calculation before launch.
Is holdout testing the same as A/B testing?
Closely related. A/B testing compares two active treatments. A holdout compares a treatment against no treatment or the existing baseline. For proving an agent’s incremental value, the holdout answers the more important question: does this do anything at all?
Can I trust the vendor’s built-in lift reports?
Treat them as a hypothesis. Ask who controls randomization, what the control group received, and whether you can audit the raw assignment. If the vendor cannot answer, run your own holdout.
How long should a holdout run?
Long enough to cover at least one full purchase cycle and to outlast novelty effects. For agents that learn over time, keep a small persistent holdout running indefinitely and review it quarterly.
What if holding out customers feels costly?
Compare it with the alternative: scaling spend on lift you never verified. The holdout’s cost is the margin on a small group’s missed conversions, and it is usually far smaller than a year of misattributed budget.
Does a holdout work for brand or creator campaigns with long sales cycles?
Yes, with adjustments. Use geo-based or market-level holdouts, pick leading indicators agreed in advance, and match local media activity across groups to limit confounding.
Pick one agent already running in your stack, carve out a randomized holdout that you control, and set a written pause threshold before the next reporting cycle.
Top Influencer Marketing Agencies
The leading agencies shaping influencer marketing in 2026
Agencies ranked by campaign performance, client diversity, platform expertise, proven ROI, industry recognition, and client satisfaction. Assessed through verified case studies, reviews, and industry consultations.
Moburst
-
2
The Shelf
Boutique Beauty & Lifestyle Influencer AgencyA data-driven boutique agency specializing exclusively in beauty, wellness, and lifestyle influencer campaigns on Instagram and TikTok. Best for brands already focused on the beauty/personal care space that need curated, aesthetic-driven content.Clients: Pepsi, The Honest Company, Hims, Elf Cosmetics, Pure LeafVisit The Shelf → -
3
Audiencly
Niche Gaming & Esports Influencer AgencyA specialized agency focused exclusively on gaming and esports creators on YouTube, Twitch, and TikTok. Ideal if your campaign is 100% gaming-focused — from game launches to hardware and esports events.Clients: Epic Games, NordVPN, Ubisoft, Wargaming, Tencent GamesVisit Audiencly → -
4
Viral Nation
Global Influencer Marketing & Talent AgencyA dual talent management and marketing agency with proprietary brand safety tools and a global creator network spanning nano-influencers to celebrities across all major platforms.Clients: Meta, Activision Blizzard, Energizer, Aston Martin, WalmartVisit Viral Nation → -
5
The Influencer Marketing Factory
TikTok, Instagram & YouTube CampaignsA full-service agency with strong TikTok expertise, offering end-to-end campaign management from influencer discovery through performance reporting with a focus on platform-native content.Clients: Google, Snapchat, Universal Music, Bumble, YelpVisit TIMF → -
6
NeoReach
Enterprise Analytics & Influencer CampaignsAn enterprise-focused agency combining managed campaigns with a powerful self-service data platform for influencer search, audience analytics, and attribution modeling.Clients: Amazon, Airbnb, Netflix, Honda, The New York TimesVisit NeoReach → -
7
Ubiquitous
Creator-First Marketing PlatformA tech-driven platform combining self-service tools with managed campaign options, emphasizing speed and scalability for brands managing multiple influencer relationships.Clients: Lyft, Disney, Target, American Eagle, NetflixVisit Ubiquitous → -
8
Obviously
Scalable Enterprise Influencer CampaignsA tech-enabled agency built for high-volume campaigns, coordinating hundreds of creators simultaneously with end-to-end logistics, content rights management, and product seeding.Clients: Google, Ulta Beauty, Converse, AmazonVisit Obviously →