Every vendor demo shows a lift number. Almost none show the control group. If you are weighing Capillary AI decisioning or any next best action (NBA) engine, the question is not whether it can pick an offer. It is whether customers would have bought anyway. Test that before rollout, not after the budget is spent.
Why NBA claims deserve suspicion
A next best action engine decides, customer by customer, which message, reward or channel to use next. Capillary Technologies sells this kind of decisioning inside its loyalty and customer engagement stack. The pitch is familiar: more relevance, higher redemption, better retention.
The trouble is attribution. Loyalty members are your most engaged customers. Many of them redeem a coupon or open a push notification because they were already about to purchase. An engine that targets likely buyers will post excellent conversion rates and prove nothing.
BCG describes this risk directly. The measurement has to separate actions that change behavior from actions that cannibalize intent the customer already had. That distinction is the whole game.
A conversion rate tells you what happened after the message. Only a control group tells you what the message caused.
The regulatory angle most teams skip
This is not only a finance problem. If your agency or brand repeats a vendor’s performance claim in a pitch deck, a case study or a creator brief, you own that claim.
The FTC has been clear on this point. When it launched Operation AI Comply, it announced five law enforcement actions against operations using AI hype or selling AI tools that could enable deception. The agency’s message was that there is no AI exemption from existing law. Your NBA lift claim may be far less dramatic than those cases, but the standard is the same: you need substantiation before you say it.
Consumer trust points the same way. A Gartner survey of 1,539 U.S. consumers found that 68% frequently wonder whether content they see is real. Unsubstantiated AI claims land in a skeptical room.
Two questions, two tests
Most pilots fail because they ask one fuzzy question: “Does it work?” Split it in two.
Does the whole system add value?
BCG calls this “Big M” measurement. The test is a global holdout: a randomly selected group of customers who receive no engine-composed actions. They may still get your standard calendar campaigns, but nothing from the decisioning layer.
Compare the holdout to the exposed group on revenue per customer, not clicks. If the gap is small, the engine is mostly reshuffling credit. Our guide to NBA holdout proof walks through sizing and duration.
Does this specific action beat the alternative?
BCG’s “Little m” asks a narrower question: does this tactic create incremental value compared with the next best alternative? You answer it with local holdouts inside the system. Customers in the test audience are randomly split between different actions or a default experience.
Why does this matter for a platform like Capillary? Because decisioning engines typically rank dozens of possible actions. A deep discount might win on redemption and lose on margin against a free-shipping nudge or no message at all. Only tactic-level tests show that.
A pre-rollout test plan you can run in 8 to 10 weeks
Here is the sequence I would hand to a CRM lead.
- Freeze the claim. Get the vendor to state the expected lift, the metric, the audience and the time window in writing. Vague promises cannot be tested.
- Audit the data feeding the model. Bad identity resolution or stale purchase history poisons everything downstream. Use a data layer audit before launch.
- Randomize before targeting. Assign the holdout at the customer level before the engine scores anyone. If the engine picks who gets excluded, you have built bias into your control.
- Set the holdout size by the smallest effect that matters. A large base can often detect a modest lift with a small holdout, while a small program needs a larger one, so have your analyst run a power calculation first.
- Run local tests on the top three actions. Include a “do nothing” arm and a simple rules-based arm that your team could build without AI.
- Measure net contribution. Subtract discount cost, incentive cost and any media spend. Gross revenue flatters every engine.
- Review at a fixed date. Decide the pass/fail threshold in week one, not week eight.
The rules-based arm deserves emphasis. If a hand-built segment performs within a few points of the AI, you have learned something expensive and useful.
What about creators and loyalty together?
This is where Influencers Time readers should pay attention. Loyalty and decisioning data increasingly feeds creator programs: which members get a creator-led offer, which creator content gets pushed to which segment, which code is attached to which audience.
That creates a double counting trap. A member sees a creator video, then receives an engine-timed push, then redeems with a creator code. Who gets credit? Without a shared holdout, both the creator and the engine claim the sale.
Two practical fixes help. First, keep a customer-level holdout that is excluded from both engine actions and creator-tied offers, so you can read each effect separately. Second, use geo and holdout tests for the creator layer, and keep the logs described in this piece on attribution audit trails. If a creator’s payout depends on attributed sales, you will want those records when a dispute arrives.
Consent matters here too. Pulling creator-exposure data into a decisioning layer requires clear permissions. See consent tags for creator data access before you connect the pipes.
Questions to put to Capillary (or any vendor)
- Can the platform support a persistent global holdout that no automated journey can override?
- Can you export raw, customer-level assignment logs, including which arm each person was in?
- Does the lift in your case studies come from randomized control groups or from before-and-after comparisons?
- Who defines the holdout, your team or the engine? Agent-designed experiments can be useful, but BCG notes that at the earlier maturity level a specialist defines the holdouts and test structure. Keep a human in that seat until you trust the process.
- What happens to customers in the holdout when the model updates mid-test?
If a vendor cannot answer these cleanly, you have your answer. For a broader checklist, see vendor AI claims evidence.
If a vendor will not let you hold out a randomized slice of customers, ask what the number would look like if you did.
Notice what is not here: I have not quoted a benchmark lift for Capillary’s decisioning. I could not verify a neutral, independent figure, and you should be wary of anyone who offers one. Your own holdout is the only number that counts.
FAQs
What is Capillary AI decisioning?
It is the AI layer in Capillary Technologies’ customer engagement and loyalty platform that selects the next action for each customer, such as an offer, message or channel. Treat it like any NBA engine: its value depends on incremental results measured against a control group, not on how relevant its recommendations look.
What is the difference between Big M and Little m testing?
BCG uses Big M for program-level measurement, which asks whether the whole NBA system beats no intervention, using a global holdout. Little m is tactic-level measurement, which asks whether one action beats the next best alternative, using local holdouts and randomized subgroups.
How big should the holdout group be?
It depends on your base size and the smallest lift worth detecting. Large programs can often detect meaningful effects with a small holdout, while smaller programs need a bigger one. Have your analyst run a power calculation before launch, and fix the number before you see any results.
Why not just trust the vendor’s dashboard?
Dashboards usually report attributed conversions, which include customers who would have purchased anyway. They rarely show randomized control comparisons. Ask for raw assignment data and run your own comparison.
Can a brand face legal risk for repeating AI performance claims?
Yes, potentially. The FTC has stated through Operation AI Comply that existing consumer protection law applies to AI claims. If you publish or pitch a lift figure, keep the test design and data that substantiate it, and have legal review the wording.
How does this connect to influencer programs?
Loyalty engines and creator campaigns often target the same customers, so credit gets double counted. A shared holdout that receives neither engine actions nor creator-tied offers lets you separate the two effects and pay creators fairly.
How long should a pilot run?
Long enough to cover at least one full purchase cycle for your category, plus time to measure repeat behavior. For many retail and consumer brands that means several weeks, not days. Set the end date and pass/fail threshold in advance.
Next step: before your next vendor call, write down the one lift number you expect, then require a randomized holdout in the pilot contract so that number can be proven or killed within a quarter.
Top Influencer Marketing Agencies
The leading agencies shaping influencer marketing in 2026
Agencies ranked by campaign performance, client diversity, platform expertise, proven ROI, industry recognition, and client satisfaction. Assessed through verified case studies, reviews, and industry consultations.
Moburst
-
2

The Shelf
Boutique Beauty & Lifestyle Influencer AgencyA data-driven boutique agency specializing exclusively in beauty, wellness, and lifestyle influencer campaigns on Instagram and TikTok. Best for brands already focused on the beauty/personal care space that need curated, aesthetic-driven content.Clients: Pepsi, The Honest Company, Hims, Elf Cosmetics, Pure LeafVisit The Shelf → -
3

Audiencly
Niche Gaming & Esports Influencer AgencyA specialized agency focused exclusively on gaming and esports creators on YouTube, Twitch, and TikTok. Ideal if your campaign is 100% gaming-focused — from game launches to hardware and esports events.Clients: Epic Games, NordVPN, Ubisoft, Wargaming, Tencent GamesVisit Audiencly → -
4

Viral Nation
Global Influencer Marketing & Talent AgencyA dual talent management and marketing agency with proprietary brand safety tools and a global creator network spanning nano-influencers to celebrities across all major platforms.Clients: Meta, Activision Blizzard, Energizer, Aston Martin, WalmartVisit Viral Nation → -
5

The Influencer Marketing Factory
TikTok, Instagram & YouTube CampaignsA full-service agency with strong TikTok expertise, offering end-to-end campaign management from influencer discovery through performance reporting with a focus on platform-native content.Clients: Google, Snapchat, Universal Music, Bumble, YelpVisit TIMF → -
6

NeoReach
Enterprise Analytics & Influencer CampaignsAn enterprise-focused agency combining managed campaigns with a powerful self-service data platform for influencer search, audience analytics, and attribution modeling.Clients: Amazon, Airbnb, Netflix, Honda, The New York TimesVisit NeoReach → -
7

Ubiquitous
Creator-First Marketing PlatformA tech-driven platform combining self-service tools with managed campaign options, emphasizing speed and scalability for brands managing multiple influencer relationships.Clients: Lyft, Disney, Target, American Eagle, NetflixVisit Ubiquitous → -
8

Obviously
Scalable Enterprise Influencer CampaignsA tech-enabled agency built for high-volume campaigns, coordinating hundreds of creators simultaneously with end-to-end logistics, content rights management, and product seeding.Clients: Google, Ulta Beauty, Converse, AmazonVisit Obviously →
