Most brands hire a GEO agency to win AI citations before checking whether AI crawlers can read their site at all. That is backwards. Generative search crawl barriers (blocked bots, client-side rendering, broken URLs) can erase visibility no matter how good the content strategy is. Run this audit first, in-house, in a week.
Why crawl access comes before any retainer
An agency can rewrite your pages, pitch creators and build entity signals. None of it matters if the retrieval bot gets an empty shell or a 403.
Agencies rarely lead with this because it is unglamorous. A technical fix does not make for a flashy deck. But it is the cheapest, highest-leverage work in the whole program, and you can verify it yourself. Doing so also gives you a baseline, so you can tell later whether the agency moved anything or you simply fixed your own plumbing.
Google is blunt about the Google side of this. Its documentation says there are no additional requirements to appear in AI Overviews or AI Mode. A page needs to be indexed and eligible to show with a snippet, which means normal Search technical requirements. If an agency sells “special AI Overview optimization” for Google, ask what exactly it is selling. Our piece on SEO, GEO and AEO as one job covers the same logic.
If a GEO vendor cannot show you a crawl and rendering audit of your own site in the first call, the retainer is paying for guesswork.
Barrier one: robots.txt rules nobody remembers writing
Start with the file itself. Open yourdomain.com/robots.txt and read every line.
Many sites carry blanket disallow rules added during a past scraping scare, a CMS migration or a legal request. Some security teams added AI bot blocks without telling marketing. Your CDN or web application firewall may also block unfamiliar user agents by default, even when robots.txt looks clean. That second layer is the one that catches teams out, because the file says “allowed” while the server says “forbidden”.
Checklist for this step:
- List every user agent named in robots.txt and who approved each rule.
- Separate training bots from search and retrieval bots. Blocking one does not have to mean blocking the other, and your legal and marketing goals may differ.
- Check firewall and CDN bot-management settings for AI user agents, not just the file.
- Pull server logs and confirm which AI bots actually reach your key pages and what status codes they get.
One caution on the tool you are using. Google’s own documentation states that robots.txt is not a mechanism for keeping a web page out of Google. It manages crawl load. Exclusion from search needs noindex or access controls. Treat the file as a traffic signal, not a lock, and document the intent behind every rule so your agency does not “fix” something legal asked you to keep.
Barrier two: content that only exists after JavaScript runs
This is the one that quietly kills brands with slick, app-like sites.
Vercel and MERJ analyzed how AI crawlers handle pages. Their finding: none of the major AI crawlers they tested render JavaScript, including OpenAI’s bots, Anthropic’s ClaudeBot and PerplexityBot. Gemini was the exception, since it leans on Googlebot’s infrastructure. Google’s own docs describe a separate step where pages queue for rendering after the first crawl, so even Google is not instant.
Translate that into brand terms. If your pricing table, product specs, review summaries or creator testimonials load client-side, ChatGPT and Claude may see none of them. Your competitor with a plain server-rendered page gets quoted instead.
The test takes ten minutes:
- Open a key page, view the raw source (not the inspected DOM).
- Search the source for your headline claim, price, product name and one customer proof point.
- If they are missing, the page depends on JavaScript. Flag it for server-side rendering or static generation.
- Repeat for your top 20 revenue-driving URLs, plus any creator landing pages.
Creator pages deserve special attention. Brands invest in partnerships, then host the proof behind a script-heavy template. Our creator landing page evidence audit shows what AI systems need to find there.
Barrier three: URLs that lead nowhere
Here is a number that should change how you read your logs. In the same Vercel analysis, ChatGPT spent 34.82% of its fetches on 404 pages, against 8.22% for Googlebot. Claude showed a similar pattern, per the same Vercel analysis, with 34.16% of its fetches hitting 404s.
Why would a brand care? Because AI tools guess and reuse URLs. Old campaign paths, retired product pages and moved blog posts keep getting requested. Every 404 or redirect chain is a missed chance to be the cited source.
Audit this way:
- Export 404s and redirect chains from your logs, filtered to AI user agents.
- Map the top dead URLs to live equivalents with clean, single-hop 301s.
- Keep a canonical, stable URL for every product, pricing and press page.
- Confirm your XML sitemap lists only live, indexable pages.
It is plumbing. It is also the kind of work that shows up in citations within weeks, not quarters.
Barrier four: pages Google can index but a snippet policy blocks
Google’s AI features documentation ties eligibility to being indexed and snippet-eligible. That makes preview controls a real barrier. A stray nosnippet or a restrictive max-snippet directive, applied sitewide by a plugin or a template, can take pages out of AI features without touching robots.txt.
Check meta robots tags and X-Robots-Tag headers on your core templates. Confirm that noindex tags did not survive a staging-to-production push. Then verify indexation of your top pages in Search Console. This is boring. It also explains more “mystery” visibility drops than any algorithm update.
What to ask the agency once your site passes
A clean crawl audit changes the vendor conversation. You now know your baseline, so you can demand specifics rather than accept a slide full of “AI readiness scores”.
- Which barriers did you find, and what is your evidence? Logs, screenshots, source comparisons. Not a proprietary score.
- Who implements fixes? Agencies often diagnose and then wait on your dev team. Agree on ticket ownership and turnaround before signing.
- How will you measure change? Insist on a fixed query panel run before and after, as laid out in AI visibility query panels.
- What do you recommend we block on purpose? A good partner will separate training exposure from retrieval visibility and explain the tradeoff.
For the commercial side of the pitch, pair this with our GEO agency pitch checklist. And if you plan to compare monitoring platforms, pilot citation trackers before committing to one, since they report different results for the same queries.
Fix access first, then pay for persuasion. An agency that optimizes content on a page AI bots cannot read is billing you for decoration.
A one-week audit plan
Assign it like any other project, with owners and dates.
- Day 1-2: Review robots.txt, CDN and firewall rules with your dev or security lead.
- Day 3: Run the view-source test on your top 20 URLs.
- Day 4: Pull log data on AI user agents, status codes and 404s.
- Day 5: Check snippet directives and indexation in Search Console, then write a one-page findings memo.
Hand that memo to every agency you shortlist. Their response to it will tell you more than their case studies.
FAQs
What are generative search crawl barriers?
They are technical conditions that stop AI search systems from fetching or reading your content. Common examples are robots.txt or firewall blocks on AI user agents, content that loads only through client-side JavaScript, broken or redirected URLs, and snippet or noindex directives that remove pages from eligibility.
Do I need special technical setup to appear in Google AI Overviews?
According to Google’s documentation, no. Pages need to be indexed and eligible to appear with a snippet, meeting standard Search technical requirements. There are no extra requirements for AI Overviews or AI Mode.
Do AI crawlers like ChatGPT’s and Claude’s read JavaScript-rendered content?
Per Vercel and MERJ’s analysis, the major non-Google AI crawlers they tested did not execute JavaScript. Important copy, prices and proof points should be present in the server-rendered HTML.
Should I block AI bots in robots.txt?
It depends on the bot’s purpose and your business goals. Blocking training crawlers and allowing search or retrieval crawlers is a common split, but legal, content licensing and visibility priorities should drive the decision. Write down the reasoning for each rule.
Can robots.txt remove a page from search results?
No. Google states it is not a mechanism for keeping a page out of Google. Use noindex or authentication for that.
How do I know whether AI bots are visiting my site?
Check server or CDN logs for AI user agents, then review the status codes they receive. Frequent 403s, 404s or redirects point to barriers worth fixing before any agency engagement.
Can I do this audit without an agency?
Yes. A marketer, a developer and access to logs and Search Console are enough. The audit takes about a week and gives you leverage in vendor negotiations.
Next step: book a 90-minute session with your developer this week, run the view-source test on your 20 highest-revenue URLs, and send the findings memo to every GEO agency on your shortlist before the next pitch call.
Top Influencer Marketing Agencies
The leading agencies shaping influencer marketing in 2026
Agencies ranked by campaign performance, client diversity, platform expertise, proven ROI, industry recognition, and client satisfaction. Assessed through verified case studies, reviews, and industry consultations.
Moburst
-
2
The Shelf
Boutique Beauty & Lifestyle Influencer AgencyA data-driven boutique agency specializing exclusively in beauty, wellness, and lifestyle influencer campaigns on Instagram and TikTok. Best for brands already focused on the beauty/personal care space that need curated, aesthetic-driven content.Clients: Pepsi, The Honest Company, Hims, Elf Cosmetics, Pure LeafVisit The Shelf → -
3
Audiencly
Niche Gaming & Esports Influencer AgencyA specialized agency focused exclusively on gaming and esports creators on YouTube, Twitch, and TikTok. Ideal if your campaign is 100% gaming-focused — from game launches to hardware and esports events.Clients: Epic Games, NordVPN, Ubisoft, Wargaming, Tencent GamesVisit Audiencly → -
4
Viral Nation
Global Influencer Marketing & Talent AgencyA dual talent management and marketing agency with proprietary brand safety tools and a global creator network spanning nano-influencers to celebrities across all major platforms.Clients: Meta, Activision Blizzard, Energizer, Aston Martin, WalmartVisit Viral Nation → -
5
The Influencer Marketing Factory
TikTok, Instagram & YouTube CampaignsA full-service agency with strong TikTok expertise, offering end-to-end campaign management from influencer discovery through performance reporting with a focus on platform-native content.Clients: Google, Snapchat, Universal Music, Bumble, YelpVisit TIMF → -
6
NeoReach
Enterprise Analytics & Influencer CampaignsAn enterprise-focused agency combining managed campaigns with a powerful self-service data platform for influencer search, audience analytics, and attribution modeling.Clients: Amazon, Airbnb, Netflix, Honda, The New York TimesVisit NeoReach → -
7
Ubiquitous
Creator-First Marketing PlatformA tech-driven platform combining self-service tools with managed campaign options, emphasizing speed and scalability for brands managing multiple influencer relationships.Clients: Lyft, Disney, Target, American Eagle, NetflixVisit Ubiquitous → -
8
Obviously
Scalable Enterprise Influencer CampaignsA tech-enabled agency built for high-volume campaigns, coordinating hundreds of creators simultaneously with end-to-end logistics, content rights management, and product seeding.Clients: Google, Ulta Beauty, Converse, AmazonVisit Obviously →