AI search fails: brands block their own bots
The gist
Major brands are accidentally blocking AI from their own websites, sabotaging their visibility in the next wave of search results.
What to know
- About 30% of websites are actively blocking AI crawlers, often due to technical defaults rather than intent.
- Mixing up robots.txt, sitemaps, and llms.txt files breaks AI discovery—over 68% of llms.txt files are auto-generated and often misused.
- SEMrush found only 36 global brands with consistent top-100 AI search visibility across ChatGPT, Gemini, and Google AI tools in 2026.
Invisible by Default
Brands are unintentionally locking AI out of their own content due to overlooked technical settings and outdated site architectures.
The web’s AI visibility problem begins with basic access failures, not model quality: as a16z speedrun noted, many sites blocked AI crawlers during the training data debates of 2023-24, while SiliconANGLE theCUBE reported that “about 30% of all the websites we've analyzed… are actively blocking… they are actively blocking LLMs from accessing their own content and learning from it.” Those barriers are often created by security or IT defaults rather than marketing intent, which means brands can accidentally prevent the very retrieval bots that shape AI answers from seeing their own authoritative pages.
Even when bots are allowed in, many sites still fail to present readable content: a16z speedrun warned that “AI crawlers don’t execute JavaScript,” and that “analysis shows ~11.5% of ChatGPT’s requests are JS files that go unused,” while Equity said “On any given page you may see a hundred thousand tokens” even though “We find generally these agents aren't going to read more than 10,000. So we get well below that 10,000 token mark.” That is why technical cleanup changes outcomes: Tastewise “reported a 600% visibility spike in one week” after “restructuring for LLM summarization” and “adding FAQ schema markup,” with one article rising from 42 to 294 AI impressions and earning citations in ChatGPT.
File Confusion Blocks Bots
Misconfigured robots.txt, sitemaps, and llms.txt files are breaking AI access, with most llms.txt files either empty, misplaced, or misused.
These files and settings do not do the same job, so mixing them up can break AI access in different ways. Google’s 2015 deprecation of its AJAX crawling scheme marked a shift away from URL fragments as a durable way to expose crawlable pages, showing that rendering still determines whether machines can reliably see content, while schema, FAQs, and summaries help interpretation rather than replacing crawlable delivery.
The same separation applies to machine-readable files: robots.txt governs access, sitemaps enumerate indexable URLs, and llms.txt is at most a curated aid, not a gatekeeper or sitemap substitute. Common Crawl’s analysis of 584,107 LLMs TXT files found 68% originated from a plugin or template, about 22% did not contain any links at all, more than 136,000 robots TXT files were being served at the lms txt path, 32 LMS TXT files appeared to tell Commencrawl's CC bot that it was not allowed to crawl the site, and one site even had an lms txt file saying cc bot; Google Search Central said llms.txt can’t function as a sitemap in its systems. It also said Google and Apple have let you do this manually since 2023 through separate robots.txt tokens, Google extended and Applebot extended, that only affect training, not search, and Marketing Graham found 583 of 752 audited sites lacked at least one key machine-readable marker.
AI Search Leaves Brands Behind
Even top-ranked brands in traditional search are missing from AI results, as only a handful achieve consistent visibility across major AI platforms.
The evidence now shows the gap is systemic and countable, not anecdotal. According to SEMrush’s 2026 AI Visibility Index, the analysis expanded from “2,500 prompts to 126 million US AI search prompts” from January through April 2026 and found that “mentions are not citations”: on Gemini, overlap between mentioned brands and cited domains can fall to 30%, meaning a brand can be named while none of its own pages support the answer. The same report says “only 36 global brands maintain top 100 visibility across all four major platforms… ChatGPT, Gemini, Google AI mode and Google AI overviews,” tested every single month of the study period, underscoring how rare consistent cross-platform discoverability is. That pattern is reinforced by concentration data showing the top three brands captured 82.9% of visibility in news and media and more than 76% in consumer electronics.
Other large studies show the same discovery gap from a different angle: strong web visibility does not reliably carry into AI answers. Rocket’s 100,000-keyword study found that ranking in Google positions 1 to 10 translated to only about a 44% inclusion rate in AI results, while for ChatGPT the overlap was only around 15%; as the team put it, if this were just SEO, “you would expect to see much more similar results.” That is why GTM Strategist recommends a monthly “AEO audit” cadence of 15 to 25 prompts, and why BrightEdge’s finding that 97% of the top 20,000 websites are not set up for AI agents matters commercially.
