Most businesses that want customers to find them should block only the AI training bots in robots.txt, if anything. Each AI company operates separate bots for training and for search. Blocking the search bots removes a website from those AI answers.
Why "block all AI bots" advice causes problems
Many lists online tell business owners to block every AI bot in robots.txt. That advice mixes up two different jobs. Each AI company operates separate bots: one set crawls content for training, and another set crawls content to show search results. Blocking the wrong bot can remove a website from AI answers. Google's AI Overviews and AI Mode work differently still. They are controlled through Googlebot access, plus a separate setting inside Search Console.
An llms.txt file does not control any of these bots. Our guide on why small business websites don't need llms.txt covers that file and what it does.
OpenAI operates three separate bots for ChatGPT
According to OpenAI developer docs, GPTBot collects content that may train its models, and blocking it signals that content "should not be used in training." OAI-SearchBot is a different bot. It's used to show websites in ChatGPT search, so a blocked website won't appear in those answers. OpenAI says it can still appear as a plain navigational link. ChatGPT-User is triggered when a person asks ChatGPT to visit a page directly. OpenAI says "robots.txt rules may not apply" to that bot, and it doesn't decide whether a website shows up in search results. These three settings work independently of each other, and OpenAI says robots.txt changes can take about 24 hours to take effect. Our guide on how ChatGPT search finds websites walks through these checks in more detail.
Anthropic's Claude bots follow a similar pattern
Anthropic's Claude Help Center describes three bots with three jobs. ClaudeBot handles training, and blocking it signals that future content "should be excluded from our AI model training datasets." Claude-SearchBot is separate. Anthropic says blocking it "prevents our system from indexing your content for search optimization." Claude-User responds to a specific person's request inside Claude, and blocking it "prevents our system from retrieving your content in response to a user query." Anthropic states that its bots honor robots.txt rules.
Perplexity and Bing set their own crawling rules
Perplexity docs describe PerplexityBot as the bot that surfaces and links websites in Perplexity's results. Perplexity says this bot is "not used to crawl content for AI foundation models." It recommends allowing the bot so a website can appear in results. Perplexity-User handles fetches tied to a specific person's request, and Perplexity says this bot "generally ignores robots.txt rules." Perplexity says robots.txt changes can take up to 24 hours to apply.
Microsoft's Bing Webmaster Guidelines cover Copilot and grounding results, and they advise against blocking Bingbot. A NOARCHIVE tag keeps content out of Copilot responses specifically.
Google-Extended and AI Overviews use separate controls
Google's Crawling Infrastructure documentation explains that Google-Extended controls how content is used for Gemini training and for grounding the Gemini app. Google states that this token "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
AI Overviews and AI Mode are a separate matter. Search Console Help describes a "Search generative AI" control that applies to all websites, and inclusion is the default setting. Turning it off means a website won't get traffic or impressions from AI Overviews or AI Mode. Google says this choice does not affect other rankings or AI training. Google Search Central lists the controls for Search: Googlebot access through robots.txt, plus the nosnippet, data-nosnippet, max-snippet, and noindex tags.
What a business owner should check
A few checks cover most of this ground:
- To stay visible in AI answers, allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Bingbot.
- To opt out of training, block GPTBot, ClaudeBot, and Google-Extended. None of these remove a website from Google Search.
- Avoid blocking Googlebot as a way to stop AI Overviews. Use the Search Console control for that instead.
- Ask the host or security provider whether a firewall "AI bot" switch is also blocking the search bots listed above.
- Remember that robots.txt is a request, and not every bot follows it. User-triggered fetchers like ChatGPT-User and Perplexity-User may skip those rules, so content meant to stay private needs a login instead.
Frequently asked questions
If I block GPTBot, will ChatGPT stop showing my website in answers?
No. ChatGPT search uses OAI-SearchBot, a separate bot from GPTBot. Blocking GPTBot only signals that content shouldn't be used for training.
Does Google-Extended remove a website from AI Overviews?
No. Google-Extended covers Gemini training and grounding for the Gemini app. AI Overviews and AI Mode are controlled separately through the Search Console "Search generative AI" setting.
What happens if I block Claude-SearchBot?
Anthropic says blocking Claude-SearchBot stops its system from indexing that content for search. This is a separate setting from ClaudeBot, which handles training.
Do AI bots follow robots.txt rules?
The training and search bots covered here say they honor robots.txt. User-triggered fetchers, like ChatGPT-User and Perplexity-User, may skip those rules since they respond to a specific person's request.
How long does a robots.txt change take to apply?
OpenAI says changes can take about 24 hours to take effect. Perplexity says its changes can take up to 24 hours as well.
Blocking a search bot removes a website from that tool's AI answers. Blocking a training bot limits training use, so check each bot by name.
For help reviewing these settings alongside the rest of a website's setup, see TygerLabs' SEO, AEO and GEO services.
