AI crawlers are not all doing the same job
The term AI crawler covers several different activities. Some automated systems gather material that may be used for model training. Others retrieve pages for AI-powered search results. Some fetch a page only because a user has asked an assistant to visit it.
That distinction changes the decision a website owner needs to make. OpenAI, for example, documents OAI-SearchBot for ChatGPT search and GPTBot for content that may be used in training its generative AI models. It also documents ChatGPT-User for certain user-triggered visits. Blocking one does not necessarily express the same preference for the others.
Anthropic similarly documents separate bot activity for model development and user-directed retrieval. Google uses the Google-Extended robots.txt token to let publishers control particular uses of crawled content for Gemini products while stating that Google-Extended does not affect inclusion or ranking in Google Search.
Why a blanket block can have unintended consequences
A rule written to stop training crawlers can become a visibility problem if it also blocks a search crawler you intended to allow. The reverse is also true. Opening every automated user agent may expose more content than the organisation wants crawled or create unnecessary traffic on a constrained server.
The first task is therefore classification. Identify the crawler, confirm its documented purpose with the provider and decide whether that purpose is acceptable. Do not rely only on a user-agent name copied from an old blog post because crawler names, documentation and policies can change.
For businesses interested in ChatGPT search visibility, OpenAI states that OAI-SearchBot should not be blocked if pages are to be eligible for summaries and snippets in ChatGPT search. GPTBot has a separate control. Google takes a different approach: pages appearing as supporting links in AI Overviews and AI Mode follow the normal Google Search eligibility rules.
robots.txt, firewalls and server rules
robots.txt is normally the first policy layer. It tells compliant automated crawlers which paths they are permitted to fetch. It is not an authentication system and it does not make private information private. Sensitive content should be protected using proper access controls rather than a Disallow instruction.
Server and CDN controls solve a different problem. A bot may be permitted in robots.txt but still receive a 403 response from a firewall, rate-limit rule or bot-management service. OpenAI specifically notes that security systems can inadvertently block legitimate crawler traffic. In practice, troubleshooting therefore means checking both the published robots policy and the HTTP response the crawler actually receives.
Server blocking may also be appropriate for unwanted automated traffic, but rules need care. A badly written firewall expression can block legitimate search crawlers, ordinary visitors or shared infrastructure. Where providers publish verification methods or IP information, those should be checked rather than trusting a claimed user-agent string alone.
A practical policy for common AI crawlers
A good crawler policy starts with the organisation's objective rather than somebody else's universal allowlist or blocklist. A publisher concerned about training use may make different choices from a retailer that wants maximum visibility in AI-assisted product research.
- OpenAI: OAI-SearchBot controls automatic crawling for ChatGPT search discovery, while GPTBot is documented separately for potential model-training use.
- Google: Google-Extended controls specified Gemini training and grounding uses and does not control Google Search ranking or inclusion.
- Anthropic: ClaudeBot is documented for model development, alongside separate user-related bot activity.
- Common Crawl: CCBot collects material for the Common Crawl open web dataset and documents robots.txt opt-out controls.
The policy should be reviewed whenever a provider changes its documentation or introduces a new crawler. Our wider AI SEO service deals with visibility and machine-readable content, while crawler management deals specifically with who is allowed to fetch what.
How we manage AI crawler access
We start by reviewing the existing robots.txt file, server configuration and any CDN or security layer that can affect automated requests. Existing Googlebot access is treated separately because a change intended for an AI crawler should not casually alter normal search crawling.
We then define a policy by crawler purpose, implement the appropriate robots.txt instructions and use server-level controls where they are justified. Testing includes requesting affected paths, checking returned HTTP status codes and confirming that unrelated crawlers and public pages still behave as intended.
AI crawler management is available from £39 a month. Because crawler behaviour and provider documentation evolve, ongoing management is most useful where a business wants the policy checked as those systems change.
What crawler controls cannot do
Blocking a crawler does not remove copies of content that were collected previously, nor does it remove information held by third parties. It also does not guarantee that every unidentified or malicious scraper will obey robots.txt.
Allowing a crawler is equally limited. Permission to fetch content does not guarantee indexing, inclusion, citation, ranking or traffic. It simply removes one potential access barrier for that crawler.
If server logs show unexplained automated traffic, the useful diagnostic information includes the user-agent, source IP where available, requested URLs, request rate, HTTP response codes and timestamps. Those details make it possible to distinguish a policy question from a performance or website security problem.