
A robots.txt file sits at yoursite.com/robots.txt and tells crawlers which parts of your site they may fetch. This generator writes one in the standard format (RFC 9309) with your private paths and sitemap, and adds a section most generators skip: rules for the AI crawlers. Pick whether OpenAI, Anthropic, Google, Apple, Perplexity, Meta and Common Crawl may train on your pages, show them in AI answers, or fetch them when a user asks. Then test any URL against the file and copy or download it. Nothing leaves your browser.
We checked every AI user-agent token below against the company's own documentation on 10 November 2026. The sources are linked in the table after the tool.
GPTBot OpenAIClaudeBot AnthropicGoogle-Extended GoogleApplebot-Extended AppleCCBot Common Crawlmeta-externalagent MetaOAI-SearchBot OpenAIClaude-SearchBot AnthropicPerplexityBot Perplexitymeta-webindexer MetaChatGPT-User OpenAIClaude-User AnthropicPerplexity-User Perplexitymeta-externalfetcher MetaWhich AI crawlers should a robots.txt name?
AI companies now run more than one crawler each, and they do different jobs. Blocking one does not block the others, so a single "block AI" line no longer does what most people expect. These are the tokens each company documents today:
| Company | Training | AI search and answers | Fetches when a user asks | Source |
|---|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User | OpenAI crawlers |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User | Anthropic help center |
Google-Extended (control token) | Googlebot (same as Search) | n/a | Google common crawlers | |
| Perplexity | None (PerplexityBot is not used for training) | PerplexityBot | Perplexity-User | Perplexity crawlers |
| Apple | Applebot-Extended (control token) | Applebot | n/a | About Applebot |
| Meta | meta-externalagent | meta-webindexer | meta-externalfetcher | Meta web crawlers |
| Common Crawl | CCBot | n/a | n/a | Common Crawl FAQ |
Five details from those pages change what you should write:
- Training and search are separate switches. OpenAI says a site can allow OAI-SearchBot and disallow GPTBot, and that a site that blocks OAI-SearchBot "will not be shown in ChatGPT search answers". Anthropic says turning off Claude-SearchBot or Claude-User may reduce your visibility in Claude's answers.
- Google-Extended does not touch Search. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It controls whether Google can use your content to train future Gemini models and to ground answers in the Gemini apps and Vertex AI. It is a token you name in robots.txt, not a separate crawler.
- Applebot-Extended does not crawl. Apple says it only decides how data crawled by Applebot is used for training, and pages that disallow it can still appear in search results.
- User-triggered fetchers may not obey the file. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, Perplexity says Perplexity-User "generally ignores robots.txt rules", and Meta says meta-externalfetcher "may bypass" them. Anthropic says its bots honor robots.txt.
- Changes are not instant. OpenAI says a change can take about 24 hours to reach its search systems, Meta says its crawlers may cache the file for up to 24 hours, and Google generally caches it for up to 24 hours.
Older tokens such as anthropic-ai and Claude-Web are not on Anthropic's current list. A file that only names them does not cover Anthropic's crawlers today. Google has also said it plans to support a separate AI-training signal, __usage_ai-training, in early 2027; we will add it to the generator when Google documents the syntax.
Should a B2B company block AI crawlers?
Usually not the ones that send buyers. When a buyer asks ChatGPT, Claude or Perplexity for vendors in your category, the answer can only cite pages the assistant's search crawler was allowed to read. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot takes you out of those answers. For a company that wants to be on a shortlist, that costs more than it saves.
Training crawlers are a real choice. Blocking GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent says you do not want your pages in future training sets. According to OpenAI, Google and Apple, it does not remove you from their search or answer features. The cost is harder to measure: a model trained without your pages knows less about your product. For most B2B marketing sites we recommend allowing everything, publishing the facts you want repeated (pricing model, integrations, who the product is for) and blocking only private areas. If your content is the product, as with paid research or a publisher, blocking training and allowing search is the safer policy. The preset "Visible in AI answers, no training" writes that second policy.
robots.txt controls access. To tell AI tools which pages matter and what each one says, add an llms.txt file with our llms.txt generator. Being cited still comes down to the pages: our guide to generative engine optimization covers that work, and the answer engine optimization entry explains how answer engines pick sources.
How does robots.txt syntax work?
| Line | Example | What it does |
|---|---|---|
User-agent | User-agent: GPTBot | Starts a group for the named crawler. Several User-agent lines in a row share one group. * means every crawler without its own group |
Disallow | Disallow: /admin/ | The crawler may not fetch URLs that start with this path. Disallow: / blocks the whole site |
Allow | Allow: /admin/public-assets/ | An exception inside a disallowed path |
Sitemap | Sitemap: https://site.com/sitemap.xml | Full URL of a sitemap. Not tied to any group |
* and $ | Disallow: /*.pdf$ | * matches any characters; $ marks the end of the URL |
# | # AI training crawlers | A comment. Crawlers ignore it |
Three rules from Google's robots.txt documentation catch people out:
- A crawler follows one group only. If GPTBot has its own group, it ignores the
User-agent: *group. That is why the generator repeats your private paths inside every AI group it allows. - The longest matching rule wins.
Allow: /admin/public-assets/beatsDisallow: /admin/because it is longer. When an Allow and a Disallow match with the same length, Google uses the less restrictive one, the Allow. - Paths are case-sensitive.
/Admin/and/admin/are different paths. Field names likeuser-agentare not case-sensitive.
Google reads only the first 500 KiB of the file, and RFC 9309 says crawlers must parse at least that much. The file must be UTF-8 plain text, named robots.txt, at the root of each host: blog.yoursite.com needs its own file.
What should a B2B SaaS site block?
| Block | Why | Do not block |
|---|---|---|
Internal search results (/search, /?s=) | Endless thin pages that waste crawling | CSS and JavaScript files: Google needs them to render the page |
Sort and filter parameters (/*?sort=) | Duplicate copies of the same list | Pricing, product, integration and docs pages, which AI answers quote |
| Admin and app login areas | Nothing there for a crawler (and a login is the real protection) | Pages you want removed from Google: use noindex instead, so Google can read it |
| Thank-you and confirmation pages | They inflate conversion reports when visited directly | Your sitemap file |
| The whole site, on staging only | Keeps test copies out of search | The whole site, on production. Check this after every launch |
Blocking a page does not remove it from Google. Google says a disallowed URL can still be indexed and shown without a snippet if other pages link to it. Our technical SEO entry covers indexing controls, and the SaaS technical SEO guide covers what to keep out of the index on a product site.
How do you publish and test your robots.txt?
- Upload it to the root. On Astro, Next.js and other static builds, put
robots.txtin thepublic/folder. On WordPress, use your SEO plugin's robots.txt editor or upload the file by SFTP. On site builders, look for a robots.txt setting in the site's SEO settings or help center. - Open it in a browser.
https://yoursite.com/robots.txtshould load as plain text with a 200 status. Google treats a 4xx response (except 429) as if there were no file, and a 5xx error as a reason to pause crawling. - Check it in Search Console. The robots.txt report under Settings shows the version Google fetched and any lines it could not parse.
- Test your key pages. Use the tester in the generator above on your home page, pricing page and a blog post, for Googlebot and for each AI crawler you allowed.
- Watch your logs for a month. Your server or CDN logs show which crawlers came back and whether blocked ones stopped. Some CDNs, such as Cloudflare, can also block AI crawlers at the network level; check that a CDN rule is not overriding the file.
Where Growigami fits
Our AI search and SEO service sets up the technical side (robots.txt, llms.txt, sitemaps and structured data) and then does the part that moves results: asking the assistants your buyers use the questions they ask, and writing the pages those answers draw on. To check whether search can pay for itself first, try the SEO ROI calculator.
Frequently asked
What is a robots.txt file?
A plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which URLs they may fetch. It is made of groups: one or more User-agent lines naming the crawler, then Allow and Disallow rules with URL paths, plus optional Sitemap lines. The format is a standard, RFC 9309, published by the IETF in September 2022.
How do I block AI crawlers like GPTBot and ClaudeBot?
Add a group for each crawler with Disallow: /, for example "User-agent: GPTBot" then "Disallow: /". Each AI company publishes its own tokens: OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User, Perplexity uses PerplexityBot, and Google uses the Google-Extended token for Gemini training. The generator on this page writes these groups for you.
Will blocking GPTBot remove my site from ChatGPT?
No. OpenAI says its settings are independent: GPTBot is for training, OAI-SearchBot is for ChatGPT search. A site can block GPTBot and still allow OAI-SearchBot. If you block OAI-SearchBot, OpenAI says your site will not be shown in ChatGPT search answers, though it can still appear as a navigational link.
Does robots.txt stop a page from being indexed?
Not reliably. Google says a page blocked by robots.txt can still be indexed if other pages link to it, and may show in results without a snippet. To keep a page out of Google, let it be crawled and add a noindex meta tag or X-Robots-Tag header, or put it behind a login. Do not block a page in robots.txt and add noindex at the same time: Google cannot read the noindex on a page it is not allowed to fetch.
Does Google support crawl-delay in robots.txt?
No. Google's robots.txt documentation lists only user-agent, allow, disallow and sitemap as supported fields, and says other fields such as crawl-delay are not supported. Some other crawlers do read it: Anthropic says its bots support the non-standard Crawl-delay extension.
Is robots.txt a security measure?
No. RFC 9309 says the rules "are not a form of access authorization", and well-behaved crawlers follow them by choice. The file is public, so listing a secret path in it tells everyone where it is. Protect private pages with a login, and use robots.txt only to steer crawlers.



















