Tool Β· Technical SEO

Robots.txt Generator

Free robots.txt generator with current AI crawler rules: block GPTBot, ClaudeBot and Google-Extended training while staying visible in ChatGPT, Claude and Perplexity answers. Test any URL.

Free, no signup Updated November 10, 2026

A robots.txt file sits at yoursite.com/robots.txt and tells crawlers which parts of your site they may fetch. This generator writes one in the standard format (RFC 9309) with your private paths and sitemap, and adds a section most generators skip: rules for the AI crawlers. Pick whether OpenAI, Anthropic, Google, Apple, Perplexity, Meta and Common Crawl may train on your pages, show them in AI answers, or fetch them when a user asks. Then test any URL against the file and copy or download it. Nothing leaves your browser.

We checked every AI user-agent token below against the company's own documentation on 10 November 2026. The sources are linked in the table after the tool.

1. Rules for all crawlers
Used to build the default sitemap link
Search engines like Googlebot and Bingbot follow this group
Start each path with /. Use * for any characters and $ for the end of the URL. Leave empty to allow everything.
2. AI crawlers
Or set each crawler. "Follow default" writes no group, so the crawler obeys the User-agent: * rules above.
AI training crawlers
GPTBot OpenAI
Collects content that may be used to train OpenAI models
ClaudeBot Anthropic
Collects content that could be used to train Claude models
Google-Extended Google
Control token only (no separate crawler): Gemini training and grounding. Does not affect Google Search
Applebot-Extended Apple
Control token only: use of Applebot data to train Apple models. Does not affect search
CCBot Common Crawl
Builds the Common Crawl open web archive
meta-externalagent Meta
Training Meta AI models and indexing content for its products
AI search and answer crawlers
OAI-SearchBot OpenAI
Shows sites in ChatGPT search answers
Claude-SearchBot Anthropic
Improves search results inside Claude
PerplexityBot Perplexity
Surfaces and links sites in Perplexity answers. Not used for training
meta-webindexer Meta
Improves Meta AI search results
User-triggered AI fetchers
ChatGPT-User OpenAI
Visits a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply
Claude-User Anthropic
Fetches a page when a Claude user asks a question about it
Perplexity-User Perplexity
Fetches a page for a user. Perplexity says it generally ignores robots.txt
meta-externalfetcher Meta
Fetches links at a user request. Meta says it may bypass robots.txt
3. Extras
Google ignores this line. Leave empty unless a bot is overloading your server
Lines starting with # are ignored by crawlers
Groups
0
AI crawlers blocked
0
AI crawlers allowed
0
File size
0 KB
πŸ“Š Set your rules to build the file.
    4. Test a URL

    Uses Google's rules: the crawler follows the most specific group that names it (or *), the longest matching rule wins, and Allow wins a tie.

    Which AI crawlers should a robots.txt name?

    AI companies now run more than one crawler each, and they do different jobs. Blocking one does not block the others, so a single "block AI" line no longer does what most people expect. These are the tokens each company documents today:

    CompanyTrainingAI search and answersFetches when a user asksSource
    OpenAIGPTBotOAI-SearchBotChatGPT-UserOpenAI crawlers
    AnthropicClaudeBotClaude-SearchBotClaude-UserAnthropic help center
    GoogleGoogle-Extended (control token)Googlebot (same as Search)n/aGoogle common crawlers
    PerplexityNone (PerplexityBot is not used for training)PerplexityBotPerplexity-UserPerplexity crawlers
    AppleApplebot-Extended (control token)Applebotn/aAbout Applebot
    Metameta-externalagentmeta-webindexermeta-externalfetcherMeta web crawlers
    Common CrawlCCBotn/an/aCommon Crawl FAQ

    Five details from those pages change what you should write:

    • Training and search are separate switches. OpenAI says a site can allow OAI-SearchBot and disallow GPTBot, and that a site that blocks OAI-SearchBot "will not be shown in ChatGPT search answers". Anthropic says turning off Claude-SearchBot or Claude-User may reduce your visibility in Claude's answers.
    • Google-Extended does not touch Search. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It controls whether Google can use your content to train future Gemini models and to ground answers in the Gemini apps and Vertex AI. It is a token you name in robots.txt, not a separate crawler.
    • Applebot-Extended does not crawl. Apple says it only decides how data crawled by Applebot is used for training, and pages that disallow it can still appear in search results.
    • User-triggered fetchers may not obey the file. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, Perplexity says Perplexity-User "generally ignores robots.txt rules", and Meta says meta-externalfetcher "may bypass" them. Anthropic says its bots honor robots.txt.
    • Changes are not instant. OpenAI says a change can take about 24 hours to reach its search systems, Meta says its crawlers may cache the file for up to 24 hours, and Google generally caches it for up to 24 hours.

    Older tokens such as anthropic-ai and Claude-Web are not on Anthropic's current list. A file that only names them does not cover Anthropic's crawlers today. Google has also said it plans to support a separate AI-training signal, __usage_ai-training, in early 2027; we will add it to the generator when Google documents the syntax.

    Should a B2B company block AI crawlers?

    Usually not the ones that send buyers. When a buyer asks ChatGPT, Claude or Perplexity for vendors in your category, the answer can only cite pages the assistant's search crawler was allowed to read. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot takes you out of those answers. For a company that wants to be on a shortlist, that costs more than it saves.

    Training crawlers are a real choice. Blocking GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent says you do not want your pages in future training sets. According to OpenAI, Google and Apple, it does not remove you from their search or answer features. The cost is harder to measure: a model trained without your pages knows less about your product. For most B2B marketing sites we recommend allowing everything, publishing the facts you want repeated (pricing model, integrations, who the product is for) and blocking only private areas. If your content is the product, as with paid research or a publisher, blocking training and allowing search is the safer policy. The preset "Visible in AI answers, no training" writes that second policy.

    robots.txt controls access. To tell AI tools which pages matter and what each one says, add an llms.txt file with our llms.txt generator. Being cited still comes down to the pages: our guide to generative engine optimization covers that work, and the answer engine optimization entry explains how answer engines pick sources.

    How does robots.txt syntax work?

    LineExampleWhat it does
    User-agentUser-agent: GPTBotStarts a group for the named crawler. Several User-agent lines in a row share one group. * means every crawler without its own group
    DisallowDisallow: /admin/The crawler may not fetch URLs that start with this path. Disallow: / blocks the whole site
    AllowAllow: /admin/public-assets/An exception inside a disallowed path
    SitemapSitemap: https://site.com/sitemap.xmlFull URL of a sitemap. Not tied to any group
    * and $Disallow: /*.pdf$* matches any characters; $ marks the end of the URL
    ## AI training crawlersA comment. Crawlers ignore it

    Three rules from Google's robots.txt documentation catch people out:

    1. A crawler follows one group only. If GPTBot has its own group, it ignores the User-agent: * group. That is why the generator repeats your private paths inside every AI group it allows.
    2. The longest matching rule wins. Allow: /admin/public-assets/ beats Disallow: /admin/ because it is longer. When an Allow and a Disallow match with the same length, Google uses the less restrictive one, the Allow.
    3. Paths are case-sensitive. /Admin/ and /admin/ are different paths. Field names like user-agent are not case-sensitive.

    Google reads only the first 500 KiB of the file, and RFC 9309 says crawlers must parse at least that much. The file must be UTF-8 plain text, named robots.txt, at the root of each host: blog.yoursite.com needs its own file.

    What should a B2B SaaS site block?

    BlockWhyDo not block
    Internal search results (/search, /?s=)Endless thin pages that waste crawlingCSS and JavaScript files: Google needs them to render the page
    Sort and filter parameters (/*?sort=)Duplicate copies of the same listPricing, product, integration and docs pages, which AI answers quote
    Admin and app login areasNothing there for a crawler (and a login is the real protection)Pages you want removed from Google: use noindex instead, so Google can read it
    Thank-you and confirmation pagesThey inflate conversion reports when visited directlyYour sitemap file
    The whole site, on staging onlyKeeps test copies out of searchThe whole site, on production. Check this after every launch

    Blocking a page does not remove it from Google. Google says a disallowed URL can still be indexed and shown without a snippet if other pages link to it. Our technical SEO entry covers indexing controls, and the SaaS technical SEO guide covers what to keep out of the index on a product site.

    How do you publish and test your robots.txt?

    1. Upload it to the root. On Astro, Next.js and other static builds, put robots.txt in the public/ folder. On WordPress, use your SEO plugin's robots.txt editor or upload the file by SFTP. On site builders, look for a robots.txt setting in the site's SEO settings or help center.
    2. Open it in a browser. https://yoursite.com/robots.txt should load as plain text with a 200 status. Google treats a 4xx response (except 429) as if there were no file, and a 5xx error as a reason to pause crawling.
    3. Check it in Search Console. The robots.txt report under Settings shows the version Google fetched and any lines it could not parse.
    4. Test your key pages. Use the tester in the generator above on your home page, pricing page and a blog post, for Googlebot and for each AI crawler you allowed.
    5. Watch your logs for a month. Your server or CDN logs show which crawlers came back and whether blocked ones stopped. Some CDNs, such as Cloudflare, can also block AI crawlers at the network level; check that a CDN rule is not overriding the file.

    Where Growigami fits

    Our AI search and SEO service sets up the technical side (robots.txt, llms.txt, sitemaps and structured data) and then does the part that moves results: asking the assistants your buyers use the questions they ask, and writing the pages those answers draw on. To check whether search can pay for itself first, try the SEO ROI calculator.

    Frequently asked

    What is a robots.txt file?

    A plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which URLs they may fetch. It is made of groups: one or more User-agent lines naming the crawler, then Allow and Disallow rules with URL paths, plus optional Sitemap lines. The format is a standard, RFC 9309, published by the IETF in September 2022.

    How do I block AI crawlers like GPTBot and ClaudeBot?

    Add a group for each crawler with Disallow: /, for example "User-agent: GPTBot" then "Disallow: /". Each AI company publishes its own tokens: OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User, Perplexity uses PerplexityBot, and Google uses the Google-Extended token for Gemini training. The generator on this page writes these groups for you.

    Will blocking GPTBot remove my site from ChatGPT?

    No. OpenAI says its settings are independent: GPTBot is for training, OAI-SearchBot is for ChatGPT search. A site can block GPTBot and still allow OAI-SearchBot. If you block OAI-SearchBot, OpenAI says your site will not be shown in ChatGPT search answers, though it can still appear as a navigational link.

    Does robots.txt stop a page from being indexed?

    Not reliably. Google says a page blocked by robots.txt can still be indexed if other pages link to it, and may show in results without a snippet. To keep a page out of Google, let it be crawled and add a noindex meta tag or X-Robots-Tag header, or put it behind a login. Do not block a page in robots.txt and add noindex at the same time: Google cannot read the noindex on a page it is not allowed to fetch.

    Does Google support crawl-delay in robots.txt?

    No. Google's robots.txt documentation lists only user-agent, allow, disallow and sitemap as supported fields, and says other fields such as crawl-delay are not supported. Some other crawlers do read it: Anthropic says its bots support the non-standard Crawl-delay extension.

    Is robots.txt a security measure?

    No. RFC 9309 says the rules "are not a form of access authorization", and well-behaved crawlers follow them by choice. The file is public, so listing a secret path in it tells everyone where it is. Protect private pages with a login, and use robots.txt only to steer crawlers.

    Start with one process.

    We learn the business and build your first go-to-market process on us this month.