Robots.txt Generator guide
Create a robots.txt file for your site in seconds. Choose which paths crawlers should skip, add exceptions and sitemap URLs, optionally block AI training bots, then copy or download the file for your site root.
What robots.txt does, and what it does not
robots.txt is a plain text file at the root of your domain that tells crawlers which URLs they may request. It was formalized as RFC 9309 in 2022, after nearly 30 years as an informal convention. Googlebot, Bingbot, and most reputable crawlers follow it.
It controls crawling, not indexing. If other sites link to a URL you have disallowed, Google can still list it, typically with no snippet. To remove a page from search results, let Google crawl it and serve a noindex meta tag or X-Robots-Tag header. Blocking a noindexed page in robots.txt actually hides the noindex from Google, the opposite of what you want.
How the generator builds the file
It writes one group for all crawlers, User-agent: *, followed by your Allow exceptions and Disallow rules. Paths without a leading slash are fixed and flagged, because Disallow: admin matches nothing. If you leave both lists empty, it writes an empty Disallow: line, which is the valid way to say "crawl everything".
Ticking the AI option adds a separate group per AI crawler with Disallow: /. Crawlers obey the most specific group that names them, so GPTBot follows its own group and ignores the * rules. Sitemap lines go at the end; they can appear anywhere in the file and must be absolute URLs.
Worked example: a small online store
You want Google to skip the admin area, the cart, and session-tracking URLs, but one public folder inside admin holds images used on product pages. The defaults produce one line each for User-agent: *, Allow: /admin/public/, Disallow: /admin/, Disallow: /cart/, and Disallow: /*?sessionid=, then a blank line and Sitemap: https://example.com/sitemap.xml.
Google resolves conflicts by picking the most specific matching rule, meaning the longest path. /admin/public/logo.png matches both Allow: /admin/public/ (14 characters) and Disallow: /admin/ (7 characters). The longer Allow wins, so the image stays crawlable. When an Allow and a Disallow are the same length, Google uses the less restrictive Allow.
Pattern matching rules
Rules are prefix matches and case-sensitive. Disallow: /admin blocks /admin, /admin/, /admin.html, and /administrator. Add a trailing slash, /admin/, to block only the folder.
* matches any run of characters and $ anchors the end. Disallow: /*.pdf$ blocks PDFs but not /guide.pdf?download=1. Disallow: /*? blocks every URL with a query string, which is often too aggressive because it can block paginated or filtered pages you want indexed.
Common robots.txt mistakes
Shipping a staging file to production. Disallow: / on a live site can drop it out of Google within days. Check yours after every deploy by loading /robots.txt in a browser.
Blocking CSS and JavaScript. Google renders pages like a browser. If it cannot fetch your stylesheets and scripts, it may judge pages as broken or not mobile friendly. Do not disallow /assets/, /static/, or /_astro/ folders that pages need.
Serving the file with the wrong status. If robots.txt returns a 5xx server error, Google treats the whole site as disallowed for a while. A 404 is fine: it means no restrictions.
Exceeding the size limit. Google reads only the first 500 KiB of the file. That is enormous for rules, but auto-generated files with thousands of lines can hit it.
AI crawlers in 2026
Most major AI companies now publish user-agent tokens and say they honor robots.txt: GPTBot for OpenAI training, ClaudeBot for Anthropic, Google-Extended for Gemini training (it is a control token, not a separate crawler), Applebot-Extended for Apple, CCBot for Common Crawl, which many models train on, and PerplexityBot. Blocking them is a business choice. You keep your content out of training sets, but you may also appear less often when people ask AI assistants for recommendations.
Blocking Google-Extended does not change how Googlebot crawls or ranks your pages in regular Search. Some operators also run separate user-triggered fetchers that may not follow the same rules, so check each company's documentation if this matters to you.
How we calculate: sources
Frequently asked questions
Where do I put robots.txt?
At the root of the host, so it loads at https://yourdomain.com/robots.txt. Crawlers ignore robots.txt files in subfolders, and each subdomain needs its own file.
Does robots.txt stop a page from appearing in Google?
No. It stops crawling, not indexing. A blocked URL can still be indexed from links, usually without a description. To keep a page out of search results, allow crawling and add a noindex meta robots tag instead.
What does Disallow: / mean?
It blocks the whole site for the user-agent in that group. An empty Disallow: line means the opposite: nothing is blocked.
Should I block AI crawlers like GPTBot?
It depends on whether you want your content used for AI training or answers. GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot (Common Crawl), and others honor robots.txt. Blocking Google-Extended does not affect normal Google Search rankings.
Does Google support crawl-delay?
No. Googlebot ignores the crawl-delay directive. Bing and some other crawlers respect it. Google adjusts crawl rate automatically based on how your server responds.
Can I use wildcards in robots.txt?
Yes. Google and Bing support * to match any sequence of characters and $ to anchor the end of a URL. Disallow: /*.pdf$ blocks every URL ending in .pdf.
Is robots.txt a security measure?
No. It is a public request that polite bots follow. Anyone can read it, and it can reveal paths you would rather hide. Protect private areas with authentication.
Is data uploaded?
Everything runs in your browser. Nothing you enter is uploaded to a server or stored by us.
How do I test my robots.txt?
Open Google Search Console, go to Settings, and view the robots.txt report. It shows the version Google fetched, when, and any parse errors. For a single URL, the URL Inspection tool tells you if it is blocked.
How long does Google take to notice robots.txt changes?
Google generally caches robots.txt for up to 24 hours, so changes can take a day to apply. You can request a recrawl from the robots.txt report in Search Console.