Free tool
robots.txt generator.
Build a valid robots.txt from user-agent groups, allow and disallow rules and a sitemap line, with presets for WordPress, Shopify, a staging site and the AI crawlers. It flags the errors that actually cause damage, and it is direct about the thing most generators leave out: Disallow controls crawling, not indexing, and blocking a URL is not how you remove it from Google.
Start from a preset
Each one loads a real starting point you can then edit. Nothing is locked.
Your groups
One rule per line. Write Disallow: /path or Allow: /path, or just /path and it is treated as Disallow.
robots.txt
Pick a preset or add a group to start.
Read this before you block anything
Disallow does not remove a page from Google.
This is the single most common and most costly misreading of the file, and it is why the tool warns rather than just generating.
Disallow stops crawling, not indexing. If Google already knows a URL exists, usually because something links to it, it can list that URL in results without ever fetching it. You get a result with no description and a line saying no information is available for this page. The page is still there, it just looks broken.
Blocking guarantees a noindex tag is never seen. Reading the tag requires fetching the page. Disallow that URL and the crawler never gets far enough to read the instruction telling it to drop the page, so the two directives cancel each other out. If you want something out of the index, allow the crawl and use noindex, or return a 404 or 410.
Blocking does not make content private. The file is public at your own root and it is the first place anybody curious looks. Listing /secret-admin-area/ in it advertises the path to every scraper on the internet. Anything that genuinely must not be reached needs authentication, not a polite request.
The longer version, with the Search Console statuses it produces, is in robots.txt does not stop indexing.
The AI question
Blocking AI crawlers, without the two mistakes.
The AI preset writes a group per known training crawler rather than one line, because they are separate agents and a single wildcard would take Googlebot with it. Two things are worth knowing before you use it.
Google-Extended is not Googlebot. It governs whether your content trains Gemini and related products. Google has said it does not affect crawling for Search or how you rank, so blocking it costs you nothing in the results. Blocking Googlebot itself removes you from Search entirely, which is why the tool warns if you ever write that rule.
It is a convention, not a control. Well behaved crawlers honour the file. Anything determined to take your content ignores it, and no line you write changes that. If scraping is the actual problem, the answer is rate limiting and blocking at the edge, which is a firewall job.
Whether to block at all is a genuine trade rather than an obvious yes. You are choosing between keeping your work out of a training set and staying visible in the assistants a growing number of people now ask instead of searching. We publish llms.txt on this domain and do not block the training crawlers, but that is a decision for a business that sells expertise by being read, and yours may differ.
FAQ
Questions about robots.txt
Will blocking a page in robots.txt remove it from Google?
No, and this is the most expensive misunderstanding about the file. Disallow controls crawling, not indexing. If Google already knows a URL exists, usually from a link somewhere, it can index that URL without ever fetching it, which produces a result with no description and the note that no information is available for this page.
Worse, blocking the URL guarantees Google cannot see a noindex tag on it, because reading the tag requires fetching the page you just blocked. To remove something from the index, allow the crawl and use noindex, or return a 404 or 410, or use the removals tool for something urgent. There is a full write up in robots.txt does not stop indexing.
Does blocking Google-Extended hurt my rankings?
No. Google-Extended is a separate control that governs whether your content is used for training Gemini and other AI products. Google has stated it does not affect crawling for Search or how pages rank.
So blocking it is a genuine choice about AI training with no ranking cost. Blocking Googlebot itself is an entirely different act and will remove you from Search, which is why the two are kept apart in the output above.
Should I block AI crawlers?
It depends on what your content is for, and we would rather set out the trade than answer for you. Blocking training crawlers protects your work from being absorbed into a model. It also removes you from a growing source of referral and citation, and some assistants that fetch pages live to answer a question will simply not see you.
Note too that these are honoured by convention rather than enforced. A crawler that ignores robots.txt is not stopped by a line in it, which is a firewall question rather than a file question.
Do I need a robots.txt at all?
Not necessarily. A site with nothing to hide and no crawl budget problem works perfectly well without one, and an absent file simply means everything is allowed. A missing robots.txt is not an error and Search Console will not complain.
It becomes worth having when you have genuinely useless URLs eating crawl budget, an admin area, a staging environment, or a decision to make about AI crawlers. Adding one with a single empty Disallow achieves nothing except a request.
Where does the file go?
At the root of the host, always: https://example.com/robots.txt. It cannot live in a subfolder, and each subdomain needs its own, so a file on the main domain does not govern a shop or blog subdomain.
Protocol and host must match too. The file on https does not apply to http, and www and non-www are treated as different hosts unless one redirects to the other.
Keep reading
Related
Book me
Not sure whether a block is costing you?
Send the domain. You get a straight answer on whether anything important is being blocked, whether pages are indexed without being crawled, and what the file should actually say.