robots.txt Generator
Choose how crawlers should treat the site, list the paths to keep out of search, and copy the file to your web root.
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?s=
Sitemap: https://example.com/sitemap.xml
About this tool
robots.txt lives at the domain root and tells crawlers which paths to skip. It is a crawling instruction, not an access control: blocked URLs can still be indexed if other sites link to them, so use a noindex meta tag or authentication for anything that must stay out of results.
Rules are grouped per user-agent, and each crawler follows the most specific group that matches it. The AI option adds explicit blocks for GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and other model-training crawlers that honour the file.
Frequently asked questions
- Where do I put the file?
- At the root of the domain, reachable as https://example.com/robots.txt. Subdirectories do not work.
- Does Google honour crawl-delay?
- No. Googlebot ignores it; Bing and Yandex honour it. Set the crawl rate in Search Console instead.
- Will blocking a page remove it from Google?
- Not reliably. A disallowed URL can still appear without a snippet. Use a noindex meta tag on a crawlable page for real removal.