Robots.txt Generator
Create a robots.txt draft with crawler groups, disallowed paths and a sitemap URL, including clear allow-all and block-all modes.
Start with your input
Edit the example or replace it with your own text.
Choose your options
These settings are optional. Keep the defaults to get started.
How to use Robots.txt Generator
- Choose the crawler policy and user-agent token.
- For custom policy, enter one disallowed root-relative path per line.
- Add your absolute sitemap URL, generate the file, and review it before placing it at your site root.
Generate crawler rules deliberately
A robots.txt file is a public instruction to cooperating crawlers. Custom mode adds Disallow rules for the listed paths, allow-all mode writes an empty Disallow rule, and block-all mode disallows the root path. Paths are case-sensitive and belong to the host serving the file. The generator emits one group and an optional Sitemap directive; it does not change this website’s own robots.txt.
Crawling, indexing and access are separate
Disallow prevents compliant crawling; it does not authenticate visitors or reliably remove a discovered URL from search. A noindex directive must be visible to the crawler, so blocking the page can prevent that directive from being read. Keep production assets needed for rendering crawlable and inspect deployed rules carefully, especially after moving a staging configuration to production.
Example input
/admin/ /private/
Can robots.txt keep a private admin page secret?
No. The file is public and its rules do not enforce access control. Protect private routes with authentication and authorization.
Is input uploaded?
Processing takes place in your browser. This tool does not send your input or selected files to a conversion API.