Common website
Static worked example — not your live result.
Input
allow / + disallow /admin/
Output
User-agent: *\nAllow: /\nDisallow: /admin/
Generators
robots.txt Generator
Create a robots.txt for crawl directives. The robots.txt Generator includes helpful presets and warns when rules block the whole site so you do not publish restrictive files by accident.
How the robots.txt Generator Works
Step 1
Choose Crawl Rules
Pick a preset or enter custom paths.
Step 2
Generate robots.txt
Create the file text instantly.
Step 3
Copy or Download
Publish only after reviewing restrictive rules.
robots.txt Features
- Allow all / block all / common / CMS presets
- Custom allow and disallow paths
- Sitemap URL support
- Warning for whole-site blocks
- Copy or download robots.txt
Output Details
Input formats
- Options / text inputs
Output formats
- robots.txt
Supported formats
- robots.txt
Examples
Frequently Asked Questions
Can robots.txt block every search engine?
A Disallow: / rule can ask crawlers not to fetch pages, but compliance varies and authentication still matters for private content.
Does this submit rules to Google?
No. You host robots.txt on your site; search engines fetch it themselves.
Why the warning on block-all?
Blocking “/” is highly restrictive and easy to mispublish.
Related Tools
Explore related Tool Base utilities that pair well with robots.txt Generator.
Sitemap XML Generator
New
Generate an XML sitemap from a URL list.
Generators
Meta Tags Generator
New
Generate HTML meta tags for pages.
Generators
HTML Boilerplate Generator
New
Generate a basic HTML5 document template.
Generators
Open Graph Generator
New
Generate Open Graph and optional Twitter card tags.
Generators
Need another utility?
Browse related tools or explore the full Tool Base directory.