Robots.txt Generator
Write a robots.txt that crawlers obey.
Input
This tool runs entirely in your browser. The data you paste or load is never uploaded.
Result
Enter your input, then run the tool.
About this tool
Builds a valid robots.txt with per-bot allow and disallow rules, a sitemap reference and an optional crawl delay. It explains that robots.txt is a request, not access control, and warns against blocking assets a page needs.
How to use it
- Choose whether to allow all crawlers or list specific user agents.
- Add the paths you need to block, one per line.
- Copy the output to /robots.txt at the root of your domain.
Frequently asked questions
Does robots.txt hide a page?
No. It only asks crawlers not to fetch it. Use authentication or noindex for real protection.
Should I block CSS and JavaScript?
No. Crawlers need them to render the page correctly, and blocking them harms indexing.
Keep going
Related tools
Sitemap XML Generator
Build a valid sitemap.xml from a list of URLs.
Basic Website SEO Audit
A quick, honest health check of one page.
Meta Tag Generator
Generate title, description, Open Graph and Twitter tags.
Meta Tag Analyzer
Fetch a live page and audit its title, description and social tags.
Open Graph Preview
See how your link looks when shared on social platforms.
Twitter/X Card Preview
Preview and generate summary or large-image cards.