robots.txt Generator
Generate a robots.txt file for your website. Control search engine crawling with allow/disallow rules.
robots.txt Content
How to Use
Set the user-agent, add allow/disallow paths, optionally add your sitemap URL and crawl delay. The robots.txt content generates in real-time.
Frequently Asked Questions
robots.txt is a text file placed at your website's root that tells search engine crawlers which pages to crawl and which to skip. It's part of the Robots Exclusion Protocol.
No, robots.txt only prevents crawling, not indexing. Pages can still appear in search results if other pages link to them. Use meta robots noindex to prevent indexing.
robots.txt must be placed at the root of your domain: https://example.com/robots.txt. It won't work in subdirectories.