Robots.txt Generator
Guide: How to Write a robots.txt File (With Examples)
Save this as robots.txt in the root folder of your site, so it loads at yourdomain.com/robots.txt.
Nothing you enter or choose here leaves your device. This tool runs in your browser.
How to use
- 1Choose a crawler and list the paths to block or allow.
- 2Add your sitemap address.
- 3Copy the file to your site's root folder.
Frequently asked questions
Where does robots.txt go?
In the root folder, so it loads at yourdomain.com/robots.txt.
Does it hide pages from search?
No. It asks well-behaved crawlers not to visit. Use a noindex tag to keep a page out of results.
How do I block AI crawlers?
Add a rule for the crawler, such as GPTBot or CCBot, and block /.
What does * mean?
It means every crawler.
Is my data uploaded?
No. Everything runs in your browser, so what you paste never leaves your device.
Read the guide
Learn how this works and how to get the most from it.
Web & SEO
How to Write a robots.txt File (With Examples)
What robots.txt does and doesn't do, how Disallow and Allow rules work, how to add your sitemap, and mistakes that can hide your whole site.
1 min read
Related tools
Meta Tag Generator
Generate SEO, Open Graph and Twitter Card meta tags with live previews.
.htaccess Generator
Redirects, HTTPS, caching and more for Apache sites.
UTM Link Builder
Add campaign tracking to any link.
URL Parser & Query Builder
Split a URL into its parts and edit its query parameters.