How to Write a robots.txt File (With Examples)

A robots.txt file is a plain text file at the root of your site that tells crawlers which parts they may visit. Well-behaved crawlers, such as search engines, read it first. It is a request, not a lock, so it doesn't stop anyone determined.

The basic format

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

• User-agent names the crawler. * means all of them.
• Disallow lists a path crawlers shouldn't visit.
• Allow makes an exception inside a blocked folder.
• Sitemap points to your sitemap with a full address.

Common examples

• Allow everything: User-agent: * followed by an empty Disallow: line.
• Block everything: Disallow: /. Use this only on a staging site.
• Block one crawler: add a separate group with its name, such as User-agent: GPTBot, then Disallow: /.

What robots.txt doesn't do

Blocking a page doesn't remove it from search results. If other sites link to it, the address can still appear without a description. To keep a page out of results, leave it crawlable and add a noindex meta tag, or protect it with a password. Also, don't list secret folders in robots.txt, because the file is public and anyone can read it.

Mistakes to avoid

• Leaving Disallow: / in place after launching a site. This is the most common reason a new site never appears in search.
• Blocking CSS and JavaScript folders, which stops search engines from rendering your pages properly.
• Putting the file in a subfolder. It must load at yourdomain.com/robots.txt.
• Expecting every crawler to obey Crawl-delay. Google ignores it, though some other crawlers honor it.

Build one in a minute

The Robots.txt Generator lets you add rules for each crawler and your sitemap, then copy the result. For page-level control, see the Meta Tag Generator. If your site runs on Apache, the .htaccess Generator can add redirects and password-style blocks.