Robots.txt Generator

Build a correct robots.txt in a minute: choose what to block, add your sitemap, and decide which AI crawlers can read your site.

Search engines (all crawlers)
Block AI crawlers

robots.txt

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/

Upload it to the root of your site so it loads at yourdomain.com/robots.txt. Google ignores crawl-delay; Bing respects it, but most sites don't need it.

What robots.txt does

robots.txt is a plain text file that tells crawlers which parts of your site they may visit. Search engines check it before crawling. It's useful for keeping crawlers out of areas that don't belong in search, like admin screens, shopping carts, internal search results and endless filter combinations.

A safe setup for most small business sites

  • Allow everything by default.
  • Block admin and checkout areas (WordPress sites usually block /wp-admin/ but allow admin-ajax.php).
  • List your sitemap so crawlers find every page.
  • Don't block CSS or JavaScript files. Google needs them to see your pages like a visitor does.

AI crawlers explained

AI companies run two kinds of crawlers. Training crawlers (such as GPTBot, ClaudeBot, Google-Extended and CCBot) collect content that may be used to build AI models. Search and user crawlers (such as OAI-SearchBot, ChatGPT-User and PerplexityBot) fetch pages so AI assistants can answer questions and link to sources. If you want ChatGPT or Perplexity to recommend your business, don't block their search crawlers.

Test before you trust it

After uploading, open yourdomain.com/robots.txt in a browser to confirm it loads, then check Google Search Console's robots.txt report for errors. Want to see what a crawler sees on a page? Try the spider simulator.

Questions

Does robots.txt keep a page out of Google?

Not reliably. It stops crawling, but Google can still index a blocked URL if other sites link to it. To keep a page out of search results, use a noindex tag and let Google crawl it.

Should I block AI crawlers?

It's a business choice. Blocking training crawlers like GPTBot or Google-Extended keeps your content out of future model training. Blocking search crawlers like OAI-SearchBot or PerplexityBot can keep you out of AI answers that send customers your way. Most local businesses want to be found, so they leave search crawlers allowed.

Do all bots obey robots.txt?

Reputable crawlers from Google, Bing, OpenAI, Anthropic and others say they follow it. Bad bots ignore it, so it isn't a security tool. Never use it to hide private information.

Where does the file go?

At the root of your domain: https://yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored.

Can a mistake hurt my rankings?

Yes. A stray 'Disallow: /' blocks your whole site. After uploading, check it in Google Search Console's robots.txt report.

More free tools

See all tools