Decide which AI bots can read your website. Check which AI crawlers a site currently allows or blocks, then build robots.txt rules that keep you visible in AI search while opting out of AI training, or whatever mix you prefer.
See which AI crawlers a site's robots.txt allows right now.
| Crawler | Company | Used for | Access |
|---|
Start from a preset, then fine-tune each crawler.
Add these lines to the top of your existing robots.txt file. Bots you allow aren't listed, so they keep following your normal rules.
Training crawlers like GPTBot and ClaudeBot collect content to build future AI models. Search crawlers like OAI-SearchBot and PerplexityBot index pages so AI assistants can find and cite them in answers. You can block one type and allow the other.
It blocks the crawlers used for AI training but allows the ones used for AI search and user requests. Your business can still be found and cited in ChatGPT, Claude and Perplexity answers, without your content going into model training.
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google Search, including AI Overviews, uses the regular Googlebot, which these rules don't touch.
If a bot has its own section in robots.txt, it ignores your general rules for all crawlers. Leaving allowed bots out means they keep following your normal rules, such as staying out of admin or checkout pages.
The major companies listed here say their crawlers follow robots.txt. Some agents that act on a user's behalf, and less reputable scrapers, may not. robots.txt is a request, not a lock.
Open your existing robots.txt file (yourdomain.com/robots.txt), paste the rules at the top, and save. On WordPress, SEO plugins like Yoast and Rank Math have a robots.txt editor. The Download button gives you a complete file with your current rules included.
