Home › Features

Choose which AI assistants may read your site

The four switches

Every site we build has a robots file that names the AI assistants' crawlers one by one. Each has its own choice, allow or block:

• ChatGPT's crawler (the one that reads sites for ChatGPT).
• ChatGPT fetching a page a person asked about, which is a separate request from the crawler above.
• Perplexity's crawler.
• Claude's crawler.

Google's AI-training choice is a fifth switch that sits beside these, and it has its own page: choose whether Google trains its AI on your content. It is separate on purpose, because blocking training is not the same as blocking an assistant that reads a page to answer a question.

Allowed by default, and what that means

Each assistant is allowed unless you say otherwise, so assistants can find your pages and quote them when someone asks a question your site answers. Only a block is stored. A site where you have changed nothing keeps exactly the robots file it had before, and no existing site changes until its owner changes a choice.

The choice is yours because it is your content. An agency whose client does not want their pages read by a particular assistant can block that one and leave the others alone.

What blocking does, and what it does not

Blocking adds a rule to your site's robots file asking that assistant's crawler not to read your pages. It is a request that well-behaved crawlers follow. It does not recall anything an assistant has already read, and it cannot force a crawler that ignores robots files.

These four switches do not change how search engines index your site. The robots file keeps its other rules, and the page on robots and indexing describes them.

How the readiness check treats it

The readiness score includes a line that notes whether the main assistants' crawlers are explicitly welcomed in your robots file. If you block one of them, that line will not be marked as passed. Blocking is your decision to make, and the page on the readiness score and the gate behind it explains how the score is built.

The publish check also looks for an llms.txt file, a plain summary of your business written for AI assistants, so an assistant that is allowed to read your site has something short and accurate to read first.

Industry guidance on being understood by AI assistants

Your industry playbook covers AI search: what AI assistants look for when they answer questions about a business like yours, which facts to state plainly on your pages, and the claims to avoid in your field. You can ask the assistant about it in the conversation. The page on asking the assistant describes what it can do.

What it does not do

• It does not promise that any assistant will mention or quote your site.
• It cannot make every crawler follow a robots file.
• It does not change what visitors see, and it does not hide a page from people.
• It does not affect how search engines index your pages.

If you would like a switch for another assistant, send a feature request; a real person reads each one. The page on support and feature requests explains how.

Updated October 2026