The mistake that catches smart people: not all AI bots are the same
Part 2 of 3 in the series: The #1 Reason your website isn't getting cited by AI (probably)
Skip the next section: How to fix it.
The short version
"AI bot" isn't one thing. Some bots train AI, some build the search index that AI answers come from, and some fetch a page live when someone asks.
Blocking the search bots can remove you from AI answers, even if you only meant to stop training.
"Block all the AI bots" is not a safe instruction.
Pick a policy on purpose. There are three sensible ones.
Time to read: 4 minutes.
Skill level: none.
Evidence: solid. Bot roles are documented by each AI company.
In Part 1, we covered the three doors that can be locked. Before you unlock anything, you need to know this: "AI bot" is not one thing. Treating it as one thing is how careful, protective site owners end up invisible.
Three kinds of AI bots
Each does a different job, and each can be allowed or blocked on its own.
Training bots collect content to teach future models. Examples: GPTBot, ClaudeBot.
Search bots build the index AI tools pull answers from. Examples: OAI-SearchBot, PerplexityBot, Claude-SearchBot.
User bots fetch a page live when someone asks about it. Examples: ChatGPT-User, Perplexity-User, Claude-User.
Bot names and roles change. Check each company's own documentation before you write any rules.
Why the mix-up costs visibility
OpenAI's agents can be allowed or blocked independently, so a business can opt out of training while staying citable in ChatGPT search. The costly mistake runs the other way: blocking the search bot removes a site from ChatGPT's search citations, even though training was never the concern.
The same pattern shows up with every AI company. Someone reads a headline about AI scraping, adds a rule to block "the AI bots," and the rule catches the bots that would have recommended them.
One more trap: Google-Extended appears on many "AI bot lists," but it isn't a crawler at all, just a robots.txt switch for AI training use, separate from Google Search.
Bottom line: "block all the AI bots" is not a safe instruction. It can quietly delete you from the answers you want to be in.
Which bots should you allow?
There's no single right answer, but there are three sensible policies.
Pick one on purpose.
Allow everything. Best for most small businesses that want to be found. Keep robots.txt open and make sure your firewall isn't blocking AI bots.
Be cited, opt out of training. Best for businesses with original content they don't want used to train models. Allow the search and user bots. Block the training bots.
Block everything. Rarely right for a business that wants customers. Only do it if you truly don't want AI visibility, and expect to fall out of AI answers.
Two honest limits. First, robots.txt is a request, not a wall. Well-behaved bots follow it, but some crawlers ignore it, so only a firewall rule guarantees a block. Second, blocking now doesn't remove anything a model has already collected. It only affects what bots can read from here on.
Also, some platforms only give you an all-or-nothing switch. Squarespace's AI crawler setting is one example, per one source. If you want the middle policy there, you'd need a firewall or a robots.txt editor that lets you name individual bots.
What to do next
Now you know what you're looking for. Part 3 is the hands-on check: five steps, about 15 minutes, with exactly where to click on the most popular website platforms.
Not sure which policy fits? We set it out for you in the AI Report Card, along with a check of every bot rule on your site. See what's included
\
Comments