How to Make Sure AI Bots Can Read Your Website
Part 3 in The #1 Reason your website isn't getting cited by AI (probably), and what to do about it. Also in this section: What Smart People Miss.
The short version
The check has five steps and takes about 15 minutes.
You're looking in three places: your robots.txt file, your website platform's settings, and your firewall.
You don't need to be technical. If your menus look different, a chatbot or a web expert can help.
Never use "hide my site from search engines" to block AI. It hides you from Google too.
Time to read: 4 minutes.
Skill level: none.
Evidence: solid. Bot roles are documented by each AI company.
Note: This is the hands-on part of the article called The #1 Reason Your Website Isn't Getting Cited (Probably). If you haven't yet, read Part 2 first. It explains why you should never block "all the AI bots," which matters in Step 4.
The 5 steps
Step 1: Read your robots.txt
Type your address plus /robots.txt into your browser (for example, yoursite.com/robots.txt). You don't need to understand all of it.
Look for these:
User-agent: *
Disallow: /
This blocks every bot from your whole site, including Google. Fine for a site under construction, a disaster on a live one.
User-agent: OAI-SearchBot
Disallow: /
Any Disallow: / under a named AI bot shuts that bot out. Pay special attention to the search bots, not just the training ones.
If there are no AI-bot rules at all, that's usually fine. Most crawlers are allowed by default.
Step 2: Check your platform's setting
Your website builder may have a switch that blocks bots without touching robots.txt. Find your platform in the guide below.
Step 3: Ask about the firewall
Ask whoever manages your hosting or domain: "Do we use Cloudflare or another firewall? Is any AI-bot blocking turned on?" If you don't know who manages this, that's worth finding out on its own.
Step 4: Decide your policy on purpose
Some businesses want to be cited but don't want their content used for training.
That's a legitimate choice: allow the search and user bots, block the training bots. If you'd rather keep it simple, allow everything. Part 2 compares the three sensible policies.
Step 5: Test it
Paste one of your page URLs into ChatGPT or Perplexity and ask it to summarize the page. If it can't, that's a hint something may be blocking it. It isn't proof, since AI tools fail to fetch pages for other reasons too. For a definitive answer, ask your host for a server-log check on the bot names (in Part 2).
Where to find it on your platform
Checked against each platform's help pages in September 2026. Menu names change often, so if yours looks different, see "Can't find it?" below.
Applies to every platform: don't use "hide my site from search engines" to block AI bots. It hides you from Google and Bing too.
WordPress
This one is from general knowledge, since WordPress setups vary widely. Check it against your own screen.
Site-wide switch: Settings, then Reading. Make sure "Discourage search engines from indexing this site" is unchecked.
Robots.txt: SEO plugins usually manage it. In Yoast, look under Tools, then File editor. In Rank Math, General Settings, then Edit robots.txt.
Also check your security plugins and your hosting dashboard's bot protection.
Wix
Robots.txt: go to SEO & GEO in your dashboard, scroll to Tools and settings, and open the Robots.txt Editor. Wix warns that editing this file can cause pages to stop appearing in search results.
One page: in the editor, open Pages & Menu, click More Actions next to the page, choose SEO basics, and check the "Let search engines index this page" toggle.
Watch for: preventing indexing for the entire site overrides the group and page-level options.
Webflow
Where: Site settings, then SEO, then Indexing.
What's there: Traffic control toggles that auto-generate robots.txt rules for groups of known bots. Webflow itself warns that blocking search or AI bots may reduce your visibility in traditional and AI-powered search.
Watch for: a page-level sitemap indexing toggle that adds a noindex tag when turned off.
Squarespace
Where: Settings, then Crawlers.
What to check: uncheck "Block known artificial intelligence crawlers" if you want to appear in AI answers. Also make sure the search engine crawler block is off.
Honest limit: one source notes that there's currently no way to block AI training while still allowing AI-powered search from the same company. On Squarespace it's all or nothing.
Shopify
Good news: all Shopify stores have a default robots.txt that's optimized for SEO, and network-level bot management is handled by Shopify. Many stores never need to touch it.
If it's been customized: Online Store, Themes, Actions, Edit code, then look for robots.txt.liquid in the Templates folder.
Watch for: Shopify doesn't recommend putting a proxy in front of a Shopify store and can't support it.
GoDaddy Websites + Marketing
I couldn't find a documented AI-bot setting for GoDaddy's builder. Look in your site's SEO or visibility settings for a "hide from search engines" option, then view yoursite.com/robots.txt to see what's actually being served. If something looks wrong, ask GoDaddy support. If your site is WordPress hosted at GoDaddy, use the WordPress steps above.
Cloudflare (or any firewall in front of your site)
The door people forget.
Where: Cloudflare dashboard, select your domain, then Security, then Bots, and look for AI Crawl Control. It shows which AI services access your content and lets you set allow or block rules for individual crawlers.
What to check: whether a blanket "block AI bots" setting is on. The sensible default is to block training crawlers and allow search crawlers.
Heads up: Cloudflare has reworked these controls several times. One source reports they were split into Search, Agent and Training categories in July 2026. Verify the labels on your own screen.
Can't find it? Three honest options
Screens change every few months, and there are dozens of builders (Framer, Weebly, Hostinger, Duda, and more).
Option 1: Ask a chatbot. ChatGPT, Claude or Gemini can walk you through it if you give it the right details. Try this:
"I have a website on [platform name]. I want AI search bots like OAI-SearchBot, PerplexityBot and ClaudeBot to be able to read my site, and I'm okay blocking training-only bots like GPTBot. Where in [platform name] do I check whether any of them are blocked? Walk me through it step by step, and tell me what to look for."
Paste in your robots.txt contents too and ask it to flag anything blocking those bots. Chatbots can be out of date on menu names, so if a button doesn't exist, tell it what you see on your screen and ask again.
Option 2: Ask an expert. If your site sits behind a firewall, runs on custom hosting, or was built by someone you can't reach, a 30-minute check by a web developer or AI visibility specialist is faster and safer than guessing. A wrong robots.txt edit can hide you from Google too. Ask them to confirm three things: robots.txt, the platform switch, and the firewall layer.
Option 3: Let us check it for you. Bot access is the first thing we test in our AI Report Card, along with the other nine reasons in this series. You get a clear pass or fail on each door and a plan for anything that needs fixing.
You'll know it worked when
Your robots.txt has no Disallow: / under * or under any search bot you want
Your platform's "hide from search engines" setting is off
Your firewall isn't blocking AI search and user bots, and you can say who decided that
A fair warning
Fixing this removes an obstacle. It doesn't guarantee a citation. The door is unlocked, but bots still need a reason to recommend you, and that's what reasons #2 through #10 are about.
A block also mostly limits what engines can pull live. AI tools may still know things about you from older training data or from other sites that mention you.
Want to know where you stand as a whole? The AI Report Card gives you the full picture for an introductory price of $497: a visibility score, a check of bot access and the other technical signals in this series, the customer questions you're losing, and a plan for what to fix first. Get your AI Report Card
Next: Reason #2, Google and Bing may not fully know your website exists (coming soon).
Comments