Check whether AI crawlers can reach your website
A site can publish a permissive robots.txt and still return 403 to every AI crawler, because the block lives at the CDN rather than in the file. Nothing in your analytics shows it and nothing in your browser reveals it. This tool requests one of your pages as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot and CCBot, then tells you the status each one got and which fix applies.
No account, no card. We request one page as each crawler and report what your server sends back.
The block you cannot see
Fulcru’s own site sat blocked for months. The robots.txt allowed every AI crawler by name. The content was there. And every one of them was getting a 403 from Cloudflare, which sits in front of the site and answers first. Nobody noticed, because there is nothing to notice: Google Analytics counts people, and a crawler is not a person. You can lose every AI engine at once and your dashboard stays flat and cheerful.
Three causes, in order of how often they are the real one
1. A CDN bot rule
Cloudflare’s AI Crawl Control has a “Block AI bots” control that applies to every page, and it is on by default for new zones. Bot Fight Mode and hand-written WAF rules do the same thing. The request never reaches your server, so no amount of editing robots.txt changes the answer. Fix it in the Cloudflare dashboard under AI Crawl Control.
2. A managed robots.txt
Some platforms serve their own robots.txt in place of the one your application generates, with AI crawlers disallowed. Your code says one thing and the file on the internet says another. Load your robots.txt in a browser and read what is actually served, not what you wrote.
3. Your own robots.txt
The honest case, and the easiest to fix. A Disallow line for GPTBot or an over-broad wildcard rule. Well-behaved crawlers obey it, so the page is served but never read.
Check it yourself, without any tool
One command per crawler. A 200 means you are readable, a 403 means you are not:
curl -A "GPTBot" -I https://yoursite.com
curl -A "ClaudeBot" -I https://yoursite.com
curl -A "PerplexityBot" -I https://yoursite.comThe tool above does the same thing across six crawlers, adds your robots.txt to the reading, and tells the two kinds of block apart. That is the only difference.
Being read is not being cited
Unblocking the crawlers gets you into the room. It does not get you named. Once the engines can read you, the question changes to whether they choose you when somebody asks for a recommendation in your category, and who they name instead when they do not. Measuring that is a separate job, and it is the one Fulcru does.
Questions
Why would my site block AI crawlers without me doing it?
The usual cause is a CDN setting rather than anything you wrote. Cloudflare has a "Block AI bots" control that is on by default for new zones, and a managed robots.txt that can replace the one your site serves. Security plugins on WordPress do something similar by rejecting unfamiliar user agents. All three return a 403 before the request reaches your site, so your own robots.txt looks fine and your analytics show nothing.
How is this different from reading my robots.txt?
A robots.txt reader only tells you what you asked crawlers to do. This makes a real request with each crawler's user agent and reports the status your server actually returns. That gap is the whole point: a permissive robots.txt and a 403 at the edge is the most common failure, and only a live request finds it.
Does a 403 for GPTBot mean ChatGPT cannot mention me?
It means OpenAI's crawler cannot read your pages, so nothing on your site can be quoted or cited. ChatGPT can still mention you from what other people wrote about you, and from what was in its training data before the block. But you have lost the part you control.
I unblocked them. How long until it matters?
Weeks, not days. The engines re-crawl on their own schedule and then have to re-index. Nothing improves the same afternoon, which is why it helps to record the date you unblocked and measure from there.
Should I allow AI crawlers at all?
That is a real choice, not an obvious one. Blocking them keeps your writing out of training sets; it also keeps you out of the answers people now ask instead of searching. Either is defensible. What is not defensible is having the decision made for you by a default setting you never saw.