Free tools

Check whether your robots.txt blocks AI crawlers

A robots.txt can look permissive and still stop an AI crawler, because the rule that applies is not always the rule you are reading. This tool fetches your file, picks the one group each crawler obeys, and names the exact line that decides for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot.

No account, no card. We fetch your robots.txt once and show which line decides for each AI crawler.

A crawler obeys one group and ignores the rest

This is the rule that catches almost everyone. A crawler looks for the group that names it. If it finds one, that group is the only thing it reads, and every rule under User-agent: * stops applying to it completely. Not merged. Not layered. Ignored.

So this file, which looks stricter than it is, lets GPTBot into your admin pages:

User-agent: *
Disallow: /admin
Disallow: /checkout

User-agent: GPTBot
Allow: /blog

The author meant “GPTBot may also read the blog.” What they wrote is “GPTBot ignores the admin and checkout rules.” If a rule should apply to a named crawler, it has to be repeated inside that crawler’s own group.

Longest match wins, not first match

Inside a group, order does not matter. The rule with the longest matching path pattern decides, and when an Allow and a Disallow tie on length, the Allow wins. That is why Disallow: / followed by Allow: /blog does work: on a blog URL the Allow is longer, so it takes it.

Google-Extended is not a crawler

It fetches nothing. Googlebot does the fetching, and the Google-Extended rule decides whether what it read may be used for Gemini and AI Overviews. The practical effect is that disallowing it costs you AI answers while leaving your normal search results untouched, which makes it the one line on this list people add deliberately and then forget.

Check it yourself, without any tool

Read what is actually served, not what is in your repository. Managed platforms sometimes replace the file:

curl https://yoursite.com/robots.txt

The tool above does the same fetch, then does the group selection and the longest-match arithmetic for six crawlers so you do not have to hold the rules in your head.

A clean file does not mean you are readable

robots.txt is a request, and it is only half the picture. A CDN can return 403 to the same crawler your file politely welcomes, and nothing in the file reveals it. Our own site sat in exactly that state for months. The AI Crawler Check makes a real request as each crawler and catches that gap. Run it after this one.

Get the AI-answer teardown, weekly

One email a week: who AI names in a category, why, and the page that changed it. No spam, unsubscribe anytime.

Questions

How is this different from the AI Crawler Check?

They answer two halves of the same question. This one reads your file and tells you what you asked crawlers to do, including which of several competing rules actually wins for each bot. The AI Crawler Check makes a real request and tells you what your server does. A file can be perfect and the CDN can still return 403, so run both.

Which rule wins when several match?

The standard says the most specific user-agent group wins, and within that group the longest matching path rule wins. So a group naming GPTBot beats the wildcard group entirely, even if the wildcard group is stricter. That is the part people get wrong most: adding a friendly GPTBot group makes the crawler ignore every rule in your star group, including the ones you wanted it to keep.

I never wrote a rule about AI crawlers. Can I still be blocking them?

Yes, and it is the common case. A wildcard group with Disallow: / blocks every crawler that has no group of its own. Plenty of sites carry that line from a staging setup and never revisited it.

Does Google-Extended work like the others?

Not quite. It is a robots.txt token rather than a crawler that fetches anything. Googlebot does the fetching, and the Google-Extended rule decides whether what it read may be used for Gemini and AI Overviews. So disallowing it costs you AI answers while leaving normal search alone.

Should I list a sitemap in robots.txt?

It is worth the one line. The Sitemap directive is group-independent, so it applies no matter which user-agent is reading, and it is the cheapest way to hand a crawler a full list of your pages instead of making it find them.

The other free tools