Your WordPress site is getting visitors you never see in your analytics. GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider and a growing list of other AI crawlers are reading websites at enormous scale, and if your site is public, chances are good some of them are already requesting your pages.

That raises a question more and more site owners are asking us: should you let them in?

There is no single right answer, but there is a wrong way to decide, which is not deciding at all. Let's walk through what these bots actually do, the real tradeoffs, and how to enforce whatever you choose.

Not all AI bots are doing the same job

"AI bots" gets used as one bucket, but there are at least three different things happening, and they deserve different treatment.

Training crawlers collect content to train future AI models. OpenAI's GPTBot, Anthropic's ClaudeBot, and Common Crawl's CCBot fall into this group. Google-Extended is not a separate crawler at all: it is a robots.txt token that tells Google whether the pages Googlebot already fetched may be used to train and ground its Gemini AI models. Content collected this way may influence what future models know, but it does not send you traffic.

AI search crawlers index your content so it can be cited in AI-generated answers. OpenAI's OAI-SearchBot and PerplexityBot are examples. When ChatGPT or Perplexity cites your site as a source, that citation can send you actual visitors, the same way a search result does.

User-triggered fetchers like ChatGPT-User retrieve a specific page because a human asked an AI assistant about it right now. Blocking these means the assistant cannot read your page even when a real person requests it.

Blocking "all AI bots" cuts off all three. That is a legitimate choice, but you should make it knowing what you are giving up.

The case for blocking

Your content is your product. If you sell courses, premium research, or paid memberships, letting training crawlers ingest your material amounts to giving away the product to systems that may later answer questions with it, without attribution or payment.

Crawl load is real. AI crawlers are hungry, and some of the less polite ones hit sites hard. On a large content site, aggressive crawling is measurable server load that serves someone else's business.

Some bots do not behave. The well-known crawlers from major labs generally respect robots.txt. Plenty of smaller scrapers do not. A policy you cannot enforce is a suggestion, which is why edge-level enforcement matters (more on that below).

The case for allowing

AI search is becoming a discovery channel. A growing share of research now starts inside ChatGPT, Perplexity, Claude, and Google's AI experiences instead of a traditional results page. Sites those systems can read and cite get visibility there. Sites that block everything do not.

Blocking does not remove what is already learned. Training cutoffs being what they are, blocking today mostly affects future crawls. The strategic question is about the future: do you want to be present in AI-mediated discovery or absent from it?

For most businesses, being findable wins. If your site exists to attract customers, being the source an AI assistant cites is the modern version of ranking well. Most business sites have more to gain from visibility than they stand to lose from training exposure.

The middle path most sites should take

You do not have to choose between wide open and locked down. A sensible default for most WordPress sites is to allow well-behaved, verified crawlers and block everything else.

Cloudflare maintains a verified bots program: crawlers that identify themselves honestly, respect crawl rules, and behave. Allowing only those gives you the discovery upside of AI search while filtering out anonymous scrapers that would ignore your robots.txt anyway.

If your content is genuinely your paid product, tighten from there. If you run a lead-generation or publishing site, you probably want the reachability.

How to actually enforce your choice

robots.txt is the polite request. You can add directives for each crawler:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Reputable crawlers honor this. Disreputable ones do not, and there is no penalty for ignoring it. Treat robots.txt as necessary but not sufficient.

Edge enforcement is the locked door. Blocking at the CDN level means the request is stopped before it ever reaches your WordPress server, regardless of whether the bot respects robots.txt. It is the strongest form of blocking you can apply, and it has a side benefit: refused requests cost your server nothing.

On Levamo, this is a built-in control. Sites fronted by our managed Cloudflare setup get an Allow AI Bots setting right in the dashboard, with three modes: allow all AI crawlers, allow only Cloudflare-approved AI crawlers, or block all AI crawlers. It is enforced at the edge through the Enterprise Cloudflare CDN, so the policy applies to bots whether or not they play by robots.txt rules. Changing your mind later is one click, not a config project.

Whatever you choose, be consistent

One last piece of advice: align your signals. We have seen sites block every AI crawler at the edge while simultaneously trying to optimize for AI search visibility. Those goals point in opposite directions.

If you decide AI systems should be able to read and cite your site, consider going a step further and publishing an llms.txt file, a curated map of your site written specifically for AI tools. And if you decide they should not, enforce it at the edge where the decision actually sticks.

The bots are coming either way. The only question is whether you have a policy or a default.