Melbourne, Australia.

[email protected]
GuideControl1 July 2026

Should AI bots read your site?

If your website exists to bring you customers, let the crawlers that answer people's questions read it, and decide separately about the ones that only collect text to train models. Since 1 July 2025 that is a real choice. Cloudflare, which says it handles traffic for about 20% of the web, changed its default to block AI crawlers that have no permission, now asks every new domain whether to allow them, and opened a private beta that lets sites charge crawlers for access (Cloudflare, 1 July 2025).

The cost of getting it wrong is quiet. Block everything and you can drop out of the answers people now get from chat tools and search summaries. Allow everything and your guides and pages feed products that never send anyone back.

Which crawlers visit your site?

Two kinds matter. Training crawlers copy pages to build future models. Search and answer crawlers fetch a page to answer a live question, and often cite and link to it. Most identify themselves by name, and your hosting or Cloudflare logs show which ones visit and how often.

Search engines' own crawlers are a separate case again. Blocking Googlebot or Bingbot takes you out of ordinary search.

Should you block them or let them in?

It depends on what the site is for.

If the content is the product (paid research, a publication, material you sell), blocking or charging protects what you sell.

If the site is a shopfront, being quoted in an answer is close to free advertising. You want the answer and search crawlers in, so that when someone asks who does this work nearby, your business can be in the reply.

Most service businesses land in the middle: allow search and answer crawlers, and decide about the training ones on purpose.

How do you control it?

Two levers. A robots.txt file at the root of your site lists which crawlers may read what. It is a request, and well-behaved crawlers follow it. A network rule, from a service like Cloudflare or your host, can actually block or meter the traffic.

What goes wrong?

Setting it once and forgetting it. Crawlers change names, new ones appear, and a default you never chose can quietly hide your site. Put a ten-minute look at the logs and rules on the calendar each quarter.

One step for this week

Ask whoever runs your website one question: which crawlers are we blocking right now, and did anyone decide that? If nobody knows, that is the answer. A one-page policy for the tools your team uses starts from the same "who decided?" question.

Being found is also part of marketing that learns, one of the eight things we build. See it in section 02.

OBSERVE.

ORCHESTRATE.

OUTPERFORM.

Every system we build starts with one call.