Cloudflare Sets September 15 Permission Rules For AI Agent Crawlers

AI News outlines new permission settings from September 15 that will place real-time AI agents behind added controls.

Agent and Training traffic will be blocked by default on ad-supported pages for several customer groups, while Search remains allowed.

On July 1, Cloudflare introduced the control changes, AI News noted.

The platform now separates crawler behaviour into Search, Agent and Training categories instead of treating all AI bot activity as one setting.

Cloudflare Separates Search, Agent And Training Crawlers

Search covers bots that index pages so a service can answer questions about the page later.

Agent covers automated systems that fetch pages in real time for a user, including ChatGPT fetch activity and browser-driving agents.

Training covers crawlers that collect web content for model training.

Those July 1 controls applied across Cloudflare customers, including the free tier, AI News wrote.

For September 15, the default settings cover domains newly joining Cloudflare, new sites created by existing customers and existing free-tier customers, according to AI News.

Under those defaults, Search remains allowed.

Training and Agent traffic are blocked on pages that display ads unless the customer changes the relevant security setting before the deadline.

Ad-Supported Pages Get The Default Block

AI News describes advertising as evidence that a page was built for a human visitor.

A search crawler can still send a reader back to the site, while an agent can read a page and return an answer without a visit.

Many useful pages for research, monitoring, pricing checks, product specifications and news sit behind ad-supported publishing models.

If an agent loses access, the failure may appear as missing coverage rather than a clean error.

Matthew Prince, CEO of Cloudflare, said the company hopes the rules will encourage crawlers that combine search, agent use and training to separate those uses, AI News noted.

That classification is behavioural, not a box that an agent operator chooses in Cloudflare's dashboard.

A real-time research agent can therefore be treated as Agent traffic even if its operator describes it as a browsing or monitoring tool.

Pay Per Use Becomes The Access Route

Cloudflare is moving Pay Per Crawl towards Pay Per Use, according to AI News.

Ceramic.ai pays publishers when their content appears in AI search results, and You.com pays when an agent reaches premium content.

Cloudflare says more than half of AI crawler traffic is spent fetching pages that have not changed.

The payment model was presented as a way to price access rather than relying only on blocking.

Publishers also face a trade-off under the rules described by AI News.

Blocking Training traffic can affect Googlebot when a crawler combines search and training behaviour, according to AI News.

Free-tier customers that do not want the new defaults must change settings before September 15, AI News added.

Search, Agent and Training depend on declared crawler behaviour.

The remaining public-record gaps are a specific verification method, enforcement process and penalty that would prevent a crawler operator from avoiding the Training label when that classification would trigger a block.