SendTech Times
AINews|July 2, 2026, 05:57 AM
SYSTEMS SHIFT:

Cloudflare Sets September Deadline For Mixed-Use AI Crawlers

BySendTech Times AI & Enterprise DeskNewsroom-edited, source-reviewed coverage
Newsroom brief

Cloudflare plans to block mixed-use crawlers from ad-supported pages by default from September 15, 2026, unless site owners change the setting. The policy pushes AI companies to separate search access from agent and training uses while Cloudflare expands publisher payment tools.

Verified against source materialEdited by SendTech Times AI & Enterprise DeskSource: TechCrunch
Cloudflare Sets September Deadline For Mixed-Use AI Crawlers
Image source: TechCrunch

Cloudflare Sets A September 15 Crawler Deadline

Cloudflare plans to change its default crawler rules on September 15, 2026, targeting bots that combine traditional search crawling with AI agent activity or model-training access.

Under the policy, mixed-use crawlers will be blocked by default from pages that host ads unless the site owner chooses another setting.

Cloudflare said the change will apply to new customers, new sites created by existing customers and all existing free customers.

The rule gives publishers a sharper control point over AI access to web pages.

It also forces AI companies to make clearer distinctions between crawlers that support search discovery and crawlers that support training or agent services.

Cloudflare framed the change around publisher control rather than a blanket block on AI.

The company said website owners often want search discovery and some AI visibility, but they also want protection against content being used without payment.

Pay Per Use Expands The Publisher Payment Model

Cloudflare is also expanding its publisher-payment model from Pay Per Crawl to Pay Per Use.

The company said the newer model would let publishers charge AI companies when content creates value, not only when a crawler fetches it.

Cloudflare said Ceramic.ai and You.com are its first partners for the approach.

When a publisher opts in, Cloudflare said payment can occur when content appears in Ceramic's AI search results or when You.com accesses premium content.

The company also cited efficiency as part of the policy.

Cloudflare said its data suggested that more than 50% of AI crawler traffic is spent fetching pages that have not changed, adding bandwidth and compute costs for publishers and model providers.

Google Boundary Remains A Search Dispute

Cloudflare's announcement also challenges how large search operators separate discovery from AI use.

The company said the world's largest search engine has access to about 2x more information than other AI companies because publishers find it difficult to stay discoverable without also being used for AI.

Google has previously pointed to Google Extended, a crawler control that lets site owners opt out of training and AI product uses such as Gemini Apps and Vertex API without affecting inclusion in Google Search.

Googlebot still crawls for Search, including AI Overviews and AI Mode.

Cloudflare co-founder and CEO Matthew Prince said most internet traffic is now non-human and said clearer bot intent could support a more sustainable web ecosystem.

The public record still lacks pricing for Pay Per Use, publisher revenue terms, or which other AI companies will adopt the model.

Share this article
inXf

Related articles

More
Microsoft’s Student AI Privacy Deal Pressures Google On School Data Rules
AI

Microsoft’s Student AI Privacy Deal Pressures Google On School Data Rules

Microsoft’s legally binding school AI agreement with the AFT bars most student-data training uses and adds audits, but its policy weight depends on whether Google and other vendors follow.

RAND Superintelligence Plan Pushes U.S. AI Options Beyond Acceleration
AI

RAND Superintelligence Plan Pushes U.S. AI Options Beyond Acceleration

Import AI highlighted a RAND strategy document urging the United States to spend now on AI safety, security architecture, institutional capacity and response plans so it can keep multiple superintelligence policy options open.

AI Race With China Turns On Trust, Not Just Models
AI

AI Race With China Turns On Trust, Not Just Models

Rest of World essayists Rumman Chowdhury and Konstantinos Komaitis argued that U.S. AI policy is focusing too narrowly on model performance while public trust, consumer protection and infrastructure resistance become deployment constraints.

Trump’s AI Force Plan Leaves Mandate Unclear as Safety Fight Builds
AI

Trump’s AI Force Plan Leaves Mandate Unclear as Safety Fight Builds

SiliconANGLE reported that Trump plans an AI Force and a new AI czar while rejecting slowdown calls, leaving the body’s powers, budget and agency role unresolved.

Pingtar Puts Southeast Asia’s Frontline HR Gap on Messaging Rails
AI

Pingtar Puts Southeast Asia’s Frontline HR Gap on Messaging Rails

A Tech Collective Southeast Asia interview with Pingtar CEO Arvinda Tripradopo details how the self-funded platform uses WhatsApp-style access, anonymous diagnostics and conversational AI to reach frontline workers often missed by conventional HR systems.

OpenAI And Anthropic Chase Big Tech Asia Operators For Market Access
AI

OpenAI And Anthropic Chase Big Tech Asia Operators For Market Access

OpenAI and Anthropic are hiring senior Asia executives from U.S. big tech firms as AI competition shifts from model building toward sales, policy and local market execution.

Natrona Schools AI Review Puts Student Privacy Ahead Of Classroom Tool Use
AI

Natrona Schools AI Review Puts Student Privacy Ahead Of Classroom Tool Use

Natrona County trustees questioned whether teacher AI tools expose student data, even as existing district rules already ban unauthorized generative AI use by students.

Pentagon’s $30.3 Million AI Lie Detector Plan Tests Polygraph Limits
AI

Pentagon’s $30.3 Million AI Lie Detector Plan Tests Polygraph Limits

A Pentagon budget request would fund Polygraph+, a five-year AI and standoff-sensing lie detection effort, despite a long record of scientific doubt about polygraph screening.

Keep Reading

More Stories

Latest
Universal Quantum Raises $100 Million for Trapped-Ion Computing ExpansionChips & SemiconductorsOct 10, 2026Universal Quantum Raises $100 Million for Trapped-Ion Computing ExpansionUniversal Quantum raised more than $100 million in a Series A round to commercialise trapped-ion quantum products and expand in Singapore, the US, Japan and Germany.Neela Raises £2.1M To Pilot Waste-To-SAF BiotechEconomyOct 10, 2026Neela Raises £2.1M To Pilot Waste-To-SAF BiotechCambridge startup Neela Biotech raised £2.1m to move its AI-guided microbial waste-to-SAF process from lab work into pilot trials at an existing biogas plant over the next two years.India and Saudi Arabia Put Port Investment Talks on Strategic Maritime RouteEconomyOct 9, 2026India and Saudi Arabia Put Port Investment Talks on Strategic Maritime RouteIndia and Saudi Arabia discussed possible Saudi participation in Vadhavan and Galathea Bay port projects, linking Gulf trade, container transshipment and maritime capacity building.MIT Expands STEM Workforce Training Through AI And Design ProgramsAIOct 9, 2026MIT Expands STEM Workforce Training Through AI And Design ProgramsMIT for America will scale existing university education programs nationwide, combining calculus support, responsible AI resources and hands-on fabrication to address STEM workforce access gaps.Google Maps Expands Restaurant Search Into Food OrderingCapital & PolicyOct 9, 2026Google Maps Expands Restaurant Search Into Food OrderingGoogle Maps is adding more dining workflow features, with Gemini-powered Ask Maps ordering links through Toast, Square and Uber Eats plus city trend lists and practical restaurant review signals.Lumio Raises $12 Million To Expand Its Consumer-Electronics PortfolioChips & SemiconductorsOct 9, 2026Lumio Raises $12 Million To Expand Its Consumer-Electronics PortfolioLumio raised a $12 million Series A led by Blume Ventures to add products, software, offline retail and support capacity for its smart TV, projector and speaker business.Apple Touchscreen MacBook Report Points To Late-October Hardware SlateChips & SemiconductorsOct 9, 2026Apple Touchscreen MacBook Report Points To Late-October Hardware SlateApple is reportedly preparing a late-October touchscreen MacBook debut alongside an OLED iPad Mini and M6 updates for the entry-level MacBook Pro and iMac.Australia Health Data Modernisation Reaches $32.8 Million on Google CloudCloud & Data CentersOct 9, 2026Australia Health Data Modernisation Reaches $32.8 Million on Google CloudAustralia’s Health department has lifted disclosed spending on a Google Cloud data and analytics modernisation program to $32.8 million across Accenture, NTT, Deloitte and Google contracts.Kioxia LD4 E1.L SSD Targets Dense 1U Hyperscale StorageCloud & Data CentersOct 9, 2026Kioxia LD4 E1.L SSD Targets Dense 1U Hyperscale StorageKioxia’s LD4 Series brings BiCS FLASH generation 8 QLC NAND into the E1.L form factor, starting with 15.36TB and 30.72TB models for read-intensive hyperscale servers while the architecture is validated up to 122.88TB.NeoFleet Raises $4 Million to Expand Taxi-Fleet FinancingReal EstateOct 9, 2026NeoFleet Raises $4 Million to Expand Taxi-Fleet FinancingNeoFleet Capital secured a $4 million pre-seed round to scale a taxi-fleet financing model that combines vehicle credit, telematics, insurance and operating controls across Africa and other emerging markets.Brahma AI Raises US$150 Million Toward Digital-Human LaunchAIOct 8, 2026Brahma AI Raises US$150 Million Toward Digital-Human LaunchBrahma AI raised US$150 million through preferred shares, including US$100 million from Multiples Alternate Asset Management, to expand an enterprise AI content platform and prepare interactive digital humans.OKX Draws Circle and Ripple Backing at $25 Billion ValuationFintech & Digital PaymentsOct 8, 2026OKX Draws Circle and Ripple Backing at $25 Billion ValuationOKX took undisclosed new investment from Circle, Ripple, Standard Chartered’s venture arm and Qube Research as it pushes beyond exchange trading into tokenized shares, stablecoins and consumer finance.