Cloudflare's September 15 Deadline: What Changes for Your Site's AI Crawler Traffic

If your site sits behind Cloudflare and carries any ads, its relationship with AI crawlers changes on September 15, 2026, whether you touch a setting or not. Cloudflare announced on July 1 that it will start blocking “mixed-use” AI crawlers by default from ad-supported pages, and it’s pairing that with a pay-per-use model that pays publishers when their content actually gets used in an AI answer, not just when a bot fetches it.

This matters for more sites than the headline suggests, because most of the internet’s request traffic already runs through Cloudflare’s network in some form. If you haven’t looked at your bot management settings recently, now’s the time.

What a “mixed-use” crawler actually is

The distinction Cloudflare is drawing isn’t “AI bot” versus “not AI bot.” It’s about crawlers that blend multiple jobs under one identity. A traditional search crawler indexes your page and, in exchange, sends you traffic when someone clicks through from search results. A mixed-use crawler does something else: it might index for search, but the same bot identity is also feeding an AI agent’s live browsing or a model’s training pipeline, and none of those uses necessarily sends a human back to your site.

That distinction has been hard to enforce because crawlers haven’t been required to declare which of those jobs they’re doing on any given request. Cloudflare’s default change forces the issue: starting September 15, if a crawler doesn’t cleanly separate those functions, and your page has ads on it, the crawler gets blocked unless you explicitly allow it.

Who this hits automatically

The rollout isn’t universal on day one. Cloudflare has scoped the new default to:

• New Cloudflare customers signing up after the change takes effect
• New sites added to existing Cloudflare accounts
• All existing free-tier customers, regardless of when they signed up

If you’re a paying customer on a site you set up before the change, your current bot settings carry over. You can still opt into the new default, but it isn’t forced on you. If you’re on the free tier, the default flips under you, so check your bot management dashboard before mid-September rather than finding out from a traffic drop.

Why Cloudflare is doing this: the re-fetch problem

Cloudflare has published its own crawl data showing that a large share of AI crawler traffic, north of 50% by the company’s own numbers, is spent re-fetching pages that haven’t changed since the last crawl. That’s bandwidth and origin compute spent for zero incremental value to the publisher, at a volume most site owners never notice because it doesn’t show up as a support ticket, it shows up as a slightly higher hosting bill and a slightly slower origin.

Blocking mixed-use crawlers by default on ad-supported pages is a blunt instrument for a specific complaint: crawlers that consume resources without a clear value exchange back to the site. It also sets up the second half of the announcement.

Pay Per Use: getting paid for outcomes, not fetches

Cloudflare’s existing Pay Per Crawl marketplace let publishers charge AI crawlers per fetch. The expanded Pay Per Use model changes what triggers payment: instead of “the bot requested this page,” it’s “this content showed up in an AI system’s output.” Cloudflare has started this with two initial partners, Ceramic.ai and You.com. When a publisher opts in, they get paid when their content surfaces in a Ceramic AI search result or gets pulled into a You.com answer.

That’s a meaningfully different economic model than a flat per-crawl fee, and it’s a more direct answer to the complaint publishers have had since AI answer engines started summarizing content instead of linking to it: getting compensated only correlates weakly with getting crawled, but it correlates strongly with getting used.

What to actually check before September 15

If you manage a Cloudflare-proxied site, here’s the practical checklist:

1. Confirm your account tier and site age. If you’re on the free tier, or setting up a new site, you’re in the group that gets the new default automatically.
2. Review your current bot management or WAF rules for AI crawler categories. Cloudflare’s dashboard breaks out verified bot categories; check which ones you currently allow or challenge.
3. Decide your actual position on AI crawlers, not just search crawlers. Do you want AI answer engines citing your content? If visibility in AI search results matters to your traffic or brand, blocking everything by default and forgetting about it isn’t free, it just trades one kind of traffic loss for another.
4. If you’re already thinking about content strategy for an AI-answer-first web, this pairs directly with the shift we covered in how AI search is eating web traffic: getting crawled and getting cited are two different problems, and Cloudflare’s changes affect the crawling side of that equation, not the citation side.

The bigger shift this is part of

This is the second major move in 2026 where infrastructure providers, not publishers or AI labs, are setting the terms for how AI crawling works at the protocol level. Publishers individually have very little leverage against a crawler that ignores robots.txt or a scraper that rotates IPs to avoid blocks. A CDN sitting in front of a meaningful fraction of the web’s traffic has real leverage, and Cloudflare is using it to push the negotiation from “please respect our wishes” to “the default now requires you to identify yourself honestly or get blocked.”

Whether that holds up depends on how many AI companies find it cheaper to pay for access than to build crawlers that dodge the classification. But for site owners, the immediate task isn’t philosophical. It’s checking a dashboard setting before a deadline that’s already on the calendar.