The decades-old deal between Google and websites is broken

Key takeaways

• For decades, websites let Google crawl their content for free, and Google sent traffic back. AI answers broke that exchange.
• A federal judge just confirmed what many of us suspected. It was never a contract. It was an expectation.
• Blocking AI crawlers usually backfires. The smarter play is to be surgical about what you protect and what you open up.

---

For more than two decades, every website owner on the planet signed a contract that nobody ever wrote down.

You let Google crawl your pages. Google indexed them, ranked them and sent you visitors. Those visitors bought your products, read your ads and subscribed to your newsletter. Everybody won.

This fall, a federal judge said the quiet part out loud. Dismissing antitrust suits from Chegg and Penske Media over Google's AI Overviews, Judge Amit Mehta wrote that the plaintiffs had only an "expectation" of traffic, and that "an expectation is not an agreement."

He's technically right. And that's exactly the problem.

The deal that built the open web

The rules of the web have always been voluntary. The robots.txt standard that tells crawlers where they can and can't go dates back to 1994, more than 30 years ago, and it has always run on good faith. Websites opened their doors because search engines gave something valuable back.

Cloudflare's CEO, Matthew Prince, summed up the old arrangement neatly: "let us copy your content for search, and we'll send you traffic."

That trade built an entire industry. It built my career. Some of the pages I'm proudest of still rank incredibly well years after they were published. The deal worked because both sides held up their end.

The receipts: what AI answers are doing to clicks

Now the search engine reads your page, writes its own answer at the top of the results and gives the reader most of what they needed before they ever see your link.

The data is hard to ignore:

• Fewer clicks. A Pew Research Center study of real browsing behavior found people clicked a traditional search result on 8% of visits when an AI summary appeared, versus 15% when it didn't. They clicked a link inside the summary itself just 1% of the time.
• More dead ends. In the same study, people ended their browsing session on 26% of pages with an AI summary, compared with 16% without one.
• Top rankings are worth less. Ahrefs found that AI Overviews correlated with a 58% lower click-through rate for the number one result, across 300,000 keywords.
• Other AI tools take even more than they give. Cloudflare measured OpenAI crawling 887 pages for every visitor it referred in August 2025. Prince says getting the same traffic from Google is now about 10 times harder than it was a decade ago.

To be fair to Google, it sees things differently. Google's head of Search says total organic clicks have been "relatively stable" year over year, and that the clicks AI search does send are higher quality. Both things can be true at once. Overall volume can hold steady while specific sites, especially ones whose product is information, get crushed.

The courts won't save you (yet)

If you were hoping a lawsuit would restore the old deal, the September ruling should reset your expectations. Antitrust law, at least as it's been argued so far, doesn't treat that unwritten deal as a promise. The Penske case was dismissed without prejudice, so this fight isn't over. But no brand should build its 2027 plan around winning it.

I have real sympathy for publishers and information businesses here. They spent years and serious money creating research and reporting, and AI tools now summarize it for free. For them, the question of whether to block AI crawlers is genuinely painful.

Why blocking AI crawlers usually backfires

When the deal breaks, the instinct is to slam the door. Block every AI bot and make them pay.

For most brands, that's a mistake. Here's why.

The AI will just cite someone else. People are turning to AI tools more every month. If your information isn't available, the answer still gets written. It just credits your competitor, a forum thread or a reseller instead of you.

You can't block AI Overviews without hurting your Google rankings. Google says AI Overviews and AI Mode use the same controls as regular Search. Blocking Google-Extended stops your content from being used to train Gemini, but Google says it does not affect inclusion in Search. Opting out of AI Overviews means opting out of a big chunk of your search visibility.

Most AI citations come from the wider web anyway. Muck Rack's research found earned media made up 84% of AI citations in May 2026. Even if you lock down your site, AI tools will keep talking about your brand. The only question is whether you're part of that conversation.

Be surgical: what to do instead

The old deal is gone. The new one is less generous, but it isn't worthless. Here's how I'd approach it.

1. Sort your content into "the product" and "the storefront"

Some content is your actual product: proprietary data, paid research, tools that took years to build. Protect that with paywalls, logins or targeted crawler rules. Everything else is your storefront: guides, product pages, expert explainers. Keep the storefront wide open. Blanket blocking treats both the same, and that's the expensive mistake.

2. Publish what a summary can't replace

AI can summarize facts. It can't replicate a calculator, an interactive tool, original data or hard-won experience. That's the same reason E-E-A-T matters more than keywords now. Give people a reason to click through that a three-sentence answer can't satisfy.

3. Win the citation, then make it worth the click

Being cited matters. Seer Interactive found that brands cited in AI Overviews earned about 120% more clicks per impression than uncited pages on the same results pages. Build the kind of Unique Entity that AI tools and journalists are compelled to reference.

4. Measure AI visibility like brand, not like last click

Clicks per search will keep falling. That doesn't mean your organic program is failing. Track the full picture: branded search, direct traffic, conversion rates and revenue per visit. Treat AI visibility like PR or word of mouth. It's hard to attribute, but it clearly shapes what people buy.

The new deal

The old deal was simple: access for traffic. The new deal is tougher: access for citation, and citation for authority.

You don't get to renegotiate with Google. You do get to decide whether you're worth citing, and whether you're worth the click.