News

Google Faces Criticism Over AI Content Crawling Practices

Cansin Cengiz

Cansin Cengiz

— 2 min read
Google Faces Criticism Over AI Content Crawling Practices

Google Faces Publisher Backlash for AI Content Crawling

The ongoing friction between major tech platforms and content creators has reached a new peak, as Neil Vogel, CEO of People, Inc. (formerly Dotdash Meredith), publicly accused Google of unfairly leveraging publisher content to fuel its AI products.

Publisher Traffic Plummets as AI Features Expand

Speaking at the 2025 Fortune Brainstorm Tech conference, Vogel described how Google uses a single web crawler to both index sites for search and to collect data for its AI applications. This dual use, according to Vogel, means Google is "stealing content" from publishers while still benefiting from the traffic they send to their sites.

"Google has one crawler, which means they use the same crawler for their search, where they still send us traffic, as they do for their AI products, where they steal our content," Vogel explained. He noted that Google Search traffic to People, Inc.'s brands has dropped from 65% of total traffic three years ago to the "high 20s" today. In past years, this figure was as high as 90%.

Blocking AI Crawlers: A New Strategy for Publishers

To regain control, People, Inc. has started blocking AI crawlers that do not offer compensation, using new tools from Cloudflare. This move has prompted several large AI companies to initiate discussions about content licensing, though no deals have been finalized. Vogel singled out OpenAI as a "good actor" for agreeing to content deals, contrasting with Google's approach.

However, blocking Google's crawler is not an option, Vogel emphasized, because it would also remove the publisher's content from Google Search, costing them the vital remaining traffic.

Industry Voices Call for Fairer AI Practices

Janice Min, CEO and editor-in-chief of Ankler Media, echoed Vogel’s concerns, calling big tech platforms "content kleptomaniacs" and arguing that partnering with AI companies currently offers little benefit for publishers. Her company also blocks AI crawlers.

Matthew Prince, CEO of Cloudflare, suggested that new regulations might eventually force AI companies to change their behavior. He cautioned against relying solely on existing copyright law, which may not cover the complexities of AI-generated derivatives. Prince predicted that by next year, Google may be compelled to pay publishers for using their content to train AI models.

The Road Ahead: Negotiation and Regulation

The debate over fair compensation and control of content in the AI era is intensifying. With publishers deploying new technical defenses and pushing for licensing deals, the relationship between content creators and tech giants like Google is set for significant change. As regulatory scrutiny increases and business models evolve, publishers may yet gain leverage in negotiations with AI players.

References

Read more

Lex Proxima Studios LTD