Publishers Block AI Bots To Protect Content

News publishers continue to fight against AI bots, suing tech companies, warning that scraping poses dangers to journalism, fair compensation systems, as well as the open web’s future.

In a new report by The Wall Street Journal (WSJ), News publishers are fighting back against AI companies that scrape their websites for content without compensation. As AI tools like ChatGPT and Google’s Gemini grow, many media companies are trying to block bots that use their work without permission.

“You want humans reading your site, not bots, particularly bots that aren’t returning any value to you,” said Nicholas Thompson, CEO of The Atlantic, which has a licensing deal with OpenAI but plans to block other AI companies, as reported by WSJ.

This tactic, known as “scraping,” has existed since the early days of Google. Back then, search engines drove traffic to publishers’ websites. Now, AI chatbots enable news summaries which redirect readers away from visiting their original sources. The combination of bot driven traffic reduction, and advertising revenue decline has become a common issue for numerous publishers.

To fight back, publishers are turning to tech companies like Cloudflare, which recently launched tools to let websites control whether AI bots can access content. Dotdash Meredith CEO Neil Vogel, whose company also licenses content to OpenAI, said, “People who create intellectual property need to be protected or no one will make intellectual property anymore,” as reported by WSJ.

Some companies, like Reddit and iFixit, have taken legal action. Reddit sued AI company Anthropic for scraping over 100,000 times despite requests to stop. iFixit said Anthropic hit its servers one million times in a single day.

The fight is also playing out in court. The New York Times is suing Microsoft and OpenAI, while News Corp and its subsidiaries are taking on Perplexity. The BBC has also threatened legal action against AI startup Perplexity, accusing it of scraping its content to train its default model.

Meanwhile, some worry that stricter anti-scraping rules could block legitimate uses like academic research, as noted by WSJ.

As Shayne Longpre of the Data Provenance Initiative warned, “The web is being partitioned to the highest bidder. That’s really bad for market concentration and openness,” as reported by WSJ.