Cloudflare’s AI Audit Helps Websites Combat Bots – And May Soon Make Bots Pay
Generative AI models, such as large language models, require vast amounts of data, much of which is collected by bots scraping the web. Cloudflare, one of the largest content delivery networks (CDNs) in the world, has introduced a powerful tool to combat these bots: AI Audit.
New AI Bot Visibility for Websites
Launched in beta on September 23, AI Audit offers website owners the ability to monitor AI bots scraping their content. This tool helps site managers identify which AI models are accessing their data, allowing them to choose whether to allow or block these bots. Currently available to Cloudflare customers, AI Audit goes beyond traditional bots’ control methods, like robots.txt, and leverages Cloudflare’s Web Application Firewall to automatically recognize AI-based traffic.
Sam Rhea, Cloudflare’s VP of emerging technology, emphasized the need to protect online content: “We aim to help build a better Internet, but some AI use cases pose risks to content creators and communities.”
Protection from Unwanted Bot Traffic
While large websites with robust infrastructures, such as SourceForge and SlashDot, can manage the impact of AI bots, smaller sites often face significant challenges. BingeClock, a site run by independent developer Billy Gardner McIntyre, struggled with AI bots that consumed resources, leading to extra server costs. Since using AI Audit, McIntyre noticed a marked decrease in unwanted traffic, reducing the number of server instances required to handle the load.
AI Bots Could Soon Face Charges
Looking forward, Cloudflare is working to enable monetization for website owners, allowing them to charge AI bots for crawling their content. While large companies like News Corp and OpenAI have already made deals to grant content access, Cloudflare envisions a future where even small website owners could benefit from similar agreements.
Although AI Audit is proving effective at blocking bots, some remain skeptical about its long-term potential for revenue generation. Nonetheless, Cloudflare is pushing the boundaries of bot management and transparency, and the future could see bots footing the bill for the data they consume.
