The Internet is at risk without proper control over web scraping: Cloudflare top exec

Cloudflare Field CTO John Engates speaks about how they want to tackle AI bot crawlers and the need to protect the internet.

Cloud and cybersecurity company, Cloudflare, is waging a war against AI bot crawlers. Web scraping, a phenomenon that began accelerating as the power of AI advanced, has crossed legal lines as AI firms are casually dismissing common online etiquette. Some tech companies have ignored ‘robot.txt’ files, which served as a binding instruction on whether data could or couldn’t be scraped from a website. Cloudflare is looking to fix such unruly behaviour with a permanent solution. They launched a bot management tool that monitors and blocks web scrapers. They also announced a marketplace for websites where they can negotiate payment terms with AI companies for using their data.

This year, the company expanded around its core offering and introduced AI-oriented services like Web Application Firewall and Cloudflare Workers, a serverless platform that allows developers to run their code fast. In an interaction with The Hindu, John Engates, Field CTO, Cloudflare explained how AI scraping can be made fair, the shift from training AI models to inferencing and the need for intelligent automation.