Multiple AI companies bypassing web standard to scrape publisher sites, licensing firm says
The Hindu
AI companies are bypassing web standards to scrape content, sparking debate between tech and media firms over generative AI.
Multiple artificial intelligence companies are circumventing a common web standard used by publishers to block the scraping of their content for use in generative AI systems, content licensing startup TollBit has told publishers.
A letter to publishers seen by Reuters on Friday, which does not name the AI companies or the publishers affected, comes amid a public dispute between AI search startup Perplexity and media outlet Forbes involving the same web standard and a broader debate between tech and media firms over the value of content in the age of generative AI.
The business media publisher publicly accused Perplexity of plagiarizing its investigative stories in AI-generated summaries without citing Forbes or asking for its permission.
A Wired investigation published this week found Perplexity likely bypassing efforts to block its web crawler via the Robots Exclusion Protocol, or "robots.txt," a widely accepted standard meant to determine which parts of a site are allowed to be crawled.
(For top technology news of the day, subscribe to our tech newsletter Today’s Cache)
Perplexity declined a Reuters request for comment on the dispute.
The News Media Alliance, a trade group representing more than 2,200 U.S.-based publishers, expressed concern about the impact that ignoring "do not crawl" signals could have on its members.
Delhi CM-designate Rekha Gupta expresses gratitude to PM Modi, vows to fulfill vision, become fourth woman CM of Delhi.
College teachers in Chennai protest for career advancement benefits, condemning unequal treatment between government and aided colleges.
Dredging of Kallai River in Kozhikode begins to restore natural flow and reduce waterlogging, long-awaited project.
Government removes roadside encroachments in Puducherry's Bharathi Street to improve traffic, traders protest, removal put on hold temporarily.
Minister for Local Self-Governments M.B. Rajesh distributed the Swaraj Media Awards at Guruvayur on Wednesday. S.R. Praveen, Principal Correspondent, The Hindu, received the Swaraj Media Award (Print Media) for his report “Foot soldiers of Kerala’s silent revolution against waste.”
AIMIM urges Maharashtra government to allow Muslim employees and students to leave an hour early during Ramzan.
Samsung India Workers Union (SIWU) plans a massive strike on March 7 over suspended workers, amid ongoing negotiations with management.
Health Minister inaugurates development projects worth ₹1.60 crore in Virugambakkam, Chennai funded by MP and MLA.
Uttar Pradesh jails to bring holy water from Maha Kumbh to all prisons for inmates to participate.
NGT criticizes UPPCB for lack of data on faecal coliform levels in Ganga and Yamuna rivers during Maha Kumbh.
Voorhees College wins All India Football Tournament at VIT's Riviera-2025, with principal congratulating team.
Retteri lake restoration project near completion, increasing storage capacity for fresh water supply in Chennai by next monsoon.
Delhi High Court advocates understanding over punishment in cases of adolescent love, prioritizing safety and well-being of young individuals.