Reddit to update web standard to block automated website scraping
The Hindu
Reddit said it will update a web standard used by the platform to block automated data scraping.
Social media platform Reddit said on Tuesday it will update a web standard used by the platform to block automated data scraping from its website, following reports that AI startups were bypassing the rule to gather content for their systems.
The move comes at a time when artificial intelligence firms have been accused of plagiarising content from publishers to create AI-generated summaries without giving credit or asking for permission.
Reddit said that it would update the Robots Exclusion Protocol, or "robots.txt," a widely accepted standard meant to determine which parts of a site are allowed to be crawled.
The company also said it will maintain rate-limiting, a technique used to control the number of requests from one particular entity, and will block unknown bots and crawlers from data scraping - collecting and saving raw information - on its website.
(For top technology news of the day, subscribe to our tech newsletter Today’s Cache)
More recently, robots.txt has become a key tool that publishers employ to prevent tech companies from using their content free-of-charge to train AI algorithms and create summaries in response to some search queries.
Last week, a letter to publishers by the content licensing startup TollBit said that several AI firms were circumventing the web standard to scrape publisher sites.
Bombay High Court grants bail to Chetan Patil in Shivaji statue collapse case; sculptor Jaydeep Apte's plea pending.
Deadly rinderpest outbreak devastates Mudumalai Wildlife Sanctuary, highlighting the threat of diseases from domestic cattle to wild populations.
Suryanarkoil Mutt's history, traditions, and leadership succession explained in detail with references and comparisons to other Saivaite Mutts.
Upalokayukta takes action against officials for neglecting century-old Kelageri Tank in Dharwad, setting 20-day deadline for rejuvenation.
On World Fisheries Day, India launches projects for marine and inland aquaculture, aiming to transform the fisheries sector and blue economy.
Lokayukta police raid uncovers ₹26.6 crore in assets from government officials in Karnataka, leading to investigations and cases filed.
Court boycott by advocates in Tamil Nadu demands Advocates Protection Bill for safety, affecting litigants from remote villages.
Army faces allegations of harassment and torture in Kishtwar, Jammu; probe ordered, sparking concerns over human rights violations.
Karnataka’s software and services exports crossed ₹4.11 lakh crore during fiscal 2023-24 (March 2024) from ₹3.2 lakh crore a year ago, marking a 27% year-on-year growth, Minister for IT&BT, Priyank Kharge, said here on Thursday quoting Software Technology Parks of India (STPI) data.
Continuous rainfall in Thoothukudi has caused severe hardship for residents as rainwater has inundated several areas. Efforts to drain stagnant water in the city have been intensified
Indictment details Adani bribery scheme to secure solar power contracts, involving Adani Green Energy and Azure Power executives.