https://www.theregister.com/2025/06/17/bot_overwhelming_websites_report/ # # Sign in / up The Register | HPE # # # Topics Security Security All SecurityCyber-crimePatchesResearchCSO (X) Off-Prem Off-Prem All Off-PremEdge + IoTChannelPaaS + IaaSSaaS (X) On-Prem On-Prem All On-PremSystemsStorageNetworksHPCPersonal TechCxOPublic Sector (X) Software Software All SoftwareAI + MLApplicationsDatabasesDevOpsOSesVirtualization (X) Offbeat Offbeat All OffbeatDebatesColumnistsScienceGeek's GuideBOFHLegalBootnotesSite NewsAbout Us (X) Special Features Special Features All Special Features Datacenter Networking Nexus The State of Storage European Supercomputing AI Infrastructure Month Spotlight on RSAC AI Software Development Week Disaster Recovery Week Nvidia GTC Ransomware in Focus The Future of the Datacenter Cybersecurity Month VMware Explore Cloud Infrastructure Month Vendor Voice Vendor Voice Vendor Voice All Vendor Voice Money Movement Hub The BigQuery Difference AWS Global Partner Security Initiative Amazon Web Services (AWS) New Horizon in Cloud Computing GE Vernova with AWS Google Gemini (X) Resources Resources Whitepapers Webinars & Events Newsletters [aiml] AI + ML 21 comment bubble on white Bots are overwhelming websites with their hunger for AI data 21 comment bubble on white GLAM-E Labs report warns of risk to online cultural resources icon Thomas Claburn Tue 17 Jun 2025 // 10:28 UTC # Bots harvesting content for AI companies have proliferated to the point that they're threatening digital collections of arts and culture. Galleries, Libraries, Archives, and Museums (GLAMs) say they're being overwhelmed by AI bots - web crawling scripts that visit websites and download data to be used for training AI models - according to a report issued on Tuesday by the GLAM-E Lab, which studies issues affecting GLAMs. GLAM-E Lab is a joint initiative between the Centre for Science, Culture and the Law at the University of Exeter and the Engelberg Center on Innovation Law & Policy at NYU Law. [aiml] Based on an anonymized survey of 43 organizations, the report indicates that cultural institutions are alarmed by the aggressive harvesting of their content, which shows no regard for the burden that data-harvesting places on websites. [aiml] [aiml] "Bots are widespread, although not universal," the report says. "Of 43 respondents, 39 had experienced a recent increase in traffic. Twenty-seven of the 39 respondents experiencing an increase in traffic attributed it to AI training data bots, with an additional seven believing that bots could be contributing to the traffic." The surge in bots that gather data for AI training, the report says, often went unnoticed until it became so bad that it knocked online collections offline. [aiml] "Respondents worry that swarms of AI training data bots will create an environment of unsustainably escalating costs for providing online access to collections," the report says. The institutions commenting on these concerns have differing views about when the bot surge began. Some report noticing it as far back in 2021 while others only began noticing web scraper traffic this year. Some of the bots identify themselves, but some don't. Either way, the respondents say that robots.txt directives - voluntary behavior guidelines that web publishers post for web crawlers - are not currently effective at controlling bot swarms. [aiml] Bot defenses offered by the likes of AWS and Cloudflare do appear to help, but GLAM-E Lab acknowledges that the problem is complex. Placing content behind a login may not be effective if an institution's goal is to provide public access to digital assets. And there may be a reason to want some degree of bot traffic, such as bots that index sites for search engines. * Salesforce study finds LLM agents flunk CRM and confidentiality tests * The launch of ChatGPT polluted the world forever, like the first atomic weapons tests * Alt cloud platform Railway forced to pause lowest tiers after onrush of GCP customers * BT chief says AI could deliver more job cuts, hints at Openreach sell-off The GLAM-E Lab survey echoes the findings of a similar report issued earlier this month by the Confederation of Open Access Repositories (COAR) based on the responses of 66 open access repositories run by libraries, universities, and other institutions. The COAR report says: "Over 90 percent of survey respondents indicated their repository is encountering aggressive bots, usually more than once a week, and often leading to slowdowns and service outages. While there is no way to be 100 percent certain of the purpose of these bots, the assumption in the community is that they are AI bots gathering data for generative AI training." The GLAM-E Lab survey also recalls complaints about abusive bots raised by The Wikimedia Foundation, Sourcehut, Diaspora developer Dennis Schubert, repair site iFixit, and documentation project ReadTheDocs. Ultimately, the GLAM-E report argues that AI providers need to develop more responsible ways to interact with other websites. "The cultural institutions that host online collections are not resourced to continue adding more servers, deploying more sophisticated firewalls, and hiring more operations engineers in perpetuity," the report says. "That means it is in the long-term interest of the entities swarming them with bots to find a sustainable way to access the data they are so hungry for." (r) Get our Tech Resources # Share More about * AI * Security * Software More like these x More about * AI * Security * Software * World Wide Web Narrower topics * 2FA * AdBlock Plus * Advanced persistent threat * AIOps * App * Application Delivery Controller * Audacity * Authentication * BEC * Black Hat * BSides * Bug Bounty * CHERI * CISO * Common Vulnerability Scoring System * Confluence * Cybercrime * Cybersecurity * Cybersecurity and Infrastructure Security Agency * Cybersecurity Information Sharing Act * Database * Data Breach * Data Protection * Data Theft * DDoS * DeepSeek * DEF CON * Digital certificate * Encryption * Exploit * Firewall * FOSDEM * FOSS * Gemini * Google AI * GPT-3 * GPT-4 * Grab * Graphics Interchange Format * Hacker * Hacking * Hacktivism * IDE * Identity Theft * Incident response * Infosec * Infrastructure Security * Jenkins * Kenna Security * Large Language Model * Legacy Technology * LibreOffice * Machine Learning * Map * MCubed * Microsoft 365 * Microsoft Office * Microsoft Teams * Mobile Device Management * NCSAM * NCSC * Neural Networks * NLP * OpenOffice * Palo Alto Networks * Password * Personally Identifiable Information * Phishing * Programming Language * QR code * Quantum key distribution * Ransomware * Remote Access Trojan * Retro computing * REvil * RSA Conference * Search Engine * Software bug * Software License * Spamming * Spyware * Star Wars * Surveillance * Tensor Processing Unit * Text Editor * TLS * TOPS * Trojan * Trusted Platform Module * User interface * Visual Studio * Visual Studio Code * Vulnerability * Wannacry * WebAssembly * Web Browser * Webcast * WordPress * Zero trust Broader topics * Network * Self-driving Car More about # Share 21 comment bubble on white COMMENTS More about * AI * Security * Software More like these x More about * AI * Security * Software * World Wide Web Narrower topics * 2FA * AdBlock Plus * Advanced persistent threat * AIOps * App * Application Delivery Controller * Audacity * Authentication * BEC * Black Hat * BSides * Bug Bounty * CHERI * CISO * Common Vulnerability Scoring System * Confluence * Cybercrime * Cybersecurity * Cybersecurity and Infrastructure Security Agency * Cybersecurity Information Sharing Act * Database * Data Breach * Data Protection * Data Theft * DDoS * DeepSeek * DEF CON * Digital certificate * Encryption * Exploit * Firewall * FOSDEM * FOSS * Gemini * Google AI * GPT-3 * GPT-4 * Grab * Graphics Interchange Format * Hacker * Hacking * Hacktivism * IDE * Identity Theft * Incident response * Infosec * Infrastructure Security * Jenkins * Kenna Security * Large Language Model * Legacy Technology * LibreOffice * Machine Learning * Map * MCubed * Microsoft 365 * Microsoft Office * Microsoft Teams * Mobile Device Management * NCSAM * NCSC * Neural Networks * NLP * OpenOffice * Palo Alto Networks * Password * Personally Identifiable Information * Phishing * Programming Language * QR code * Quantum key distribution * Ransomware * Remote Access Trojan * Retro computing * REvil * RSA Conference * Search Engine * Software bug * Software License * Spamming * Spyware * Star Wars * Surveillance * Tensor Processing Unit * Text Editor * TLS * TOPS * Trojan * Trusted Platform Module * User interface * Visual Studio * Visual Studio Code * Vulnerability * Wannacry * WebAssembly * Web Browser * Webcast * WordPress * Zero trust Broader topics * Network * Self-driving Car TIP US OFF Send us news --------------------------------------------------------------------- Other stories you might like Enterprises are getting stuck in AI pilot hell, say Chatterbox Labs execs Interview Security, not model performance, is what's stalling adoption AI + ML8 Jun 2025 | 30 Schneier tries to rip the rose-colored AI glasses from the eyes of Congress DOGE moves fast and breaks things, and now our data is at risk, security guru warns in hearing Public Sector6 Jun 2025 | 52 The launch of ChatGPT polluted the world forever, like the first atomic weapons tests Feature Academics mull the need for the digital equivalent of low-background steel AI + ML15 Jun 2025 | 103 How homegrown AI cuts through the hype to deliver real results Nutanix leverages customer interactions to develop GenAI infra solution and the AI tools to support it Sponsored feature [aiml] Meta offered one AI researcher at least $10,000,000 to join up Exclusive Mark Zuckerberg reached out to our source directly AI + ML13 Jun 2025 | 55 AI kept 15-year-old zombie vuln alive, but its time is drawing near Researchers have come up with a fix for a path traversal bug first spotted in 2010 Research5 Jun 2025 | 27 AI coding tools are like that helpful but untrustworthy friend, devs say Exclusive A survey from AI biz Qodo finds robo-coding productivity gains are unevenly distributed AI + ML12 Jun 2025 | 39 UK students flock to AI to help them cheat No need to plagiarize if you can have AI do it for you AI + ML16 Jun 2025 | 81 Boffins found self-improving AI sometimes cheated Instead of addressing hallucinations, it just bypassed the function they built to detect them AI + ML2 Jun 2025 | 28 Mozilla frets about Google's push to build AI into Chrome AI could bring a new round of browser wars AI + ML11 Jun 2025 | 21 IBM Watson zombie brand shuffles forward with new AI lab in NYC Unsurprisingly, it's all about agents, the buzzword du jour AI + ML2 Jun 2025 | 12 AI's the end of the Shell as we know it and I feel fine ... but insecure Column Model Context Protocol has many fine uses, but then it hinted at becoming a Von Neumann machine AI + ML11 Jun 2025 | 21 The Register icon Biting the hand that feeds IT About Us* * Contact us * Advertise with us * Who we are Our Websites* * The Next Platform * DevClass * Blocks and Files Your Privacy* * Cookies Policy * Privacy Policy * Ts & Cs * Do not sell my personal information Situation Publishing Copyright. All rights reserved (c) 1998-2025 no-js