Under Attack, Please Stand By.

[jwz@mastodon.social]

Under Attack, Please Stand By.

My server is getting absolutely obliterated by AI scrapers today. Load is 140+ and I just manually banned 58,000 IP addresses.

Top ten user agents: ...
https://jwz.org/b/ykqJ

[jwz@mastodon.social]

Huh, why is my / disk full?

Oh:
-rw-------. 1 root root 1104886787 Aug 23 12:29 /var/log/php-fpm/error.log

That's just 6 days worth of AI-scraping bots.

[jwz@mastodon.social]

Getting absolutely reamed by AI scrapers again today, and it seemed like my mitigations were failing. Why weren't these being blocked?

Oh. Because I had Facebook's subnets on the fail2ban whitelist, so that people sharing @dnalounge links on the Zuckerweb got link previews.

Welp. You can't whitelist "facebookexternalhit" (the link preview bot) without also whitelisting "meta-externalagent" (the AI scraper, which seems to ignore robots.txt).

I guess link previews are gonna be a casualty.

[jwz@mastodon.social]

Welp. I now have proof that Facebook is using the same outbound IP addresses for both A) scraping web sites for "AI" training and B) fetching images from my site when I use the official API to post to Instagram.

This means that I can either defend myself from injuriously voracious AI scrapers, or have a functional business Instagram account, but not both.

Not just the same subnets. The same IPs.

[jwz@mastodon.social]

I guess today is going to be all about:

"I, a person you have never spoken to before, have unsolicited advice for you, since you are obviously brand new to this whole 'running a web site' thing."

"I, a person you have never spoken to before, feel the need to point out that the way Facebook is fucking you over is perfectly sensible, actually."

[jonw@cosocial.ca]

@jwz

How about:

I, a person, publically posted something on the internet and am now inexplicably confounded why people I don't know are responding.

Gimme a break man. If you don't want interaction then don't post.

[jwz@mastodon.social]

@jonw Oh look, it's some other fuckstain telling me I'm internetting wrong. *plonk*

[oblomov@sociale.network]

@jwz @jonw OT but I really like how you preserved the good old Usenet habit of adding *plonk* at the end of a message where you announce adding someone to the killfile.

[jwz@mastodon.social]

In the last 4 hours, fail2ban has nuked nearly *two thousand* distinct IP addresses for probing .git/config