Site connection issues

[darjr]

Facebook is apparently also having issues, which matches with AI bots trying to frantically scrape everything all at once.

Yea, it looks like AI bots scraping for content.

A lot from AWS and Google and Microsoft compute servers, a couple that looked like compromised hosts. I bet that most of them are.

Also a couple older "legit" scrapers that used to play by the rules were hammering ENWorld too.

 

[Whizbang Dustyboots]

Robots.txt was always a wildly optimistic method of fending off bad behavior. It certainly was never going to last in the post-pandemic hellscape.

 

[Scribe]

Ah, AI being the root of it makes sense. I've been doing a lot of lookup's on Google, which now spits out an AI answer, and a bunch of hits are coming from Enworld.

 

[SkidAce]

If AI is constantly trawling the site, how can you fix it? Or does it come in waves for some reason?

 

[Whizbang Dustyboots]

If AI is constantly trawling the site, how can you fix it?

Historically, you could stop bots (mostly search engine bots) from crawling your site by including a text file in the top level domain folder (what's in the folder for the basic URL, and not any subdirectories accessed via slashes) called robots.txt that listed what pages and directories to stay out of. It was an honor system way of keeping stuff from showing up in search engines, etc.

Not surprisingly, AI companies don't care about playing nice and at the moment, it's hard -- maybe impossible -- to keep them off a site, unless you paywall it.

 

[trappedslider]

Historically, you could stop bots (mostly search engine bots) from crawling your site by including a text file in the top level domain folder (what's in the folder for the basic URL, and not any subdirectories accessed via slashes) called robots.txt that listed what pages and directories to stay out of. It was an honor system way of keeping stuff from showing up in search engines, etc.

Not surprisingly, AI companies don't care about playing nice and at the moment, it's hard -- maybe impossible -- to keep them off a site, unless you paywall it.

which makes me wonder how other non-pay walled forums do it like the WoW forums among others.

 

[Morrus]

which makes me wonder how other non-pay walled forums do it like the WoW forums among others.

Largely, they don't.

 

[Whizbang Dustyboots]

Yeah, Blizzard is under a 24/7 DDoS attack from all sides, from any number of entities.