Server Struggles

[NFG]

A handful of AI bots have been hammering the server, bringing performance to new lows.  When loading forum pages went from instant to 5+ seconds, I started looking into mitigation.

Currently I've stopped responding to request from five AI bots and crawlers, and CPU usage dropped from 100% to 2%.  Still refusing hundreds of requests per second, but now at least they're causing much less trouble.

Probably have to move to a more robust system eventually.  One thing that's universally true about the people and companies who use AI - they're crooks, and they will absolutely lie about who they are to keep getting their free content.

[kendrick]

One of the things that stinks about putting a website on the Internet is that you generally have to have bot protection now. At the low end, a consumer-level Cloudflare barrier is insanely expensive and causes your site to have an outage when they have an outage. At the high end, actual machine-learning data-driven attack protection like F5 Shape is monstrously expensive and requires you to have a security staff to run it. Either way it's like a child having to hire a bodyguard to run your neighborhood lemonade stand, and it doesn't feel great.

[NFG]

What's your thinking about Anubis?  Seems like it does a good job of keeping the bots out.

[kendrick]

With the caveat that I don't have any direct experience with Anubis, I'm not really a fan of something that requires you to tune it directly. That's the sort of thing that becomes a daily chore, and on that basis alone I'm not sure Anubis is worth the effort.

My worry is that over the long term, the bots will eventually learn what Anubis does and work their way around it. The hazard of having AI bots is that they're self-correcting and eventually start to bypass the barriers we erect to stop them. Remember that one of the earliest UI automation demos was showing how a script could bypass the captcha checkbox that's supposed to demonstrate that a human being is driving the web connection. I hate not knowing how long an Anubis build will last, and not being able to weigh that information against the level of effort it takes to compile it and build it.

[BlackVega]

Looks like we live in truly dystopian times already

[NFG]

With the caveat that I don't have any direct experience with Anubis, I'm not really a fan of something that requires you to tune it directly. [...] I hate not knowing how long an Anubis build will last,

My understanding of it is that it requires the visiting client to do some proof of work, a small amount of CPU time for a normal user, but an enormous hit for a distributed crawler.

But yes, it's a constant arms race, isn't it?  There are no good solutions, just less awful ones.  As long as it only costs me time and not money I'm pretty happy to apply the effort and keep things running.