I’ve seen some people reject this solution due to the anime.

You don’t by chance use traefik with this? I’ve figured out how to use it with docker on the same device, but can’t figure out how to use it with external services.

Go_Away is another alternative

Anubis just released the no-JS challenge in a update. Page loads for me with JS disabled. https://anubis.techaro.lol/blog/release/v1.20.0/

The development of Anubis remains a matter of enthusiasm: Zee is funding the project through Patreon and sponsorship on GitHub, but cannot yet afford to pursue it on a full-time basis. He would also like to hire a key community member, budget permitting.

I’ve, once again, noticed Amazon and Anthropic absolutely hammering my Lemmy instance to the point of the lemmy-ui container crashing.

I’m just curious, how did you notice this in the first place? What are you monitoring to know and how do you present that information?

Besides that point: why tf do they even crawl lemmy. They could just as well create a “read only” instance with an account that subscribes to all communities … and the other instances would send their data. Oh, right, AI has to be as unethical as possible for most companies for some reason.

Something that’s less annoying than Anubis is fail2ban tarpitting the scrapers by putting in a hidden honeypot page link that they follow, and adding the followers to fail2ban.

https://petermolnar.net/article/anti-ai-nepenthes-fail2ban/

Been seeing this on people’s invidious instances

I’ve been planning on seeing this up for ages. Love the creators vibe. Thanks for this.

I don’t like Anubis because it requires me to enable JS – making me less secure. reddthat started using go-away recently as an alternative that doesn’t require JS when we were getting hammered by scrapers.

But don’t you know that Anubis is MALWARE?

…according to some of the clowns at the FSF, which is definitely one of the opinions to have. https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots

I am planning to try it out, but for caddy users I came up with a solution that works after being bombarded by AI crawlers for weeks.

It is a custom caddy CEL expression filter coupled with caddy-ratelimit and caddy-defender.

Now here’s the fun part, the defender plugin can produce garbage as response so when a matching AI crawler fits it will poison their training dataset.

Originally I only relied on the rate limiter and noticed that AI bots kept trying whenever the limit was reset. Once I introduced data poisoning they all stopped :)

git.blob42.xyz {
    @bot <<CEL
        header({'Accept-Language': 'zh-CN'}) || header_regexp('User-Agent', '(?i:(.*bot.*|.*crawler.*|.*meta.*|.*google.*|.*microsoft.*|.*spider.*))')
    CEL


    abort @bot
    

    defender garbage {

        ranges aws azurepubliccloud deepseek gcloud githubcopilot openai 47.0.0.0/8
      
    }

    rate_limit {
        zone dynamic_botstop {
            match {
                method GET
                 # to use with defender
                 #header X-RateLimit-Apply true
                 #not header LetMeThrough 1
            }
            key {remote_ip}
            events 1500
            window 30s
            #events 10
            #window 1m
        }
    }

    reverse_proxy upstream.server:4242

    handle_errors 429 {
        respond "429: Rate limit exceeded."
    }

}

If I am not mistaken the 47.0.0.0/8 ip block is for Alibaba cloud

It doesn’t stop bots

All it does is make clients do as much or more work than the server which makes it less temping to hammer the web.

Also your avatar and the image posted here (not the thumbnail) seem broken - I wonder if that’s due to Anubis?