Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 4 months agoHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square10fedilinkarrow-up168arrow-down121
arrow-up147arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 4 months agomessage-square10fedilink
minus-squareCynicus Rex@lemmy.mlOPlinkfedilinkarrow-up7arrow-down1·4 months ago#TL;DR: User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Omgilibot Disallow: / User-Agent: FacebookBot Disallow: / User-Agent: Applebot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Bytespider Disallow: / User-agent: Claude-Web Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: YouBot Disallow: /
#TL;DR:
User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Omgilibot Disallow: / User-Agent: FacebookBot Disallow: / User-Agent: Applebot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Bytespider Disallow: / User-agent: Claude-Web Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: YouBot Disallow: /