gedaliyah@lemmy.world to Technology@lemmy.worldEnglish · 8 months agoReddit blocking all major search engines, except Googlereadwrite.comexternal-linkmessage-square64fedilinkarrow-up1498arrow-down111
arrow-up1487arrow-down1external-linkReddit blocking all major search engines, except Googlereadwrite.comgedaliyah@lemmy.world to Technology@lemmy.worldEnglish · 8 months agomessage-square64fedilink
minus-squareleopold@lemmy.kde.sociallinkfedilinkEnglisharrow-up17·8 months agothis is just going to cause indexers to ignore robots.txt
minus-squaregedaliyah@lemmy.worldOPlinkfedilinkEnglisharrow-up11·8 months ago“We always obey the robots.txt” A bunch of corporations that have no accountability and plenty of incentive to just ignore it and have all been caught training AI on off-limits data.
minus-squarecapital@lemmy.worldlinkfedilinkEnglisharrow-up3·8 months agoRate limiting could “fix” that unfortunately.
this is just going to cause indexers to ignore robots.txt
“We always obey the robots.txt”
Rate limiting could “fix” that unfortunately.