Since multiple Lemmy instances show the same, federated, content, I wonder if our posts will have more weight in the model, for a normal scraper it would be as if many people repeated the same thing over and over on different sites.
I’d imagine they’d be doing it for the whole fediverse. I mean from their perspective, why not? The whole open nature, in my limited understanding, seems to make that easier.
I swear I read they’re doing it with Lemmy too
Everything that’s freely accessible on the internet has been scraped 400 times over.
Since multiple Lemmy instances show the same, federated, content, I wonder if our posts will have more weight in the model, for a normal scraper it would be as if many people repeated the same thing over and over on different sites.
It would be trivial to remove duplicates. With a little bit of foresight they could just as easily avoid the duplicates in the first place.
They could also avoid to re-crawl the whole internet every day, but here we are, so who knows.
I’d imagine they’d be doing it for the whole fediverse. I mean from their perspective, why not? The whole open nature, in my limited understanding, seems to make that easier.
they crawl the entire internet, lemmy is definitely included