Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.
Last week, 404 Media noticed that search engines that weren’t Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: “Reddit believes in an open Internet, but not the misuse of public content.” Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.
The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).
In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It’s unclear how much Reddit’s OpenAI deal is worth.
Huffman said:
Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.
“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.
The enshittification cycle:
Phase one, attract users by providing a good service.
Phase two, once the users are locked in, squeeze them for all they’re worth by selling them to business customers (advertisers and/or data buyers).
Phase three, once the business customers are locked in, squeeze them for all they’re worth by threatening to deny them access to the users on whom they now depend.Spez seems to think Reddit has the pull to make phase 3 happen. I rather doubt it, but we’ll see.
Yeah, as soon as the API thing happened I switched to Lemmy for mobile browsing and like it more than Reddit (Connect is pretty good, but even the mobile browser site is solid).
The more they squeeze, the more popular alternatives like Lemmy, Kbin/Mbin, Tildes, etc. will become.
My guess is that phase three will work for a while. But I think you’re right that eventually they are going to drive that thing into the ground. Because it’s never enough pure profit for rent-seeking scum, and there is no lower limit to the abuse they’ll inflict on their content creators (who they call users but think of as products).
Fuck Spez. He’s probably editing the comments anyway, he literally can’t help himself.
Honestly, my biggest issue with LLMs is how they source their training data to create “their own” stuff. A meme calling it a plagiarism machine struck a chord with me. Almost anyone else I’d sympathize with, but fuck Spez.
Yep they now get paid for the data we have them. I have no sympathy lol. At least these models can’t actually store it all losslessly by any stretch of the imagination. The compression factors would have to be like 100-200X+ anything we’ve ever been able to achieve before. The numbers don’t work out. The models do encode a lot though and some of it is going to include actual full text data etc but it’ll still be kinda fuzzy.
I think we do need ALL OPEN SOURCE. Not just for AI, but I know on that point I’m preaching to the choir here lol
I’m sure plenty of others join me in the sentiment of thinking “Who the fuck are you to restrict MY free content that I contributed?”
God, fuck reddit so fucking hard
profiting off of user generated content 😒
And yet reddit is happy to make money off our content for free.
Or at least it did. Personally I overwrote and deleted all my content a while back.
You think that Reddit didn’t already have the previous content saved?
Everyone always says this like it’s some kind of gotcha, but all of my nuked posts still have my “fuck you, reddit” content and haven’t been reverted. It’s been nearly exactly a year.
Maybe reddit has an offline copy of my old content and that of others somewhere, but if so they’d be handing that directly over to whoever under some kind of agreement – that certainly wouldn’t be the subject of any kind of site crawling which is the crux of the issue here.
it never was deleted, all that happened is that an extra line was added to a database that said “comment 65432426542654 now should be displayed as “fuck you, reddit” rather than the original text”. The original post is still in an earlier row available to reddit, it just isnt being displayed on their web page.
I certainly wasn’t implying that they were going to revert your comments.
i went looking for old comments and posts i had made after i overwrote then wiped them. They’re still gone. i looked again several months later, and they were still gone.
so, unless reddit did a massive restore of everyone’s comments/posts except for my 4 accounts, then i don’t believe they did it at all except for a select number of top contributors who deleted their content.
As I said in another comment, I was not suggesting that Reddit would restore your comments to public view.
there’s no evidence to suggest that, either.
Except for incidents like This
It seemed to happen to some people but I wouldn’t be surprised it it was just some sort of coincidental database fuck-up
i suspect that was more likely incomplete deletions than reddit restoring content. those scripts were pretty janky. i had to run mine several times to get everything, as it didn’t work fully the first couple of times. same with the overwrites. took a few times for those to get everything, especially on older accounts with lots of posts and comments.
I’d read some claims that posts appeared to be deleted but then later came back. Could’ve even been some sort of caching shenanigans with their local browser though I guess.
oh, right. i suppose that’s possible. i’ve seen similar browser cache fuckery on other sites before.
as i said in a previous comment:
so, unless reddit did a massive restore of everyone’s comments/posts except for my 4 accounts, then i don’t believe they did it at all except for a select number of top contributors who deleted their content.
but there’s no evidence they’re keeping everyone’s deleted-but-restored comments from public view or whatever it is you’re suggesting. or even anything past whaat this one person found. in fact, there isn’t even any evidence that what happened to this user was intentional and not a bug or some other fluke.
sure, reddit would have a vested interest in doing this, and what you’ve presented is suspicious, but it’s hardly conclusive of anything. all it does is raise more questions. but it doesn’t provide answers.
Yep that’s how it works. Older content past a certain date is cached which is why you can’t comment or post on some old posts.
Reddit is dying anyway.
It’s easy to say this as someone who is “on the other side”. But the data doesn’t really back up that statement.
I don’t have data, but the quality of the content certainly seemed to be declining, even as the quantity went up.
I think autocorrect might’ve gotten you: You posted “quality” twice in a contradictory way.
Fixed.
What if I had an agreement with MS that they can scrape my data and anything I post online?
I never bothered to go edit or delete my comments after the API drama that caused me to move here, but now I might just go do that because the entire point of keeping old comments up was that maybe someone will find one from a search engine and find it useful. If reddit is going to monetize THAT, they can fuck right off.
Fuck Spez
Gross.
No thanks.
I’m not sure what Reddit is still doing with that spaz. Ditch the fucker.
Yeah fuck that spud guy!
Minimum royalty laws should exist!
Because if there’s one thing this world needs more its more rights for property.
Late stage capitalism
How about starting a company that gathers people’s CAD design…grabCAD!.. Oh can’t scrape out design work Microsoft, you gotta pay!..or how about a company that stores people’s records or drawings or movies… Adobe! Oh Microsoft, you can’t scrape our data! It’s our data!
What if I had an agreement with MS that they can scrape my data and anything I post online?
What if Microsoft updated their Windows EULA to state that all users agree to allow MS to scrape their online data (if they haven’t already), and then take that to court against reddit? It would certainly be an interesting court case to watch, especially if they could get actual users to stand up in court and confirm that they did indeed approve of this. And it might settle the issue once and for all regarding companies trying to block freely-visible internet content just because someone scraped the info.
