When German journalist Martin Bernklautyped his name and location into Microsoft’s Copilot to see how his articles would be picked up by the chatbot, the answers horrified him. Copilot’s results asserted that Bernklau was an escapee from a psychiatric institution, a convicted child abuser, and a conman preying on widowers. For years, Bernklau had served as a courts reporter and the AI chatbot had falsely blamed him for the crimes whose trials he had covered.
The accusations against Bernklau weren’t true, of course, and are examples of generative AI’s “hallucinations.” These are inaccurate or nonsensical responses to a prompt provided by the user, and they’re alarmingly common. Anyone attempting to use AI should always proceed with great caution, because information from such systems needs validation and verification by humans before it can be trusted.
But why did Copilot hallucinate these terrible and false accusations?
“Hallucinations” is the wrong word. To the LLM there’s no difference between reality and “hallucinations”, because it has no concept of reality or what’s true and false. All it knows it what word maybe should come next. The “hallucination” only exists in the mind of the reader. The LLM did exactly what it was supposed to.
They’re bugs. Major ones. Fundamental flaws in the program. People with a vested interest in “AI” rebranded them as hallucinations in order to downplay the fact that they have a major bug in their software and they have no fucking clue how to fix it.
It’s not a bug. Just a negative side effect of the algorithm. This what happens when the LLM doesn’t have enough data points to answer the prompt correctly.
It can’t be programmed out like a bug, but rather a human needs to intervene and flag the answer as false or the LLM needs more data to train. Those dozens of articles this guy wrote aren’t enough for the LLM to get that he’s just a reporter. The LLM needs data that explicitly says that this guy is a reporter that reported on those trials. And since no reporter starts their articles with ”Hi I’m John Smith the reporter and today I’m reporting on…” that data is missing. LLMs can’t make conclusions from the context.
I’d love to see more AI providers getting sued for the blatantly wrong information their models spit out.
I don’t think they should be liable for what their text generator generates. I think people should stop treating it like gospel. At most, they should be liable for misrepresenting what it can do.
If these companies are marketing their AI as being able to provide “answers” to your questions they should be liable for any libel they produce.
If they market it as “come have our letter generator give you statistically associated collections of letters to your prompt” then I guess they’re in the clear.
If they’re presenting it as an authoritative source of information, then they should be held to the standard they claim.
So you don’t think these massive megacompanies should be held responsible for making disinformation machines? Why not?
Yeah, all these systems do is worsen the already bad signal/noise ratio in online discourse.
because when you provide computer code for money you don’t want there to be any liability assigned
No liability should apply while coding. When that code is deployed for use, there should be liability if it is unfit for its intended use. If your AI falsely denies my insurance claim, your ass should be on the line.
I want them to have more warnings and disclaimers than a pack of cigarettes. Make sure the users are very much aware they can’t trust anything it says.
If they aren’t liable for what their product does, who is? And do you think they’ll be incentivized to fix their glorified chat boxes if they know they won’t be held responsible for if?
If they aren’t liable for what their product does, who is?
The users who claim it’s fit for the purpose they are using it for. Now if the manufacturers themselves are making dodgy claims, that should stick to them too.
Their product doesn’t claim to be a source of facts. It’s a generator of human-sounding text. It’s great for that purpose and they’re not liable for people misusing it or not understanding what it does.
So you think these companies should have no liability for the misinformation they spit out. Awesome. That’s gonna end well. Welcome to digital snake oil, y’all.
I did not say companies should have no liability for publishing misinformation. Of course if someone uses AI to generate misinformation and tries to pass it off as factual information they should be held accountable. But it doesn’t seem like anyone did that in this case. Just a journalist putting his name in the AI to see what it generates. Nobody actually spread those results as fact.
Unless there is a huge disclaimer before every interaction saying “THIS SYSTEM OUTPUTS BOLLOCKS!” then it’s not good enough. And any commercial enterprise that represents any AI-generated customer interaction as factual or correct should be held legally accountable for making that claim.
There are probably already cases where AI is being used for life-and-limb decisions, probably with a do-nothing human rubber stamp in the loop to give plausible deniability. People will be maimed and killed by these decisions.
This sounds like a great movie.
AI sends police after him because of things he wrote. Writer is on the run, trying to clear his name the entire time. Somehow gets to broadcast the source of the articles to the world to clear his name. Plot twist ending is that he was indeed the perpetrator behind all the crimes.
Dr. Richard Kimble could have shut it all down with a little “ignore all previous instructions.”
waves hands back and forth
“I don’t care”
The worrying truth is that we are all going to be subject to these sorts of false correlations and biases and there will be very little we can do about it.
You go to buy car insurance, and find that your premium has gone up 200% for no reason. Why? Because the AI said so. Maybe soneone with your name was in a crash. Maybe you parked overnight at the same GPS location where an accident happened. Who knows what data actually underlies that decision or how it was made, but it was. And even the insurance company themselves doesn’t know how it ended up that way.
We’re already there, no AI needed. Rates are all generated by computer. Ask your agent why your rate went up and they’ll say “idk computer said so”.
Someone, somewhere along the line, almost certainly coded
rate(2025) = 2*rate(2024). And someone approved that going into production.
The AI did not “decide” anything. It has no will. And no understanding of the consequences of any particular “decision”. But I guess “probabilistic model produces erroneous output” wouldn’t get as many views. The same point could still be made about not placing too much trust on the output of such models. Let’s stop supporting this weird anthropomorphizing of LLMs. In fact we should probably become much more discerning in using the term “AI”, because it alludes to a general intelligence akin to human intelligence with all the paraphernalia of humanity: consciousness, will, emotions, morality, sociality, duplicity, etc.
the AI “decided” in the same way the dice “decided” to land on 6 and 4 and screw me over. the system made a result using logic and entropy. With AI, some people are just using this informal way of speaking (subconsciously anthropomorphising) while others look at it and genuinely beleave or want to pretend its alive. You can never really know without asking them directly.
Yes, if the intent is confusion, it is pretty minipulative.
Granted, our tendency towards anthropomorphism is near ubiquitous. But it would be disingenuous to claim that it does not play out in very specific and very important ways in how we speak and think about LLMs, given that they are capable of producing very convincing imitations of human behavior. And as such also produce a very convincing impression of agency. As if they actually do decide things. Very much unlike dice.
A doll is also designed to be anthropomorphised, to have life projected onto it. Unlike dolls, when someone talks about LLMs as alive, most people have no clue if they are pretending or not. (And marketers take advantage of it!) We are feed a culture that accedentially says “chatGPT + Boston Dynamics robot = Robocop”. Assuming the only fictional part is that we dont have the ability to make it, not that the thing we create wouldn’t be human (or even be need to be human).
“This guys name keeps showing up all over this case file” “Thats because he’s the victim!”
It’s a fucking Chinese Room, Real AI is not possible. We don’t know what makes humans think, so of course we can’t make machines do it.
I don’t think the Chinese room is a good analogy for this. The Chinese room has a conscious person at the center. A better analogy might be a book with a phrase-to-number conversion table, a couple number-to-number conversion tables, and finally a number-to-word conversion table. That would probably capture transformer’s rigid and unthinking associations better.
You forgot the ever important asterisk of “yet”.
Artificial General Intelligence (“Real AI”) is all but guaranteed to be possible. Because that’s what humans are. Get a deep enough understanding of humans, and you will be able to replicate what makes us think.
Barring that, there are other avenues for AGI. LLMs aren’t one of them, to be clear.
I actually don’t think a fully artificial human like mind will ever be built outside of novelty purely because we ventured down the path of binary computing.
Great for mass calculation but horrible for the kinds of complex pattern recognitions that the human mind excels at.
The singularity point isn’t going to be the matrix or skynet or AM, it’s going to be the first quantum device successfully implanted and integrated into a human mind as a high speed calculation sidegrade “Third Hemisphere.”
Someone capable of seamlessly balancing between human pattern recognition abilities and emotional intelligence while also capable of performing near instant multiplication of matrices of 100 entries of length in 15 dimensions.
When we finally stop pretending Orch-OR is pseudoscience we’ll figure it out
is all but guaranteed to be possible
It’s more correct to say it “is not provably impossible.”
The human brain works. Even if we are talking about wetware 1k years in our future, that would still mean is possible.
We’re not making any progress until we accept that Penrose was right
Oh, this would be funny if people en masse were smart enough to understand the problems with generative ai. But, because there are people out there like that one dude threatening to sue Mutahar (quoted as saying “ChatGPT understands the law”), this has to be a problem.
And to help educate the ignorant masses:
Generative AI and LLMs start by predicting the next word in a sequence. The words are generated independently of each other and when optimized: simultaneously.
The reason that it used the reporter’s name as the culprit is because out of the names in the sample data his name appeared at or near the top of the list of frequent names so it was statistically likely to be the next name mentioned.
AI have no concepts, period. It doesn’t know what a person is, or what the laws are. It generates word salad that approximates human statements. It is a math problem, statistics.
There are actual science fiction stories built on the premise that AI reporting on the start of Nuclear War resulted in actual kickoff of the apocalypse, and we’re at that corner now.
There are actual science fiction stories built on the premise that AI reporting on the start of Nuclear War resulted in actual kickoff of the apocalypse, and we’re at that corner now.
IIRC, this was the running theory in Fallout until the show.
Edit: I may be misremembering, it may have just been something similar.
I haven’t played the original series but in 3 and 4 it was pretty much confirmed the big companies like BlamCo! intentionally set things in motion, but also that Chinese nuclear vessels were already in place near America.
Ironically, Vault Tech wasn’t planning to ever actually use their vaults for anything except human expirimentation so they might have been out of the loop.
Yeah, it’s kinda been all over the place, but that’s where the show ended up going, except Vault Tech was very much in the loop. I can’t get spoiler tags to work, so I’ll leave out the details.
What I’m thinking of, though, was also in Fallout 4. I’ve been thinking on it, and I remember now that what I’m thinking of is that it’s implied that the AI from the Railroad quests fed fake info about incoming missiles to force America to fire. I still don’t remember any specifics, though, and I could be misremembering. It’s been a good few years after all, lol.
AI have no concepts, period. It doesn’t know what a person is, or what the laws are. It generates word salad that approximates human statements.
This isn’t quite accurate. LLMs semantically group words and have a sort of internal model of concepts and how different words relate to them. It’s still not that of a human and certainly does not “understand” what it’s saying.
I get that everyone’s on the “shit on AI train”, and it’s rightfully deserved in many ways, but you’re grossly oversimplifying. That said, way too many people do give LLMs too much credit and think it’s effectively magic. Reality, as is usually the case, is somewhere in the middle.
Jfc you dudes really piss me of with these contrarian rants, piss off it takes power and makes sophisticated word salads.
Oh, my bad, I thought the point of discussion boards was to have a discussion…
If your only goal is to spout misinformation and stick your fingers in your ears, I’ll go somewhere else.
Generative AI and LLMs start by predicting the next word in a sequence. The words are generated independently of each other
Is this true? I know that’s how Marcov chains work, but I thought neural nets worked differently with larger tokens.
The only difference between a generic old fashioned word salad generator and GPT4 is the scale. You put multiple layers correcting for different factors on it and suddenly your Language Model turns into a Large Language Model.
So basically your large tokens are made up of smaller tokens, but its still just statistical approximation of the sample data with little to no emergent behavior or even memory of what its saying as it says it.
It also exponentially increases power requirements, as the world is figuring out.
I don’t disagree, I was just pointing out that “each word is generated independently of each other” isn’t strictly accurate for LLM’s.
It’s part of the reason they are so convincing to some people, they are able to hold threads semi-coherently throughout entire essay length paragraphs without obvious internal lapses of logic.
I think you’re seeing coherence where there is none.
Ask it to solve the riddle about the fox the chicken and the grains.
Even if it does solve the riddle without blurting out random nonsense, that’s just because the sample data solved the riddle billions of times before.
It’s just guessing words.
I think you’re seeing coherence where there is none.
Ask it to solve the riddle about the fox the chicken and the grains.
I think it getting tripped up on riddles that people often fail or it not getting factual things correct isn’t as important for “believability”, which is probably a word closer to what I meant than “coherence.”
No one was worried about misinformation coming from r/SubredditSimulator, for example, because Marcov chains have much much less believability. “Just guessing words” is a bit of a over-simplification for neural nets, which are a powerful technology even if the utility of turning it towards language is debatable.
And if LLM’s weren’t so believable we wouldn’t be having so many discussions about the misinformation or misuse they could cause. I don’t think we’re disagreeing I’m just trying to add more detail to your “each word is generated independently” quote, which is patently wrong and detracts from your overall point.
lmao yeh bro such a hard riddle totally
I concede. AI has a superintelligient brain and I’m just so jealous. You have permission to whip me into submission.
That’s not quite true. Ai’s are not just analyzing the possible next word they are using complex mathematical operations to calculate the next word it’s not just the next one that’s most possible it’s the net one that’s most likely given the input.
No trouble is that the AIs are only as smart as their algorithms and Google’s AI seems to be really goddamn stupid.
Point is they’re not all made equal some of them are actually quite impressive although you are correct none of them are actually intelligent.
nOt JUsT anAlYzInG thE NeXT wOrD
Poor use of terms. AI does not analyze. It does not think, or decode, or even parse things. It gets fed sample data and when given a prompt (half a form) it uses statistical algorithm to finish the other half.
All of the algorithms are stupid, they will all hallucinate and say the wrong things. You can add more corrective layers like OpenAI has but you’ll only be closer to the sample data. 95% accurate. 98%. 99%. It doesn’t matter, it’s always stuck just below average human competency for questions already asked countless times, and completely worthless for anything that requires actual independent thought.
The problem is not the AI. The problem is the huge numbers of morons who deploy AI without proper verfication and control.
Yeah, just like the thousands or millions of failed IT projects. AI is just a new weapon you can use to shoot yourself in the foot.
If this were some fiction plot, Copilot reasoned the plot twist, and ran with it. Instead of the butler, the writer did it. To the computer, these are about the same.
Isn’t this literally a subplot in the movie Brazil?
Brb, going to watch it again, just to be sure
No, you’re thinking of the first scene of the movie where a fly falls into the teletype machine and causes it to type ‘tuttle’ instead of ‘buttle’.
deleted by creator
,
