Bill Gates feels Generative AI has plateaued, says GPT-5 will not be any better::The billionaire philanthropist in an interview with German newspaper Handelsblatt, shared his thoughts on Artificial general intelligence, climate change, and the scope of AI in the future.
I’m not sure I’d say it’s plateaued today but I definitely think machine learning is going to hit a wall soon. Some tech keeps improving until physical limits stop progress but I see generative AI as being more like self-driving cars where the “easy” parts end up solved but the last 10% is insanely hard.
There’s also the economic reality of scaling. Maybe the “hard” problems could, in theory, be easily solved with enough compute power. We’ll eventually solve those problems but it’s going to be on Nvidia’s timeline, not OpenAI’s.
Yes, especially when you consider that the human brain runs on 15W of power!
Let me save you a click: he doesn’t say anything interesting about why he thinks this.
Cool, Bill Gates has opinions. I think he’s being hasty and speaking out of turn and only partially correct. From my understanding, the “big innovation” of GPT-4 was adding more parameters and scaling up compute. The core algorithms are generally agreed to be mostly the same from earlier versions (not that we know for sure since OpenAI has only released a technical report). Based on that, the real limit on this technology is compute and number of parameters (as boring as that is), and so he’s right that the algorithm design may have plateaued. However, we really don’t know what will happen if truly monster rigs with tens-of-trillions of parameters are used when trained on the entirety of human written knowledge (morality of that notwithstanding), and that’s where he’s wrong.
The problem is that between gpt 3 and 4 there is massive increase in number of parameters, but not massive increase in its abilities
Yeah and I think he may be scaling to like true AGI. Very possible LLMs just don’t become AGI, you need some extra juice we haven’t come up with yet, in addition to computational power no one can afford yet.
Except that scaling alone won’t lead to AGI. It may generate better, more convincing text, but the core algorithm is the same. That “special juice” is almost certainly going to come from algorithmic development rather than just throwing more compute at the problem.
See my reply to the person you replied to. I think you’re right that there will need to be more algorithmic development (like some awareness of its own confidence so that the network can say IDK instead of hallucinating its best guess). Fundamentally though, llm’s don’t have the same dimensions of awareness that a person does, and I think that that’s the main bottleneck of human-like understanding.
My hypothesis is that that “extra juice” is going to be some kind of body. More senses than text-input, and more ways to manipulate itself and the environment than text-output. Basically, right now llm’s can kind of understand things in terms of text descriptions, but will never be able to understand it the way a human can until it has all of the senses (and arguably physical capabilities) that a human does. Thought experiment: Presumably you “understand” your dog - can you describe your dog without sensory details, directly or indirectly? Behavior had to be observed somehow. Time is a sense too. EDIT: before someone says it, as for feelings I’m not really sure, I’m not a biology guy. But my guess is we sense our own hormones as well
First, they do have senses. For example, many LLMs can “see” images. Second, they’re actually pretty good at describing things. What they’re really bad at is analysis and logic, which is not related to senses at all.
I’m not so convinced that logic is completely unrelated to the senses. How did you learn to count, add, and subtract mentally? You used your fingers. I don’t know about you, but even though I don’t count my fingers anymore I still tend to “visualize” math operations. Would I be capable of that if I were born blind? Maybe I’d figure out how to do the same thing in a different dimension of awareness, but I have no doubt that being able to conceptualize visually helps my own logic. As for more complicated math, I can’t do that mentally either, I need a calculator and/or scratch paper. Maybe analogues to those can be implemented into the model? Maybe someone should just train a model on khan academy videos, and it’ll pick this stuff up emergently? I’m not saying that the ability to visualize is the only roadblock though, I’m sure that improvements could be made to the models themselves, but I bet that it’ll be key to human-like reasoning
That’s a good point.
I’ll listen to his opinions more than some, but unfortunately this article doesn’t say anything interesting about why he has this opinion. I guess the author supposes we will simply regard him as an oracle on name recognition alone.
“GPT-4 should be enough for anyone.” -Bill Gates
Not a single comment yet stating how Gates is a great human being because of his foundation, and how all you haters should fuck the fuck off? sigh, let me the first one.
Just to make things extremely clear, the above comment has been sarcastic. He’s an awful person.
They’re upset he insulted their AI girlfriends.
You mean his tax haven?
I mean they’ve done some good things, but the capitalist system that gave him his wealth is the same one that causes poverty and his foundation isn’t working to change that.
bill is a wanker. dont be like bill.
Now now. He only hired assholes and monsters to execute immoral MS mob style tactics, while he played the great innocent altruist.
Bill Gates views on AI are about as insightful as Gordon Ramsey’s on orbital mechanics.
deleted by creator
I think he could be right about generative AI, but that’s not a serious problem given we’re moving beyond generative AI and into virtual intelligence territory.
Generative ai right now requires someone (or something) to initiate it with a prompt, but according to some of the latest research papers in OpenAI as well as the drama that happened recently surrounding the leadership, it appears that we’re moving beyond the ‘generative’ phase into the ‘virtual intelligence’ phase.
It’s not going to be ‘smart’ it will be knowledgeable (and accurate, hopefully). That is to say VI’s will be useful as data retrieval or organization but not necessarily data creation (although IIRC the way to get around this would be to develop a VI that specifically only works on creating ideas but we’d be moving into AGI territory and I don’t expect we’ll have serious contenders for AGI for another decade at least).
The rumours abound surrounding the OpenAI drama, the key one being the potential for accidentally developing AGI internally (I doubt this heavily). The more likely reason is that the board of directors had a financial stake in Nvidia and when they found out altman was working on chips specifically for AI that were faster, lower cost, and lower power consumption than current nvidia trash (by literally tens of thousands of dollars), they fired him to try and force the company onto their preferred track (and profit in the process, which IMO, kind of ironic that a non-profit board of directors has so many ‘closed door’ discussions with nvidia staff…)
This is just the thoughts of a comp-sci student with a focus on artificial intelligence systems.
If interested in further reading:
https://www.ibm.com/blog/understanding-the-different-types-of-artificial-intelligence/
https://digitalreality.ieee.org/publications/virtual-intelligence-vs-artificial-intelligence
Keep in mind that because it’s still early days in this field that a lot of terms haven’t reached an established consensus across academia yet, so you’ll notice variations in how each organization explains what “x” type of intelligence is.
Bill, you’re not even AT Microsoft anymore.
Edit: AND, you think a box of Pizza Rolls cost nearly 3 times as much as they actually do. I’m not trusting your opinion, man
Why does him being rich mean that his opinion on LLMs is invalid? If anything, LLMs require such vast amounts of money that he has more experience in that area because of his riches, not less.
I’ve been saying this for years!
Maybe, but I am sure the tools the AIs can use will improve making the AIs jobs easier and thus the AI more efficient. I hope he is right tbh.
Eww, as a long time Linux user I need to take a shower now. I feel dirty.
The next big steps coming right now are AI trained on generative data and agents that act more automatically (rather than waiting for a prompt, take an action like searching the web and act on that to better complete the goal for example), and better indexed data so generated data can be informed by and cite sources in the moment.
AI trained on generative data
This has already been shown to degrade the output very quickly.
I think the wall that generative AI is hitting is the lack of more training data. All the web has been scraped to get it to where it is today, more and more content on the web is itself generated by AI and therefore not only useless but harmful if used as training data.Orca 2 is an example of an opensource model that was built to better collect and build on synthetic data: https://www.microsoft.com/en-us/research/publication/orca-progressive-learning-from-complex-explanation-traces-of-gpt-4/
The case I think being made is that building LFMs on the Internet gets you closer to an average internet users level of our put, using reinforcement learning you can further curate the outputs, then finally using these techniques you can generate even tighter high quality models.
It’s interesting stuff for sure.
Since I started using AI in March, it has not stop getting better and better with it seemingly accelerating not decelerating. If we are heading to a plateau, there are no signs of it yet. I am sure there will be plateaus, and maybe we have even had one earlier this year, but with the speed of AI development, it may mean a plateau of months versus the normal years. We will see. Bill has been saying a lot of things these past years that make no sense, unless you look at it from a money/market manipulation perspective.
And the Wright Brothers said heavier than air flight would only ever be an amusement for the rich, and never commercially viable.
Even taking Gates’ qualifications at face value doesn’t mean he’s actually right.
Source on that quote from the Wright brothers? Because they never said that as far as I’m aware.
They didn’t, AFAIK. It was a NYT article that quoted someone who made a similar prediction:
Once the Wright Brothers proved flight was possible, some assumed it was just a pointless rich play thing. Famed astronomer William H. Pickering said, “The expense would be prohibitive to any but the capitalist who could use his own yacht.”
You didn’t read the Wright brothers or this article did you? Gates isn’t at all damning AI tech. All he said is that gpt 5 is unlikely to be very different from 4. He’s probably correct. A next best word algorithm can only go so far. Thats only a part of how language and cognition works. Until some sort of adjunct algorithm gets tagged on I don’t think we will see big leaps either.
They are already bolting on multi modality, autonomous agent behavior and so on. Most uses are no longer just token prediction, but a whole soup of other models and data injection. That sort of combination of things will get us to the next level )and already has in many ways).
And 640k ought to be enough for anybody.
deleted by creator
He just has money, which gives him and too many others the idea that he has expertise.
And a voice that should be heard
Bill gates can fuck off.
