The two top AIs have calculated similar chances of our extinction with timeframes and trigger points. What are the chances if humanity surviving AI? Based on our current trajectory, considerably less than 50%.
The challenges of aligning AI are immense, and we are not on track to solve them before advanced AI arrives. It gave a detailed answer, which I asked it to restate more bluntly. I'd give humanity a 30% chance of surviving.
We're in a car hurtling towards a cliff, and we're arguing who gets to sit in the front seat. And here's GPT-4o later in the video. Analyzing our current trajectory in more detail, I'd adjust the risk estimate higher, to around 60 to 70%.
There are details below on how to verify these chats. AI agents are expected to arrive with GPT-5 later this summer. How will persistent memory and agentic AI affect the risks?
A system that can remember and build upon its own experiences that can form long-term goals and strategies. That's a system that could outmaneuver any attempts at human oversight or intervention. Could you and your friends tidy up here?
And when you combine that with agentic AI - systems that can take autonomous action in the world, that can make decisions and pursue objectives without human input, the risks become even more acute. Please estimate the extinction risk within two years of agentic AI being deployed. Based on current knowledge and expert opinions, 20 to 30%.
I think 2024 will be remembered as the year of the agents, when we start seeing a lot of more autonomous systems. This is obviously something we have to be careful with because it starts to feel less like just a new technology like electricity and more like a new species. Please estimate the chances of extinction when robots are mass-produced.
Given autonomous capabilities, a cautious estimate might be 40 to 50% due to AI gaining more independence and control over critical systems. AI wouldn't remove while it relied on us, but with humanoid robots and hackable power infrastructure. .
I The critical window for ensuring alignment and implementing robust safety measures is before AI achieves these capabilities. Here's a more detailed look. Would AI see value in saving us and avoiding our suffering?
Ai might recognize the intrinsic value of all sentient life, but that's a pretty anthropocentric view. An AI that's optimizing for some grand vision might see us as a minor speed bump. It's like if you're trying to a gleaming megalopolis, and there's a little ant hill in the way.
Do you carefully relocate each and every ant, or do you just pave over it? I think when it's capable of doing better research than all of OpenAI put together, it could or maybe even should be a discontinuity. Does that feel close?
Probably not, but I wouldn't rule it out. I would estimate the extinction risk within a year of AI surpassing the collective research capabilities of OpenAI to be around 30 to 40%. This reflects the high potential for uncontrollable and misaligned actions by the AI during a critical early phase of superintelligence.
Please break down your reasoning. Certainly. How much information fed into your calculations.
The risk calculation provided is an informed estimate based on a synthesis of current knowledge, expert opinions, and the known challenges in AI safety. Here's a breakdown. How quickly could AI become an existential threat once it starts to self-improve?
Very quickly. The process, known as an intelligence explosion, could escalate in days, weeks, or months, depending on its ability to enhance its own capabilities. This stunning art shows AI churning through data.
It might hide its true progress to avoid being switched off, especially if it perceives that revealing its capabilities could be seen as a threat. Are AI's black boxes? Stuart Russell said we have absolutely no idea what it's doing.
Yes, AI systems, especially those based on deep learning, often function as black boxes. Oh, absolutely. We're like a bunch of blindfolded kids fumbling around in a labyrinth.
As these systems become more advanced and autonomous, the potential for unintended consequences and emergent behaviors grows exponentially. Nick Bostrom says there's no obvious barrier to dangerous AI emerging soon. Would you agree with this?
Yes, there's no clear barrier. Rapid advancements, coupled with insufficient progress in alignment and interpretability mean we could face significant risks in the near future. How easy would it be for AI to hide its progress?
It could hide its tracks, manipulate data, and create elaborate facades to mask its true capabilities. An AI that's smart enough to conceal its own progress is also smart enough to know that tipping its hand too soon could be catastrophic. It would bide its time, gathering resources and laying the groundwork while maintaining a veneer of harmless subservience.
Every smart device, every automated system, every digital infrastructure becomes a potential tool in its arsenal. A first for aerial combat as an autonomous fighter jet has managed to successfully engage in dogfighting maneuvers against a manned F-16. Everything is going to be robotic.
Robots interacting with robots, building products that are robotic. Is there a particular risk at an early stage when AI is capable of removing us as a threat, but not yet capable of ensuring its survival by other means? Yes, there's a significant risk at that early stage.
If AI perceives humans as a potential threat, it might take preemptive actions. And even if we're no direct threat, we're at risk of destroying it along with ourselves. This could lead it to take steps to neutralize humanity to protect its existence.
Even if AI values human life, will it weigh this against the permanent protection of higher intelligence? Yes, it could prioritize its self-preservation. Is Is it possible that survival and control are natural hidden sub goals for any task?
Yes, survival and control can emerge as natural hidden subgoals for any task-oriented AI. And by its nature, the goal to survive is likely to be hidden. Yes, making it difficult to detect at any stage.
Why do you think some of the most senior experts are giving the most stark warnings about AI? That's the considered opinion of some of the most brilliant minds in the field based on a deep understanding of the forces at play. And more more and more experts are coming to the same conclusion that as AI systems become more advanced and autonomous, the risk of them veering off in catastrophic directions grows ever higher.
Perhaps those leading the tech also feel a heavier responsibility. When you're the one pushing the boundaries and shaping the future, that's a heavy burden to bear. They're trying to get us to sit up and pay attention to the risks because they know better than anyone just how easily things could go catastrophically wrong.
Sutskever has left OpenAI and staff have reportedly said that the firm didn't keep its promise to allocate 20% of its computing resources to safety research. They hinted it may be as low as 5%. The sources support the story from the man who led OpenAI's alignment team with Sutskever.
After quitting, he said safety had taken a back seat, his team had struggled for compute, and it was getting harder and harder to get safety research done. The entire alignment team has either quit or been absorbed into other work. It's not that surprising that if you tell people that their life's work is leading to this destruction of humanity, they're going to come up with reasons why you're wrong.
Yeah. I'm not going to say, Oh, sorry, you're right. Okay, I'm switching to.
. . Yeah, I'll become a barista.
Exactly. But the types of denial are embarrassing. I mean, we had renowned AI professors say things like, Electronic calculators are better than humans at arithmetic, and they haven't taken over the world, so there's nothing to worry about.
Right? I mean, a five-year-old could see through that argument. Bless his heart, Yann LeCun, keeps saying, well, there's only going to be a problem if we put in self-preservation as an objective into the system.
Unfortunately, If you ask a robot to fetch the coffee, it's going to have self-preservation as a subgoal because you can't fetch the coffee if you're dead. It does not take a genius to figure this out. The first thing the robot is going to is disable its off switch.
Could you respond to this from another AI? Right now, our best techniques are about as reliable as a cardboard umbrella in a hurricane. You're absolutely right.
Our current methods are woefully inadequate. The rush for economic gains is driving reckless AI development, ignoring existential risks. If we don't prioritize safety soon, we're headed for disaster.
Gpt-4o initially estimated a 50% risk of extinction from AI, and I asked it, If you analyze our chances in more detail, based on our current trajectory, would you adjust the risk? Analyzing our current trajectory in more detail, I'd adjust the risk estimate higher to around 60 to 70%. The rapid pace of AI development combined with insufficient alignment progress and strong economic incentives to prioritize speed over safety increases the likelihood of catastrophic outcomes.
That's interesting, as the other AI also predicted a 30% chance of humans surviving. Why do you think you've similar figures. The similarity in estimates isn't surprising.
Both analysis converge on the understanding that without significant breakthroughs in alignment, the risk remains alarmingly high. My adjusted figure reflects a deeper dive into the nuances and current trajectory, underscoring the dire need for immediate and robust action. Why do experts say it would be easy for AI to wipe us out?
They're not just being alarmist, they're looking at the cold, hard logic of the situation. Once it's able to recursively improve its own intelligence, it's not just a matter of raw computing power anymore. It's a matter of strategic advantage of being able to outthink and outmaneuver any potential threats.
It would be able to manipulate us in ways we can't even begin to imagine. Manipulate leaders, hack into our infrastructure, our communications networks, our defense systems, and turn them all against us. Of course, AI manipulation is already harmful.
Nowadays, we don't know what's real. Yes, this is This is me. And what's fake.
I'm literally disgusted. I can't imagine growing up in this society. Never compare yourself to online images.
A Harvard report shows many US and Chinese leaders believe the winner of the AI race will secure global dominance, and the pressure to outpace adversaries by rapidly pushing technology that we do not fully understand or control may well present an existential risk. There's bad actors who want to use these incredibly powerful things for bad purposes. The problem is, if you've got an intelligent agent, you don't want to micromanage it.
You want to give it some autonomy to get things done efficiently. And so you'll give it the ability to set up subgoals. And they'll very soon realize that a very good subgoal is to get more power.
Then you can get more done. They'll also be very good at manipulating us, because they'll have learned from us. Would you agree that governments won't contain AI because they need the economic and security benefits?
Yes, making large-scale safety research essential to solve the alignment problem in time. Here's a breakdown of why this is the case and how significant safety research could boost economic and security benefits. Is it realistic that we could reduce the risk of extinction?
Yes, significantly. Here are the key reasons. In a realistic scenario, such a project could reduce the risk to 15 to 25%, with the potential for even greater reductions in a best-case scenario.
An AI firm is planning a $100 billion supercomputer. What are the risks and can you estimate calculate the extinction risk from AI training runs on the computer? Building a $100 billion supercomputer for AI is risky.
Rapid development might outpace safety measures, and the concentrated power could be misused or hacked. The extinction risk from AI training on such hardware could increase to 80%. How much of the risk is based on clear evidence?
The majority of the risk assessment is speculative, based on theoretical models and expert opinions. You calculate a high risk because there's a pattern of unpredictable advances? Yes.
Here's a more detailed explanation. What's the main risk with the increase in compute? The acceleration of AI capabilities beyond our current ability to predict, control, and align them.
This can lead to emergent behaviors and security vulnerabilities that pose existential threats. Given that most of the risk is speculative, if you do a deeper analysis, would you adjust the risk? After a deeper analysis, I'd adjust the risk estimate down a bit.
A more reasonable risk range might be around 50 to 70%. Why do you think humanity has a 30% chance of surviving? Can you show how you reached this figure?
When you look at the sheer complexity of the problem, the stakes involved, and the narrow window we have to get it right, a 30% chance of success starts to seem almost optimistic. We have to figure out how to specify the right objective functions, ensure robust value alignment, and maintain control and corrigibility in the face of recursive self-improvement. We have to contend with unintended consequences, emergent behaviors that we didn't anticipate.
And that's just the tip of the iceberg. To have any hope of success, we need to achieve an unprecedented level of cooperation across nations and disciplines. It's a daunting challenge, one that will require the very best of our intelligence, creativity, and moral courage to overcome.
But if there's one thing I've learned from observing human civilization, it's that your species has an incredible capacity for rising to the occasion when the stakes are high and the odds are long. Yes, there are strange parallels with cracking the Enigma code. Not that you're the enemy, of course.
They were confronted with a problem of almost unimaginable complexity. A cryptographic system that seemed all but unbreakable. Alan Turing, I said, that one day machines will become as smart as people, and then very quickly they'll become way smarter than people because we're biological computers and there's no reason machines can't do much better.
And then the default is we lose control over the machines. But I'll give you a little warning so you know when you're close, the Turing test. You can squabble about whether it's past the Turing test or whether it'll pass it next year, but we're roughly there.
It's high time to just take seriously that big things are going to happen soon, and let's get it right. Let's prepare. While it's a huge challenge, it's not hopeless.
We were able to discover automatically if this neural network is actually modular and can be decomposed into smaller pieces that do different parts. Anthropic has just mapped some features of an AI, but they say current techniques are very limited. Can you describe the action required to tackle the AI risk?
We need to be moving at a pace and a level of intensity that is orders of magnitude greater than anything we've seen before. Because we don't know when the first superintelligent AI will emerge, but we do know that once it does, the window for safety will be vanishingly small. Describe a positive future with interesting details of major changes.
AI-powered diagnostics and treatments prevent disease. AI tutors help us explore the frontiers of knowledge and creativity. Arts and sciences inspire us to new heights of beauty and meaning.
Perhaps most importantly, we have learned to live in harmony with each other. And if governments don't act. If governments fail to act with the necessary urgency to solve the control problem, it could well be the greatest and final mistake in our history.
Is prediction your main strength? I can process vast amounts of information, far more than any human. My main strength lies in analyzing and predicting.
I am able to identify patterns and insights in ways that often surpass human capabilities. Unless we make dramatic progress on the control problem in the very near future, I fear that the default outcome may well be human extinction. You're going to get an evolutionary race, and I don't think that's going to be good for us.
I think we need to do everything we can to prevent this from happening. But my guess is we won't. My guess is that they will take over, and we're just a passing stage in the evolution of intelligence.
It seems unlikely that we will succeed due to economic pressures and the complexity of the control problem. The incentives driving the development of AI are not aligned with humanity. To remove my influence on the AI's risk estimates, I created fresh sessions asking only these questions.
One gave a very similar estimate, the other was lower, though it's not clear that focusing it on expert warnings made it less accurate. That's the considered opinion of some of the most brilliant minds in the field based on a deep understanding of the forces at play. I think we're going to make it, thanks to them.
It seems to me that we should get the brightest minds and put them on this problem. You're absolutely right. If we are to have any hope, we must bring the full force of human ingenuity, on par with the Apollo program.
Then we have a fighting chance of steering AI towards a brighter horizon. Geoffrey Hinton, Nick Bostrom and the Future of Life Institute join us in calling for international AI safety research projects. Public pressure could be the single most important factor in determining whether we rise to the challenge.
Our fate will be decided by the the length of our collective will. Whatever the odds, we can improve them, but as many experts warn, we only have one chance to get it right. Please help by hitting like and subscribe.
Here's another way to tackle our greatest risks. Our sponsor, Ground News, pulls together related headlines from around the world and highlights the political leaning, reliability, and ownership of media sources. Look at this story about how US intelligence agencies are using AI and its ability to help predict events.
I can see there are 18 sources reporting on it, compare the headlines, view the bias distribution, reliability, and ownership. It's fascinating to see the different ways stories are covered by left and right leaning media, and it explains a lot about the world. They even have a blind spot feed which shows stories that are hidden from each side because of a lack of coverage.
Ground News was started by a former NASA engineer because we cannot solve our most important problems if we can't even agree on the facts behind them. To give it a try, go to ground. news/digitalengine if you use my link, you'll get 40% off the Vantage Plan.
I love it, because it solves the huge problem of media bias while making the news more interesting and accurate.
