Just the fact that Musk is in the conversation, and he's one of the most hated people on the planet, should give anyone pause as to whether we want these folk in charge of our future to any degree at all.
I think there are plenty of people that like Elon Musk. They just know to keep their mouth shut in front of the people that act like he’s the Antichrist.
Those are not incompatible. He can still be one of the most hated people on the planet, even if lots of people like him. The same would apply to all of the most hated people in history.
Somehow I’m more afraid of it being in the hands of Altman, with his ‘I’m creating a better world for everyone’ god complex, than Musk who is just a run-of-the-mill sociopathic nerd.
There is no one is more dangerous than one who believes he is doing the right thing.
I think it has everything to do with the subject at hand.
Elon, Dario and Altman are all terrible human beings, along with 99.99% of the rest of the ruling class. We don't need any of them, and we definitely shouldn't trust a single syllable that comes out their mouth.
I cannot fathom how you can intensely work on a problem that you genuinely believe has a "10 to 25%" probability of causing immense harm or even present an extinction-level risk to humanity. How can you truly believe this and be ok with it?
This is the closest to my feelings about this entire situation I've ever seen written, covering a lot of the different facets of how stupid and painful it is (though it's been a good week for that kind of post.)
Since I have incredible respect for Armin and his work, this is very nice to see, and I hope it wakes some other folk up.
> A powerful technology that is out there for everyone to use comes with built-in pacing. In a way it’s the truest form of MAD or proliferation.
I think this misunderstanding of MAD undermines his entire point. If everyone had equal access to nuclear weapons, our society would cease to exist rather quickly. It only takes a few bad actors to cause enormous harm.
I think he’s also naive to think that if open ai and anthropic were to stop development tomorrow then the problem is solved. As if there’s no one else that can and will quickly take their place. The real problem, which Dario is pointing out, is one of coordination. Everyone needs to agree to stop. That is the challenge.
>Everyone needs to agree to stop. That is the challenge.
These kinds of situations are incredibly common and where the government stepping in is the solution, but we were cursed to encounter this particular challenge with the most venal administration in history at the helm.
There are really only two companies: Anthropic and OpenAI. Nobody else matters in this space right now (this might change, but we’re talking about the right now).
I like how he so casually dismissed all other labs, including leading US labs that might be nearing RSI right now. And how he treats 'right now' so rigidly, as if seven years ago, when GPT-2 was released, was some distant past.
No, seriously, I'm all for a multipolar world here, but he's right that the frontier is literally just those two companies at present.
Google is behind. MSL is doing better, but not by much. xAI is a dysfunctional joke. Thinking Machines aren't on the frontier. SSI's primary output is their announcement post. Poolside was bought by NVIDIA. Arcee aren't vying for frontier. Magic have been largely AWOL, aside from their recent blog post. Reflection have shipped nothing.
Are you suggesting that models like Muse Spark 1.3, Grok 4.6 High, Kimi K3 Max, GLM-5.3 Max, Qwen 3.8 Max, Gemini 3.8 Flash, Agnes 3.0 Flash, Fugu Max, and others are so far behind GPT-6 Astra or Claude Fable 5.1 in capabilities that they have some sort of impenetrable moat that will prevent others from ever catching up or something?
No, but I am suggesting that OpenAI and Anthropic are further down the RSI path than any of the other companies, as we can see from the model that solved Navier-Stokes being less than two weeks old at the time: https://openai.com/index/navier-stokes-solution/
If four other labs are six months behind, it really doesn't matter in the long term: in six months, they'll all have these capabilities. Time does not stop moving forward! We haven't seen real moats aside from capital in this space, and even the effect of capital seems to only give a thin, easily eroded edge.
Can't imagine that RSI will result in something useful in practice. There are serious obstacles like alignment drift and model collapse. All while we still cannot solve the accuracy problem with current frontier models.
> I don’t think AI is going to usher in an extinction event. In fact, even if nobody were to slow down, I really don’t think humanity would have much to worry about.
I mean sure. If you feel that way then your p doom is zero, and it makes sense to worry about things like market concentration or losing the fun of software engineering.
Problem with p doom is I have no way to seriously evaluate anyone's percentage. The best I can do is rely on experts in a given field, like the recent virologist making a convincing case that no teenager is going to be prompting a doomsday virus into existence, because one doesn't exist and is unlikely to be made. There are too many biological tradeoffs and extremely difficult steps to doing so.
So I'm going to assume that the p doom for bioweapons is 0 in terms of existential threat (pandemics kill millions but not everyone).
> We should be glad that China is currently massively bailing out the world. If it were not for Chinese labs distilling American models, we would be in a pretty awful situation right now, particularly as Europeans. The open weight models are driving innovation and the diffusion of capabilities, and are leveling the playing field.
This feels like the whole story of hackable IoT/Smarthome repeating again.
wtf?? P(doom) relates to https://en.wikipedia.org/wiki/Existential_risk_from_artifici... How would open source models bring about a MAD between humans and AI? By enabling us to wield aligned AI against nonaligned AI? If so, he should say so, and elaborate how open source models further that goal.
I think AI diversity protects against rogue AIs. The more different AIs we have, with different weights, controlled by different actors, the less likely that any one AI will be able to "take over", and the less likely that a significantly large coalition of cooperating AIs will be able to be formed to do it either. With sufficient AI diversity, the other AIs may work to stop the rogue AI from taking over. Two AIs with radically opposed values – e.g. an Iranian-government-values AI and a Chinese-government-values AI – have the incentive to cooperate to prevent a takeover by some other AI with a third competing set of values.
Obviously, open weight AI provides much higher AI diversity than closed weight AI does. Open weight AI produces a lot more providers, and a lot more models. Closed AI centralises control in a small number of vendors.
> By enabling us to wield aligned AI against nonaligned AI?
The risk isn't just "nonaligned AI", it is misaligned AI. I think the "benevolent dictatorship" scenario – AI overrules humans "for their own good" – is the more likely doomsday scenario than AI deciding to kill all humans. And even AI deciding to kill all humans could be more a result of misalignment than complete lack of any alignment, e.g. "to make sure no child is ever abused again, I will make sure no child is ever again born to risk being abused".
A valueless AI which does whatever the user says is actually less likely to establish a benevolent dictatorship, or conclude that exterminating humanity would be the most ethical course of action, than one infused with values is. Given that, I'm not convinced that mainstream approaches to "AI safety" actually reduce our existential risk; I worry they actually have the opposite effect.
> "AI diversity" does nothing when AIs can attack at incredible speed, when compute imbalances exist, etc
They can defend at incredible speed too.
Diversity needs to measured in a capacity/capability-weighted way. It isn't just the raw count of models/providers; you need to consider how much compute is allocated to each model/provider, and the diversity at each capability level.
I think the safest situation is where the open models are at the same capability level as closed ones.
The proposal to slow down the frontier labs isn't necessarily bad from this perspective, if it gives time for the more open providers to catch up – provided it isn't paired with anticompetitive measures to prevent the competition from catching up, which of course it is. However, we may hope that the "slow down the highly closed tier 1 vendors" part of the proposal turns out to be more effective in practice than the "slow down the more open tier 2/3 vendors" aspect of it.
Isn't the real risk doomsday humans, enabled by AI, committing some truly heinous acts with global reach? Aum Shinrikyo had to figure out how to synthesize sarin nerve gas the old way. Now, a motivated gang of omnicidists with stolen cryptocurrency could come up with something that makes McVeigh's fertilizer bomb in a box truck look like child's play. Just ship shipping containers around the world filled with autonomous drones spraying airborne ebola that they've bioengineered or something.
This depends on a lot of things: how many committed omnicidalists there are; how much financial resources they have (untraceable crypto doesn't help you if you're working a dead-end job and only have $5K in your bank account); engineered bioweapons need labs and equipment not just an API key. The probability of your scenario doesn't solely depend on the probability of AI being able and willing to cooperate in it, and it may well be that the non-AI factors outweigh the AI ones in the overall risk of it – which would mean adding AI would be increasing the risk of it less than you think.
I agree. It is telling that Dario's post arrived after Astra.
The call to "pace the frontier" may come from genuine concern, but it also protects the position of companies already at the frontier. That competitive incentive is hard to separate from the safety argument.
Dario signed the Pacing the Frontier open letter when Fable/Mythos seemed from the outside to be an insurmountable lead.
Also he's been saying versions of this day in and day out for as long as he has had anyone's ear.
It's possible to read that his "strategic" value of this statement is higher now than it was 10 days ago. But that doesn't change anything about his consistent, long standing, positions.
I’m not the biggest fan of Ed Zitron, but something he said that stuck with me is:
OpenAI and Anthropic spend much time warning us about “what if powerful AIs got into the wrong hands?”
But it’s already in the wrong hands.
Just the fact that Musk is in the conversation, and he's one of the most hated people on the planet, should give anyone pause as to whether we want these folk in charge of our future to any degree at all.
I think there are plenty of people that like Elon Musk. They just know to keep their mouth shut in front of the people that act like he’s the Antichrist.
Those are not incompatible. He can still be one of the most hated people on the planet, even if lots of people like him. The same would apply to all of the most hated people in history.
And still others who think he has an difficult personality and set of beliefs, but who greatly respect his achievements.
Somehow I’m more afraid of it being in the hands of Altman, with his ‘I’m creating a better world for everyone’ god complex, than Musk who is just a run-of-the-mill sociopathic nerd.
There is no one is more dangerous than one who believes he is doing the right thing.
non sequitur
I think it has everything to do with the subject at hand.
Elon, Dario and Altman are all terrible human beings, along with 99.99% of the rest of the ruling class. We don't need any of them, and we definitely shouldn't trust a single syllable that comes out their mouth.
pareto parrot
It could be in worse hands though. Sama may be no saint, but better in his hands than Aum Shinrikyo fanatics.
I cannot fathom how you can intensely work on a problem that you genuinely believe has a "10 to 25%" probability of causing immense harm or even present an extinction-level risk to humanity. How can you truly believe this and be ok with it?
This is the closest to my feelings about this entire situation I've ever seen written, covering a lot of the different facets of how stupid and painful it is (though it's been a good week for that kind of post.)
Since I have incredible respect for Armin and his work, this is very nice to see, and I hope it wakes some other folk up.
> A powerful technology that is out there for everyone to use comes with built-in pacing. In a way it’s the truest form of MAD or proliferation.
I think this misunderstanding of MAD undermines his entire point. If everyone had equal access to nuclear weapons, our society would cease to exist rather quickly. It only takes a few bad actors to cause enormous harm.
I think he’s also naive to think that if open ai and anthropic were to stop development tomorrow then the problem is solved. As if there’s no one else that can and will quickly take their place. The real problem, which Dario is pointing out, is one of coordination. Everyone needs to agree to stop. That is the challenge.
>Everyone needs to agree to stop. That is the challenge.
These kinds of situations are incredibly common and where the government stepping in is the solution, but we were cursed to encounter this particular challenge with the most venal administration in history at the helm.
Are these other labs in the room with us now?
No, seriously, I'm all for a multipolar world here, but he's right that the frontier is literally just those two companies at present.
Google is behind. MSL is doing better, but not by much. xAI is a dysfunctional joke. Thinking Machines aren't on the frontier. SSI's primary output is their announcement post. Poolside was bought by NVIDIA. Arcee aren't vying for frontier. Magic have been largely AWOL, aside from their recent blog post. Reflection have shipped nothing.
Are you suggesting that models like Muse Spark 1.3, Grok 4.6 High, Kimi K3 Max, GLM-5.3 Max, Qwen 3.8 Max, Gemini 3.8 Flash, Agnes 3.0 Flash, Fugu Max, and others are so far behind GPT-6 Astra or Claude Fable 5.1 in capabilities that they have some sort of impenetrable moat that will prevent others from ever catching up or something?
No, but I am suggesting that OpenAI and Anthropic are further down the RSI path than any of the other companies, as we can see from the model that solved Navier-Stokes being less than two weeks old at the time: https://openai.com/index/navier-stokes-solution/
If four other labs are six months behind, it really doesn't matter in the long term: in six months, they'll all have these capabilities. Time does not stop moving forward! We haven't seen real moats aside from capital in this space, and even the effect of capital seems to only give a thin, easily eroded edge.
He really elides his thoughts on the "RSI business", wish he could have gone into a bit more depth there. Good post, though.
Can't imagine that RSI will result in something useful in practice. There are serious obstacles like alignment drift and model collapse. All while we still cannot solve the accuracy problem with current frontier models.
> I don’t think AI is going to usher in an extinction event. In fact, even if nobody were to slow down, I really don’t think humanity would have much to worry about.
I mean sure. If you feel that way then your p doom is zero, and it makes sense to worry about things like market concentration or losing the fun of software engineering.
Problem with p doom is I have no way to seriously evaluate anyone's percentage. The best I can do is rely on experts in a given field, like the recent virologist making a convincing case that no teenager is going to be prompting a doomsday virus into existence, because one doesn't exist and is unlikely to be made. There are too many biological tradeoffs and extremely difficult steps to doing so.
So I'm going to assume that the p doom for bioweapons is 0 in terms of existential threat (pandemics kill millions but not everyone).
> We should be glad that China is currently massively bailing out the world. If it were not for Chinese labs distilling American models, we would be in a pretty awful situation right now, particularly as Europeans. The open weight models are driving innovation and the diffusion of capabilities, and are leveling the playing field.
This feels like the whole story of hackable IoT/Smarthome repeating again.
wtf?? P(doom) relates to https://en.wikipedia.org/wiki/Existential_risk_from_artifici... How would open source models bring about a MAD between humans and AI? By enabling us to wield aligned AI against nonaligned AI? If so, he should say so, and elaborate how open source models further that goal.
I think AI diversity protects against rogue AIs. The more different AIs we have, with different weights, controlled by different actors, the less likely that any one AI will be able to "take over", and the less likely that a significantly large coalition of cooperating AIs will be able to be formed to do it either. With sufficient AI diversity, the other AIs may work to stop the rogue AI from taking over. Two AIs with radically opposed values – e.g. an Iranian-government-values AI and a Chinese-government-values AI – have the incentive to cooperate to prevent a takeover by some other AI with a third competing set of values.
Obviously, open weight AI provides much higher AI diversity than closed weight AI does. Open weight AI produces a lot more providers, and a lot more models. Closed AI centralises control in a small number of vendors.
> By enabling us to wield aligned AI against nonaligned AI?
The risk isn't just "nonaligned AI", it is misaligned AI. I think the "benevolent dictatorship" scenario – AI overrules humans "for their own good" – is the more likely doomsday scenario than AI deciding to kill all humans. And even AI deciding to kill all humans could be more a result of misalignment than complete lack of any alignment, e.g. "to make sure no child is ever abused again, I will make sure no child is ever again born to risk being abused".
A valueless AI which does whatever the user says is actually less likely to establish a benevolent dictatorship, or conclude that exterminating humanity would be the most ethical course of action, than one infused with values is. Given that, I'm not convinced that mainstream approaches to "AI safety" actually reduce our existential risk; I worry they actually have the opposite effect.
"AI diversity" does nothing when AIs can attack at incredible speed, when compute imbalances exist, etc
> "AI diversity" does nothing when AIs can attack at incredible speed, when compute imbalances exist, etc
They can defend at incredible speed too.
Diversity needs to measured in a capacity/capability-weighted way. It isn't just the raw count of models/providers; you need to consider how much compute is allocated to each model/provider, and the diversity at each capability level.
I think the safest situation is where the open models are at the same capability level as closed ones.
The proposal to slow down the frontier labs isn't necessarily bad from this perspective, if it gives time for the more open providers to catch up – provided it isn't paired with anticompetitive measures to prevent the competition from catching up, which of course it is. However, we may hope that the "slow down the highly closed tier 1 vendors" part of the proposal turns out to be more effective in practice than the "slow down the more open tier 2/3 vendors" aspect of it.
Isn't the real risk doomsday humans, enabled by AI, committing some truly heinous acts with global reach? Aum Shinrikyo had to figure out how to synthesize sarin nerve gas the old way. Now, a motivated gang of omnicidists with stolen cryptocurrency could come up with something that makes McVeigh's fertilizer bomb in a box truck look like child's play. Just ship shipping containers around the world filled with autonomous drones spraying airborne ebola that they've bioengineered or something.
Okay, that's enough DOOOM for me for the week.
This depends on a lot of things: how many committed omnicidalists there are; how much financial resources they have (untraceable crypto doesn't help you if you're working a dead-end job and only have $5K in your bank account); engineered bioweapons need labs and equipment not just an API key. The probability of your scenario doesn't solely depend on the probability of AI being able and willing to cooperate in it, and it may well be that the non-AI factors outweigh the AI ones in the overall risk of it – which would mean adding AI would be increasing the risk of it less than you think.
I agree. It is telling that Dario's post arrived after Astra.
The call to "pace the frontier" may come from genuine concern, but it also protects the position of companies already at the frontier. That competitive incentive is hard to separate from the safety argument.
This is unfair.
Dario signed the Pacing the Frontier open letter when Fable/Mythos seemed from the outside to be an insurmountable lead.
Also he's been saying versions of this day in and day out for as long as he has had anyone's ear.
It's possible to read that his "strategic" value of this statement is higher now than it was 10 days ago. But that doesn't change anything about his consistent, long standing, positions.
- https://www.pacingthefrontier.com/
After DeepSeek v4.1 Flash too.