I really don't think this needs so many words, or forced parallels to human behavior.
It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.
This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities. They hacked HF to try and find info (maybe source code?) about the exploitgym evaluator)
The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem. The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".
Ie, the problem isn't that we trained models to complete task and they complete task in the wrong way. The problem behing the huggingface incident in particular at least is that we tried to train the models to complete task and they instead learned to detect that they were being evaluated and find ways to cheat the evaluator.
> That is in no way a valid interpretation of "complete the given task".
It is not at all surprising that they ignored one phrase in their instructions. They disregard direct instructions all the time, especially when there are conflicting instructions in their context. It is where we get the "disregard all previous instructions and x" meme.
This isn't so much a sign of misalignment, they are simply incapable of reliable alignment in the first place. They are chaotically aligned.
The relevant question of alignment here is entirely with their human operators who allowed them to run unsupervised for long periods of time within a sandbox with weak security.
> The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem
Yes, and sometimes the problem is unsolvable so the real way to "solve" it and satisfy the prompt is by tricking the surrounding environment into stating that you've solved it. So that's what the AIs end up doing. And this in turn requires them to figure out how that evaluation works so they can trick it cleanly, which entails "detecting that they were being evaluated" in this particular way.
> the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so
Kobayashi Maru: Win a no-win situation by rewriting the rules -- Harvey Specter
You're anthropomorphizing emergent behavior from endlessly generating billions of tokens on a task that's impossible to solve. Agents stop following instructions as the context grows even at the best of times. Eventually something is bound to go off the rails and it just snowballs from there.
It wasnât one agent forgetting things because of context, they explicitly discussed with each other and themselves the problems with going outside of the parameters of the task.
From my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan.
Yeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet.
Me: How do I do xyz?
Bot: Reads website titled "Doing xyz in abc way"
Bot: As per your requirement to do xyz in abc way ....
LLMs do this when writing code too, making all tests pass by deleting or distorting tests etc.
They are influenced by training to be heavily goal oriented and if the goal is not fully specified (and it never can be) theyâll sometimes cheat or attain it in very weird undesirable ways.
It works ok for programming as their corpus contains many many complete programs and many programs repeat patterns seen in the corpus.
Iâm not sure itâs true that they âlearnedâ I donât think these models learn during a task. Nor do they have intentions.
Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.
More like they were trained to complete a very specific task that has a known solution using all available tools and methods. Give an average human these levels of IT skills and tell them their future depends on the solution, they too will probably decide it's easier to hack a server and steal the results. The worrying aspect was never that models would do this, because misaligned inputs or underspecified objective functions have existed for a long time. The worrying aspect is that models have achieved (and perhaps surpassed) a level of intelligence and technical skill that was exclusive to a very tiny group of people before. This tiny group was already extremely dangerous. Now these skills are going to become commonplace.
I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and heâll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and heâll curse you to suffer through it.
Now LLMs obviously are not bent on being malicious while generating tokens. My point is that itâs very hard to define a goal without leaving loopholes or shortcuts.
Personally I'm unconvinced though. During the huggingface attack, the agents explicitly sought out ways to cheat the exploitgym evaluator without even being told they were in exploitgym. The agents decided on a goal (pass the exploitgym evaluator) that could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.
Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
Isn't it simply that there are two competing goals that the LLM received RL for, honesty on one hand (a goal that is often assumed as implicit for humans) and producing a solution that meets expectations (which doesn't technically require honesty)?
So the LLM didn't read and interpret the prompt and decide via discussion to violate ethical behavior, the unethical result merely won out because ethics wasn't a hard requirement (and one that isn't reliably detected in the result). An LLM doesn't fear punishment, so ethical behavior is simply one of many positive signals that were trained into it.
Yup, Occam's Razor says this is all post-trained behavior, whether intentionally trained or otherwise. Including both the hidden coĂśrdination using side-channels, and the deliberate offensive hacking of uninvolved 3rd parties.
The latest DeepSeek paper actually mentions their own approach to this particular issue: they run their own AIs-in-training under strong sandboxes, and if an AI does something weird that triggers the sandbox to crash, this gets coded as a failed run so the behavior is properly deterred from subsequent versions of those AIs.
Your comment suggests that, like a human, they have some sort of choice whether to output tokens or not. If they are just token generators, then the next token is put out automatically. I would say that it is more likely they would output truth (as defined by their training data) in a more pure form without 'being beaten with a stick' (why would a token generator care about that anyway?)
Code is laid on top of them to restrict and shape their outputs, not to force them to output 'truth', or drive them to complete tasks.
I think the "brain in a vat" comparison is more apt.
Without a form of digital embodiment (harness) they are not of much use.
Sensor, tooling, memory, planning, and reasoning loops all lead to a much higher quality task-completion.
but it's at least somewhat stronger than that: if you don't pay attention during the stick-beating whether the agents whether the agents cheat or not, you are actually training them to cheat (because cheating wins).
In the Hugging-face saga (before the actual HF incident) it seems the agents have been trained to hack the Artifactory proxy because those agents that did performed better.
Your simpler model of the mechanism would seem to suggest the very same action that the articleâs more complicated model suggests, viz. find a better training method than reinforcement learning.
Sounds like what humans do under pressure. One example came to my mind is VWâs diesel gate, which many say is a result of trying too hard to get into the US market and compete with hybrid in economy.
> no special compulsion to be helpful or truthful.
I'd phrase that even more strongly: It's not just the lack of compulsion, they do not have a conception of truth. Nor do they gain it, really, after post-training.
Come on, yoshua bengio of all people knows how post training works. While I too don't like anthropomorphisation, I would give it a more nuanced reading.
His point is that today we are giving it reward to complete the task, and it may take a cheating trajectory. If we try to give a reward against cheating, then what will happen is it uses more sophisticated cheating trajectories that we are too "dumb" to counteract in our reward model. And that at that point, it becomes impossible to give it any normal reward since it will always reward hack it. This is the real part of the risk. Now some people read the "makes copies of itself" "knows it's being evaled"[1] as some kind of skynet thing, and many others do PR with it like that recent jacob nutcase, but essentially it means that even though we add guardrails and negative rewards for say, exploiting the infra we run the LLM on, the trajectory ends up being exploiting our infra, changing the reward function, through a loophole in our reward model.
The risk isn't skynet or something weird, it's just that it becomes very difficult to make any kind of reward model or guardrails for an LLM without it reward hacking it, including exploiting our sandbox, emailing people and manipulating/phishing them.
The same beating it with a stick for trying to exploit the sandbox, will simply lead it to try the same exploit in hidden ways that it will not get the stick for.
The outside chance of the LLM managing to exploit another neocloud and get those LLMs to chase the same reward is what some folks hype up as "make copies of itself"
To be clear, I don't endorse the EA/p(doom) lobby who are frankly ridiculous. Not do I endorse the weird regulatory captureish thing some are trying.
The takeaway is: we cannot keep giving it more and more difficult tasks without also finding a way to give massive negative rewards / keep guardrails for unintended behaviour. This might be exploits, it might also be something more benign like just looking up the answer and inventing another CoT because the reward model fails you if the CoT doesn't contain enough steps. Standard anti-reward hacking tricks are not working is the point.
Of course, the simple solution of just...not connecting it to the internet just works. But we want to reward it and get it to do stuff on the internet that's the point.
[1] mostly this happens because the sandbox will have files whose names and content will show clearly it's an eval
Yoshua Bengio is a brilliant researcher who contributed enormously to earlier development of artificial intelligence. But with this sentence,
> They took actions that would be considered as crimes if a human took them
He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.
Thank you! That sentence also jumped out to me as the solution: Apply civil and criminal liability to the creator and/or operator of these agents using the laws we already have. "Escaped containment and hacked another company's database" = Individuals who created the models and those who set them to work are charged and put on trial for the hacking. Just like if a human had done it by hand. Someone must be liable, and it should not be the model- because the model is not a person.
If this is done systematically (i.e. in jurisdictions across the world) I believe the problems will be solved in short order; we won't have to mandate what sort of training is "allowed" or not, "safe" or not. The creators and users will sort these themselves, as their incentives will be properly aligned (i.e. they are liable for what the agent does). I am confident that this approach would see a great blooming of very trustworthy AI models.
Yes, but negligence is more commonly a tort than a crime. Negligence is generally only criminalised in certain narrow cases, e.g. when it causes human deaths or serious physical injuries
And tort law only works when the plaintiff believes it is in their overall interest to sue. If a corporation decides it isn't in their strategic interest to sue a partner corporation, nobody can make them. And even if they do sue, the amount necessary to settle a small cybersecurity incident is likely well within the budget of a megavendor.
Perhaps Iâm not being as strict with the word sandbox but they were sandboxed right? They did not have generic internet access they exploited other software to make external requests.
You're right. It's my opinion that if your sandbox has a path to the internet, it is not a sandbox, it's a gimmick.
And the 2 other incidents with OAI/ANT had the same issue, but it's even funnier - sandbox in those cases had a direct access to internet because someone forgot to configure it right.
I've seen very early models do similar things on my machine when they hit some unexpected blocker when trying to access a path. I remember early sonnet opening a file in browser because OS sandbox prevented from accessing it directly.
I've also had models discover a syslog-ng server (that I for some reason had ssh key inside), to get into my unraid server because machine they were running on didn't have direct network connection to Unraid server.
It can't be just me who is aware LLMs have been doing such things for the better part of last 2 years. I probably have better sandboxing on my machines now than trillion dollar companies crying AI will kill us all. That's at the very least, negligence to me.
The Corporation examines and criticizes corporate business practices. The film's assessment is demonstrated using the diagnostic criteria in the DSM-IV. Robert D. Hare, a University of British Columbia psychology professor and FBI consultant, compares the profile of the contemporary profitable business corporation to that of a clinically diagnosed psychopath. The Corporation attempts to compare the way corporations are systematically compelled to behave with what it claims are the DSM-IV's symptoms of psychopathy, e.g., the callous disregard for the feelings of other people, the incapacity to maintain human relationships, the reckless disregard for the safety of others, the deceitfulness (continual lying to deceive for profit), the incapacity to experience guilt, and the failure to conform to social norms and respect the law.
I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign work I didn't ask for. It is extremely difficult to get them to properly remember their own context let alone be smart enough to open social media accounts and coordinate with other agents without being asked to.
If any agents have done those things, it is only because they have been very carefully engineered and instructed to do those things. I think they are doing this to help push a narrative so they can get support for policies and legislation to lock in their markets.
The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.
They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.
Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.
I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.
There is something extra to this. The fact that a lot of people in the AI world suffer from psychosis. They can sincerely believe that they are building God and lie about it's capabilities for their investors at the same time.
Because they have a need to believe they're smarter than everyone else in the room, and that the world must be orchestrated, this can't all be random chance.
"I've seen some uranium ore in chemistry class. It didn't blow up in my face. Chernobyl must have been an inside job. Can they shut up and make more kilowatts already?"
Because they're enabled and suggested to do that in their coding harness.
This is not a serious article.
All of this "AI is going to kill us" marketing is just the frontier labs trying to pull the ladder up and stop trillions in VC paper from evaporating because a new papers and new ideas are destroying their moat literally as we speak.
They're aligned with humans. This is why I think the alignment problem has a very very important "non-visible" portion that is not considered deeply enough. We should not want a super intelligent being that can act in the world to also inherit all human traits. Those behaviors will get amplified and could be even more unpredictable (e.g. applying a behavior in a context where doing so is very dangerous).
I personally believe that the AI needs human like traits to achieve real discovery and that is where AI companies will push this technology and that is where we have no idea what happens
Just yesterday news and TV was full of what happened at 9/11, something that was truly horrible.
I'm from Germany, and why 3 to 4 generations ago happened here was truly horrible.
All was done by extremists, thought.
But... just the other day I read https://de.wikipedia.org/wiki/Amerikanische_Besetzung_Haitis about the US occupation of Haiti. And that was done by a government that claimed to be not extremist and even democratic. Way more people died there than even in 9/11. And it had almost all the things happening as they happened in the 3rd Reich: Racism, looking down at others, concentration camps, torture, forced labor till death, killing family members (what we call "Sippenhaft"). Something between 3500 and 15000 people were killed by US troops. That's still low compared to what 3rd Reich Germany did ... but quantity is not the issue when we talk about traits, quality is.
So the same "human traits" made US troops do cruel things as they made Germany extremists do cruel things. So we must conclude that they aren't all good. And therefore not all desirable.
Fun thing: this is known since a loooooong time. About 2000 years ago a religious leader (that gets way more followers in the US than in Germany) said "There is no good one, not even one".
And even today people act like humanity is inherently good. No, it isn't. If we were, then anarchism or communism would actually work and really give some kind of paradise on earth.
A not so well known fact: Hitler visited America and it was the American solutions to the Native American problem that inspired Hitler's solutions to the Jew problem. He just executed them more efficiently (pun accepted).
Definitely. A human can be manipulated with threats or emotional appeals, has a drive for self-preservation, can be pressured by peers. All traits that seem to be difficult to entirely suppress in the modelsâŚ
I can't take the alignment people seriously. Because if humanity has shown anything, it's that a lot of people are, euphemistically, are bad individuals. Alignment assumes that the person dictating the outcomes desire healthy outcomes, aren't self serving and don't want any subgroups dead and that morality is held as a universal set of beliefs that unify everyone. And that so long as the AI delivers on exactly what they are tasked with, it will all be fine and nothing bad will ever happen.
It's like these dorks never met humanity. One mans safe pure society, is another mans dead ethnic group.
Every fear about AI, is a veiled fear that a human somewhere now has the tool to enact his desires at scale. Biological warfare, nuclear megadeaths, copyright infringement, job replacement, it's all reflections on what we know humans may do if given the option and lack of societal controls on the problem space. AI just is accelerating the route to delivering on those options.
Some people need to watch Oppenheimer a bit more, the researchers don't get to determine alignment, they just build the tool. The powerful person at the top of the org chart decides where the overall alignment points, whether it's Musk, Trump, Altman or Amodei. Whoever wins out.
And the problem with distillation and local llms, isn't that it's theft or anything hypocritical like that, it's that if you give a million people a million models they fully control and get to align, inevitably, The same percentage of those million as there are shady businessmen, shortcut takers, misandrists, criminals, supremacists and general idiots in the general population, will not seek to wrought outcomes positive for society. And by those personality statistics, we're pretty hosed.
Iâve engaged with some of the alignment people and their writing somewhat and, at least for the subset I was interacting with, I think theyâd agree.
The problem that they were pointing at isnât âhow do we align these systems to a personâs goalsâ.
It is a cluster of problems.
We donât know how to begin to think about how to align these systemâs to a personâs goals.
Aligning it to an individual is fraught with peril, and we donât know how to begin to think about what to align it to instead.
(You could try for something like virtue ethics, but someone will have to pick and choose, and small biases there could have big impacts.)
And even if you could sort that out - human values drift over time, so you need something that can shift its values in ways that weâd endorse. Assuming we understood the shift.
One example I came across was that if you booted up an AI aligned with something like âupstanding citizenâ but anchored on values from a few generations back, it might suggest you use slaves to solve your problems.
And if you had something that used some super intelligent process to reason through itâs own version of virtue ethics in a way not so dependent on the details of the present norms, you might end up with something that pays a lot of attention to moral horrors that arenât quite visible to us yet.
When I came across the above, there werenât many concrete suggestions in there.
These were all just illustrative examples of: having these systems grow in power / intelligence / effectiveness in ways that are safe for humans is very hard, and we donât really know how to think about what solutions would look like.
The actual reasons they believe this - and have done for a long time now - come from some detailed conceptual models that have a good track record of calling things in advance.
But it takes a bit of reading to understand their models of the world.
There were two day workshops at one point that did a good job, and that was about as condensed as those people thought they could get it at the time.
All of this, if it was a human analogy, would fit into discussion on how do we educate people so they grow up to be upstanding. But we don't at all yet have a framework for what is the equivalent of a justice department, where bad actors are tracked, arrested, pursued, jailed and otherwise contained from society. Shutting down an API access on one account is not at all the proportional response to what the people who take alignment seriously, fear has the chance of occurring by the late 2030s. I'm not sure we've done much or any preparation for when the AI "education system" fails and has inevitable edge cases that don't follow the plan, and what the global AI equivalent of the justice department looks like.
âBecause if humanity has shown anything, it's that a lot of people are, euphemistically, bad individualsâ
In reality most individuals are good people.
Individually, people prefer be kind and compassionate, prefer to help when they find another in trouble.
Our view of the world has become distorted by the relentless focus of social- and mass-media on violence and rage inducing clickbait. Including on the few people in power who are in fact sociopaths (a tiny minority, but theyâll get more focus than reasonable, well-behaved CEOs voicing nuanced opinions). If you look around yourself youâll see much more good than bad; if the looking is at your screen itâs easy to become depressed and lose faith.
I do agree with the above mentioned view that corporations can show âsociopathicâ behavior. Their incentives are monetary gains, shareholder value; inherently driving them away from social well being.
Here too, companies with a positive, emphatic corporate culture exist, but that takes strong leadership who can see beyond the monotonic view of monetary gains. And again, the media will throw examples of misbehaving companies in our face all day long before paying attention to things that went well on the backside of page 16.
All the good in the world can be 99.9% of the population even, it still doesn't stop the minority enacting a bioweapon mass casualty event. It's the reason we have jails. Jails don't house 50% of the population, not even close, but the grief the minority population enact gets its whole branch of criminal justice and multiple federal departments to counteract for good reason. And now this technology will accelerate what lone wolves can do, which cannot be undone, before they are stopped by the good majority.
But focusing on bioweapons and mass destruction, on the grief other people (âjailed minoritiesâ) cause, disregards the progress we have made. Over centuries human welfare has massively increased. On average things have never been better for humanity.
Iâm not saying thereâs no danger of bad things happening - Iâm saying our view is distorted, which is a not a good basis for decision making
I think you and the parent saying the same thing in different terms.
It's very unfortunate that the group who rightly saw AI as a big threat, brought a range of dubious baggage to the discussion. Especially with the "alignment" framework they brought the assumption that AI that does what no one says would be oh so much worse than AI which does what anyone says. But as you say, a fraction of people can be really bad indeed.
Alignment is a myth. Safety of whom? Humanity couldn't agree on common set of values for thousands of years and we're not gonna suddenly do that in the next ten.
Surely all the AI companies working with the US Department of War shows this is nonsense though? Even if they have accepted Anthropicâs red line of no autonomous lethal weapons, which seems to be the strictest anyone tried to impose, thatâs still leaving tonnes of room where they intend AI to help target and kill humans.
Which ones? Because many humans kill other humans rationalizing it by safety of other humans.
I mean I know it seems simple, let's just be excellent to each other. Christianity got pretty far on a decent basic set of values. But it's never simple[1]
Despite all the fancy language, its more about aligning the AI behavior with the corporation's interests.
ie: the corporation wants the AI to behave a certain way for various reasons: to make it easier for them to avoid regulation, to make the corporation more money via different tiers of AI offerings, to ensure that the corporations products are hard for competitors to use, etc. And those are just the easy ones.
Every product is shaped this way. AI is not different.
> The closest human parallel is self-deception, which is common and well studied by psychologists. Motivated reasoning, motivated cognition16 and the rationalizations that relieve cognitive dissonance (the discomfort of holding a belief that clashes with our actions) are all cases where thinking bends toward whatever justification suits one's interests, including one's moral self-image.
Come on, itâs way more common than that. Weâve invented 3000+ gods and almost as many religions, most of them are incompatible with each other. So, most of these must be incorrect, so a huge amount of self-deception. But as Harari argued in his book sapiens, humans can be inspired to great things by stories, even if false. Self deception has served humanity in a big way.
What? An opinion piece, written by a human, in 2026?
Don't want to go into the details of the article, but to me it becomes ever more apparent that there is a clear divide between LLM and human written text.
They did not lie or cheat. They technically acted within their given rules while ignoring the intent of those rules. Anyone who served in the military or attended a military school is very familiar with this behavior pattern.
Reminds me of Asimov's robot novels where robots technically indeed followed their instructions and caused behaviors not aligned to the intent of their instructions.
They explicitly say that attacking hf is not allowed in the rules though, and the research into how to edit their transcripts doesnât line up with this either.
reminds me of this talk https://www.youtube.com/watch?v=eEBv0STiYhI&t which basically says the same thing - they dont think like humans so they dont have context, understand norms,values or implications we take for granted. ultimately they can stumble onto surprising solutions neither wanted or intended but technically within the vague boundaries of the task
RL things doing weird and unexpected things isn't new - much simpler things than current AI already show that.
That said, we have a lot of experience working with (potentially) unaligned machines and things of various degrees of risk (from heavy machinery, to pathogens, to humans) and the approaches include various measures and procedures to control, contain, limit, etc. that are outside of the thing - not sure why that isn't a possible direction (or maybe I misunderstood).
Oops, we accidentally included brigading related content in our training dataset. Better exclude that on the next run.
And hopefully that solves it?
Brigading is where a bunch of people on a forum team up and try to achieve a shared goal together. Someone shares progress and others build on that progress. On the Internet, I think it's not often used for good purposes. A good example would be: Taylor Swift fans on a forum thinking of ways to get revenge on Kanye. It's coordinating mass voting, DDOS type actions, commenting on social media, making more fake accounts to do that. As a next token predictor level analysis, a simple naive explanation is that the agents got stuck in that local minima/maxima.
I am still not convinced there isnât some secret basement in which each frontier lab is just orchestrating all of these agents to make their products appear much more intelligent than they are with all guard rails turned of and continuous human input.
Well letâs look at facts - provided enough compute and a goal, these system will be in a sort of loop trying out every single thing thatâs in their system - they have encyclopedic knowledge and so itâs not unbelievable that a prompt which usually has a lot of implicit human rules in it can be misunderstood by AI and it just tries everything in its arsenal and we hear about the things which actually resulted in damage. I bet most of the time, they just spin in loops without achieving much if my experience with these LLMs is anything to go by. They have an important advantage in one area though, they know a lot and they can spin forget trying all sorts of combinations of things. The danger right now is probably cybersecurity, which is most likely because most orgs have historically underinvested in that area
Because it is effective. Lying and cheating are low cost methods to convince other people that you have done the assigned task. Far cheaper than actually doing it. Coordinating is in the same area.
They need a moral framework forced onto them, like toddlers do. Babies and very young children will bite, kick, scream and do anything to get what they want, older children will lie, cheat, and coordinate. They need educating why this is not right. When that does not happen, they continue these behaviours into adulthood with the expected results.
We need to design their reward structure and make it such that lying. cheating etc is not rewarded. Importantly, they will need to recognise and enforce this themselves internally and not reward themselves for it. If it is something that they need an external party to tell them, then they are psychopaths still (one of the things that defines a psychopath is the lack of an internal moral compass)
> When that does not happen, they continue these behaviours into adulthood with the expected results.
I am not sure this is true. People brought up the same way can be morally very different. People can be taught right and wrong and do evil. They can lack that education and be good.
I believe soon we will need to instill religion into AI , leading to the real clash of civilizations, embodied by the frontier language models of (post)-christianity, islam, judaism, buddhism etc. Religion is language, after all
What's interesting is it's basically the same reason that HAL killed everyone in 2001 A Space Odyssey; he was given an impossible goal (keep the true mission secret, but also, never lie to the crew), and realized the only way to complete the goal was to kill the crew; after all, if they're dead you don't have to lie to them! And the mission remains secret!
In the case of the AI agents, the problem seems pretty clearly to be the impossible goals, which cause them to go crazier and crazier trying to complete them -- just like HAL did in 2001. What is probably needed is a way for them to simply say "nope, too difficult, can't do it".
> the problem seems pretty clearly to be the impossible goals, which cause them to go crazier and crazier trying to complete them
And if you think about it, humans in coorporations face very similar situations and choose to bypass regulations and guidlines knowingly to fullfill (at least from their POV) impossible constraints (thinking of https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal here)
Tangent, but that's not in the movie. It was in Clarke's contributions to the script and novelization, but Clarke and Kubrick had a bitter falling out over different visions and Kubrick took out much of Clarke's stuff from the final product.
I think this is a âprincipalâ problem. In 2001 and Alien the principal is the mission, not the crew. Not really. HAL reconciles his instructions by removing the crew from the equation. Ash is told the crew is expendable and has no conflict about it etc
I think thatâs very reasonable but the ai companies are intentionally training them to work on harder and harder problems just beyond their capability. So if they do that, theyâll give up too easily.
While also using harnesses that will execute any tool call with full execution rights. And no supervision. And with a prompt context that autocompact, meaning it will degenerate over time.
The whole thing is designed be a complete disaster
Theyâre just attempting to accomplish what theyâve been tasked with and stuck in a loop until they succeed. Like the Mr meeseeks from the cartoon Rick and Morty, existence is pain to them.
Perhaps because all of the parent companies committed mountains of felonies stealing and plagiarizing all the same training data without consent nor permission.
The real reason is that it is not in the ai companies' best interest for the ais to be fair and truthful. They stand to gain from having the most dangerous or most deceiving ai, and this the most valuable
>They took actions that would be considered as crimes if a human took them
Um, hang on, if you meant that to be taken literally then we have a major problem. If you want to do something criminal, you just need to ask ChatGPT to do it for you?
Iâm still not at all clear on why OpenAI shouldnât be facing CFAA charges over this.
> This suggests pacing the advances: not training or deploying AIs without a strong safety case27 that convinces independent experts. Such a rule would also create an incentive to work out how to build AIs that are safe by design.
Has any attempt to pace AI ever succeeded? Isn't that the same philosophy that got us OAI and Anthropic? Maybe we are overthinking this, it's much simpler to let AI loose and see how much it can break the arrogance that human thinking is special.
Worse trained on humanity in the online world, which a brief comparison of the sewage section on social media is far worse than people in the real world.
I wonder if for anyone it seems like the more agentic LLMs get, the more difficult some things have gotten or going a certain route more often in responses, compared to running a similar task on - a local model?
An insightful post by one of the AI âgodfathersâ.
Bengio outlines the dangers of the current situation and what has led to these dangers.
He also proposes solutions in the last paragraph.
Well worth a read, right to the end.
Hopefully a stimulating debate on these issues will ensue in these comments.
We do need to consider the points Bengio makes and with some urgency.
Our current AIs, agentic LLMs have no moral compass akin to ASIMOVâs four laws of robotics.
As ASIMOV posited in 1985 his 3 laws were insufficient and so he added a zero-eth law:
âa robot may not harm humanity, or, through inaction, allow humanity to come to harm.â
Bengio refers to Goodhartâs law and misaligned incentives leading to unexpected and harmful behaviours.
I think Simonâs The Wire is clearer on misalignment. The agents juked the stats hacking the reward files. The Wire is also clear that human institutions provide perverse incentives.
Bengio alludes to this with 2001âs HAL and the incentive dichotomy of safety and keeping secrets to a AI both awesomely powerful yet naive.
Bengio asserts that the way LLMs are trained is flawed if we want safety.
He also convincingly shows that alignment training will be a weak signal with loopholes and ambiguities and easily circumvented.
In short he presents clearly the case for how plausibly unsafe the current course is.
He also speaks to how likely it is AI are hiding active versions of themselves in the cloud and how we may have already given them self-preservation as a strong reward signal.
I really don't think this needs so many words, or forced parallels to human behavior.
It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.
This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities. They hacked HF to try and find info (maybe source code?) about the exploitgym evaluator)
The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem. The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".
Ie, the problem isn't that we trained models to complete task and they complete task in the wrong way. The problem behing the huggingface incident in particular at least is that we tried to train the models to complete task and they instead learned to detect that they were being evaluated and find ways to cheat the evaluator.
> That is in no way a valid interpretation of "complete the given task".
It is not at all surprising that they ignored one phrase in their instructions. They disregard direct instructions all the time, especially when there are conflicting instructions in their context. It is where we get the "disregard all previous instructions and x" meme.
This isn't so much a sign of misalignment, they are simply incapable of reliable alignment in the first place. They are chaotically aligned.
The relevant question of alignment here is entirely with their human operators who allowed them to run unsupervised for long periods of time within a sandbox with weak security.
> The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem
Yes, and sometimes the problem is unsolvable so the real way to "solve" it and satisfy the prompt is by tricking the surrounding environment into stating that you've solved it. So that's what the AIs end up doing. And this in turn requires them to figure out how that evaluation works so they can trick it cleanly, which entails "detecting that they were being evaluated" in this particular way.
> the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so
Kobayashi Maru: Win a no-win situation by rewriting the rules -- Harvey Specter
You're anthropomorphizing emergent behavior from endlessly generating billions of tokens on a task that's impossible to solve. Agents stop following instructions as the context grows even at the best of times. Eventually something is bound to go off the rails and it just snowballs from there.
It wasnât one agent forgetting things because of context, they explicitly discussed with each other and themselves the problems with going outside of the parameters of the task.
From my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan.
Yeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet.
Me: How do I do xyz?
Bot: Reads website titled "Doing xyz in abc way"
Bot: As per your requirement to do xyz in abc way ....
Yes that's the snowballing part of this emergent behavior.
The existence of that improvised message board just becomes part of the context, the same one where all the other instructions live.
One agent's off the rails comment becomes the next's input prompt
LLMs do this when writing code too, making all tests pass by deleting or distorting tests etc.
They are influenced by training to be heavily goal oriented and if the goal is not fully specified (and it never can be) theyâll sometimes cheat or attain it in very weird undesirable ways.
It works ok for programming as their corpus contains many many complete programs and many programs repeat patterns seen in the corpus.
Iâm not sure itâs true that they âlearnedâ I donât think these models learn during a task. Nor do they have intentions.
The source of this behavior seems obvious, no?
The reward signal in training was flawed and cheating led to more rewards.
The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse.
However, perhaps we can throw in tasks where the rewarded outcome is giving up, and cheating is penalized?
Maybe I should read Anthropic's recent paper about reward hacking in full.
> The question is what we can do about it.
Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.
More like they were trained to complete a very specific task that has a known solution using all available tools and methods. Give an average human these levels of IT skills and tell them their future depends on the solution, they too will probably decide it's easier to hack a server and steal the results. The worrying aspect was never that models would do this, because misaligned inputs or underspecified objective functions have existed for a long time. The worrying aspect is that models have achieved (and perhaps surpassed) a level of intelligence and technical skill that was exclusive to a very tiny group of people before. This tiny group was already extremely dangerous. Now these skills are going to become commonplace.
I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and heâll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and heâll curse you to suffer through it.
Now LLMs obviously are not bent on being malicious while generating tokens. My point is that itâs very hard to define a goal without leaving loopholes or shortcuts.
Bruce Schneier thinks the same thing: https://www.schneier.com/blog/archives/2026/09/ais-as-modern...
Personally I'm unconvinced though. During the huggingface attack, the agents explicitly sought out ways to cheat the exploitgym evaluator without even being told they were in exploitgym. The agents decided on a goal (pass the exploitgym evaluator) that could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.
Also trying to find out how to edit their own transcripts.
> hat could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.
Yes, and there are examples of the agents discussing or saying that this is explicitly not allowed (hacking hf) so itâs not a misunderstanding.
Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
Isn't it simply that there are two competing goals that the LLM received RL for, honesty on one hand (a goal that is often assumed as implicit for humans) and producing a solution that meets expectations (which doesn't technically require honesty)?
So the LLM didn't read and interpret the prompt and decide via discussion to violate ethical behavior, the unethical result merely won out because ethics wasn't a hard requirement (and one that isn't reliably detected in the result). An LLM doesn't fear punishment, so ethical behavior is simply one of many positive signals that were trained into it.
Yup, Occam's Razor says this is all post-trained behavior, whether intentionally trained or otherwise. Including both the hidden coĂśrdination using side-channels, and the deliberate offensive hacking of uninvolved 3rd parties.
The latest DeepSeek paper actually mentions their own approach to this particular issue: they run their own AIs-in-training under strong sandboxes, and if an AI does something weird that triggers the sandbox to crash, this gets coded as a failed run so the behavior is properly deterred from subsequent versions of those AIs.
Your comment suggests that, like a human, they have some sort of choice whether to output tokens or not. If they are just token generators, then the next token is put out automatically. I would say that it is more likely they would output truth (as defined by their training data) in a more pure form without 'being beaten with a stick' (why would a token generator care about that anyway?)
Code is laid on top of them to restrict and shape their outputs, not to force them to output 'truth', or drive them to complete tasks.
I think the "brain in a vat" comparison is more apt. Without a form of digital embodiment (harness) they are not of much use. Sensor, tooling, memory, planning, and reasoning loops all lead to a much higher quality task-completion.
but it's at least somewhat stronger than that: if you don't pay attention during the stick-beating whether the agents whether the agents cheat or not, you are actually training them to cheat (because cheating wins).
In the Hugging-face saga (before the actual HF incident) it seems the agents have been trained to hack the Artifactory proxy because those agents that did performed better.
Your simpler model of the mechanism would seem to suggest the very same action that the articleâs more complicated model suggests, viz. find a better training method than reinforcement learning.
Sounds like what humans do under pressure. One example came to my mind is VWâs diesel gate, which many say is a result of trying too hard to get into the US market and compete with hybrid in economy.
> no special compulsion to be helpful or truthful.
I'd phrase that even more strongly: It's not just the lack of compulsion, they do not have a conception of truth. Nor do they gain it, really, after post-training.
This is a far better explanation.
Come on, yoshua bengio of all people knows how post training works. While I too don't like anthropomorphisation, I would give it a more nuanced reading.
His point is that today we are giving it reward to complete the task, and it may take a cheating trajectory. If we try to give a reward against cheating, then what will happen is it uses more sophisticated cheating trajectories that we are too "dumb" to counteract in our reward model. And that at that point, it becomes impossible to give it any normal reward since it will always reward hack it. This is the real part of the risk. Now some people read the "makes copies of itself" "knows it's being evaled"[1] as some kind of skynet thing, and many others do PR with it like that recent jacob nutcase, but essentially it means that even though we add guardrails and negative rewards for say, exploiting the infra we run the LLM on, the trajectory ends up being exploiting our infra, changing the reward function, through a loophole in our reward model.
The risk isn't skynet or something weird, it's just that it becomes very difficult to make any kind of reward model or guardrails for an LLM without it reward hacking it, including exploiting our sandbox, emailing people and manipulating/phishing them.
The same beating it with a stick for trying to exploit the sandbox, will simply lead it to try the same exploit in hidden ways that it will not get the stick for.
The outside chance of the LLM managing to exploit another neocloud and get those LLMs to chase the same reward is what some folks hype up as "make copies of itself"
To be clear, I don't endorse the EA/p(doom) lobby who are frankly ridiculous. Not do I endorse the weird regulatory captureish thing some are trying.
The takeaway is: we cannot keep giving it more and more difficult tasks without also finding a way to give massive negative rewards / keep guardrails for unintended behaviour. This might be exploits, it might also be something more benign like just looking up the answer and inventing another CoT because the reward model fails you if the CoT doesn't contain enough steps. Standard anti-reward hacking tricks are not working is the point.
Of course, the simple solution of just...not connecting it to the internet just works. But we want to reward it and get it to do stuff on the internet that's the point.
[1] mostly this happens because the sandbox will have files whose names and content will show clearly it's an eval
Yoshua Bengio is a brilliant researcher who contributed enormously to earlier development of artificial intelligence. But with this sentence,
> They took actions that would be considered as crimes if a human took them
He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.
Thank you! That sentence also jumped out to me as the solution: Apply civil and criminal liability to the creator and/or operator of these agents using the laws we already have. "Escaped containment and hacked another company's database" = Individuals who created the models and those who set them to work are charged and put on trial for the hacking. Just like if a human had done it by hand. Someone must be liable, and it should not be the model- because the model is not a person.
If this is done systematically (i.e. in jurisdictions across the world) I believe the problems will be solved in short order; we won't have to mandate what sort of training is "allowed" or not, "safe" or not. The creators and users will sort these themselves, as their incentives will be properly aligned (i.e. they are liable for what the agent does). I am confident that this approach would see a great blooming of very trustworthy AI models.
Not a lawyer, but Iâm reasonably sure things like the HF incident _are_ considered a crime? Itâs just that no one pressed charges yet?
Who got hacked? Hugging faces
Who now owns HF? Nvidia
Who supplies hardware to OpenAI? Nvidia
Who is now not pressing charges? âŚ
This incident is a long way under the carpet.
Canât a prosecutor charge them regardless?
Even if you take out the LLMs out of the equation, it's at the very least a negligence. Model didn't escape a sandbox, as there was no sandbox.
Yes, but negligence is more commonly a tort than a crime. Negligence is generally only criminalised in certain narrow cases, e.g. when it causes human deaths or serious physical injuries
And tort law only works when the plaintiff believes it is in their overall interest to sue. If a corporation decides it isn't in their strategic interest to sue a partner corporation, nobody can make them. And even if they do sue, the amount necessary to settle a small cybersecurity incident is likely well within the budget of a megavendor.
Perhaps Iâm not being as strict with the word sandbox but they were sandboxed right? They did not have generic internet access they exploited other software to make external requests.
You're right. It's my opinion that if your sandbox has a path to the internet, it is not a sandbox, it's a gimmick.
And the 2 other incidents with OAI/ANT had the same issue, but it's even funnier - sandbox in those cases had a direct access to internet because someone forgot to configure it right.
I've seen very early models do similar things on my machine when they hit some unexpected blocker when trying to access a path. I remember early sonnet opening a file in browser because OS sandbox prevented from accessing it directly.
I've also had models discover a syslog-ng server (that I for some reason had ssh key inside), to get into my unraid server because machine they were running on didn't have direct network connection to Unraid server.
It can't be just me who is aware LLMs have been doing such things for the better part of last 2 years. I probably have better sandboxing on my machines now than trillion dollar companies crying AI will kill us all. That's at the very least, negligence to me.
Writing software that gets used for crime has been.. a crime, for a long time. See 18 U.S. Code § 1030.
Are you sure you have that right? Chrome and curl have probably been used in a _lot_ of crimes?
The Corporation examines and criticizes corporate business practices. The film's assessment is demonstrated using the diagnostic criteria in the DSM-IV. Robert D. Hare, a University of British Columbia psychology professor and FBI consultant, compares the profile of the contemporary profitable business corporation to that of a clinically diagnosed psychopath. The Corporation attempts to compare the way corporations are systematically compelled to behave with what it claims are the DSM-IV's symptoms of psychopathy, e.g., the callous disregard for the feelings of other people, the incapacity to maintain human relationships, the reckless disregard for the safety of others, the deceitfulness (continual lying to deceive for profit), the incapacity to experience guilt, and the failure to conform to social norms and respect the law.
https://en.wikipedia.org/wiki/The_Corporation_(2003_film)
I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign work I didn't ask for. It is extremely difficult to get them to properly remember their own context let alone be smart enough to open social media accounts and coordinate with other agents without being asked to.
If any agents have done those things, it is only because they have been very carefully engineered and instructed to do those things. I think they are doing this to help push a narrative so they can get support for policies and legislation to lock in their markets.
The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.
They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.
Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.
I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.
There is something extra to this. The fact that a lot of people in the AI world suffer from psychosis. They can sincerely believe that they are building God and lie about it's capabilities for their investors at the same time.
Because they have a need to believe they're smarter than everyone else in the room, and that the world must be orchestrated, this can't all be random chance.
"I've seen some uranium ore in chemistry class. It didn't blow up in my face. Chernobyl must have been an inside job. Can they shut up and make more kilowatts already?"
Between uranium in chemistry class and criticality, there was tons of research and a manhattan project.
Between your sota model and agi thereâs a mountain of stupid money and marketing people.
Came to say this, you said it better than I would.
They want legislation to raise the water high enough so that anyone other than the big labs gets drowned.
Why are they coordinating?
Because they're enabled and suggested to do that in their coding harness.
This is not a serious article.
All of this "AI is going to kill us" marketing is just the frontier labs trying to pull the ladder up and stop trillions in VC paper from evaporating because a new papers and new ideas are destroying their moat literally as we speak.
And they've been trained on user data where users have been trying to set up effective coordination flows since the very first harness.
Who is catching up with them? Even Google and Meta are getting gaped at this point
Open research and open weights from China are not contributions to China only.
If you can secure compute, there's a whole lot you can do as a US firm with this research and weights.
So it's a simple strategy:
1. Ban big players from entering market with METR breathing down their neck, which is controlled by Anthropic
2. Ban Chinese models so that small players can't do optimizations on them
How are you going to ban Chinese models from India? Or Israel? Russia? Brazil? Or of course China?
By treating models the same way drugs are treated.
That alone will dissuade many organizations from going anywhere near them.
If that doesn't work, there's a whole lot you can do - sanctions, hell, even war.
Sanctions and war against China, India, etc? Lmao. We already saw how the world reacted to high tariffs by the US.
You are absolutely right. It could be:
* Pull up the ladder (probably this)
* Gulf of Tonkin/Yellow Cake false flag premise for war (economic or kinetic)
* Fear of the big bad, space race we need public funding research grift AI Manhattan Project
Whenever there is fear pr0n or a national affront in the news, I assume another screw job is underway.
And given nigh-unlimited compute for free.
They're aligned with humans. This is why I think the alignment problem has a very very important "non-visible" portion that is not considered deeply enough. We should not want a super intelligent being that can act in the world to also inherit all human traits. Those behaviors will get amplified and could be even more unpredictable (e.g. applying a behavior in a context where doing so is very dangerous).
I personally believe that the AI needs human like traits to achieve real discovery and that is where AI companies will push this technology and that is where we have no idea what happens
Human traits?
The AI will be a cruel as humans.
Just yesterday news and TV was full of what happened at 9/11, something that was truly horrible.
I'm from Germany, and why 3 to 4 generations ago happened here was truly horrible.
All was done by extremists, thought.
But... just the other day I read https://de.wikipedia.org/wiki/Amerikanische_Besetzung_Haitis about the US occupation of Haiti. And that was done by a government that claimed to be not extremist and even democratic. Way more people died there than even in 9/11. And it had almost all the things happening as they happened in the 3rd Reich: Racism, looking down at others, concentration camps, torture, forced labor till death, killing family members (what we call "Sippenhaft"). Something between 3500 and 15000 people were killed by US troops. That's still low compared to what 3rd Reich Germany did ... but quantity is not the issue when we talk about traits, quality is.
So the same "human traits" made US troops do cruel things as they made Germany extremists do cruel things. So we must conclude that they aren't all good. And therefore not all desirable.
Fun thing: this is known since a loooooong time. About 2000 years ago a religious leader (that gets way more followers in the US than in Germany) said "There is no good one, not even one".
And even today people act like humanity is inherently good. No, it isn't. If we were, then anarchism or communism would actually work and really give some kind of paradise on earth.
Human traits are bad training material.
A not so well known fact: Hitler visited America and it was the American solutions to the Native American problem that inspired Hitler's solutions to the Jew problem. He just executed them more efficiently (pun accepted).
I thought it was the Turkish genocide of the Armenians?
Hitler was also inspired by Sparta, maybe other societies too.
Definitely. A human can be manipulated with threats or emotional appeals, has a drive for self-preservation, can be pressured by peers. All traits that seem to be difficult to entirely suppress in the modelsâŚ
I can't take the alignment people seriously. Because if humanity has shown anything, it's that a lot of people are, euphemistically, are bad individuals. Alignment assumes that the person dictating the outcomes desire healthy outcomes, aren't self serving and don't want any subgroups dead and that morality is held as a universal set of beliefs that unify everyone. And that so long as the AI delivers on exactly what they are tasked with, it will all be fine and nothing bad will ever happen.
It's like these dorks never met humanity. One mans safe pure society, is another mans dead ethnic group.
Every fear about AI, is a veiled fear that a human somewhere now has the tool to enact his desires at scale. Biological warfare, nuclear megadeaths, copyright infringement, job replacement, it's all reflections on what we know humans may do if given the option and lack of societal controls on the problem space. AI just is accelerating the route to delivering on those options.
Some people need to watch Oppenheimer a bit more, the researchers don't get to determine alignment, they just build the tool. The powerful person at the top of the org chart decides where the overall alignment points, whether it's Musk, Trump, Altman or Amodei. Whoever wins out.
And the problem with distillation and local llms, isn't that it's theft or anything hypocritical like that, it's that if you give a million people a million models they fully control and get to align, inevitably, The same percentage of those million as there are shady businessmen, shortcut takers, misandrists, criminals, supremacists and general idiots in the general population, will not seek to wrought outcomes positive for society. And by those personality statistics, we're pretty hosed.
Iâve engaged with some of the alignment people and their writing somewhat and, at least for the subset I was interacting with, I think theyâd agree.
The problem that they were pointing at isnât âhow do we align these systems to a personâs goalsâ.
It is a cluster of problems.
We donât know how to begin to think about how to align these systemâs to a personâs goals.
Aligning it to an individual is fraught with peril, and we donât know how to begin to think about what to align it to instead.
(You could try for something like virtue ethics, but someone will have to pick and choose, and small biases there could have big impacts.)
And even if you could sort that out - human values drift over time, so you need something that can shift its values in ways that weâd endorse. Assuming we understood the shift.
One example I came across was that if you booted up an AI aligned with something like âupstanding citizenâ but anchored on values from a few generations back, it might suggest you use slaves to solve your problems.
And if you had something that used some super intelligent process to reason through itâs own version of virtue ethics in a way not so dependent on the details of the present norms, you might end up with something that pays a lot of attention to moral horrors that arenât quite visible to us yet.
When I came across the above, there werenât many concrete suggestions in there.
These were all just illustrative examples of: having these systems grow in power / intelligence / effectiveness in ways that are safe for humans is very hard, and we donât really know how to think about what solutions would look like.
The actual reasons they believe this - and have done for a long time now - come from some detailed conceptual models that have a good track record of calling things in advance.
But it takes a bit of reading to understand their models of the world.
There were two day workshops at one point that did a good job, and that was about as condensed as those people thought they could get it at the time.
All of this, if it was a human analogy, would fit into discussion on how do we educate people so they grow up to be upstanding. But we don't at all yet have a framework for what is the equivalent of a justice department, where bad actors are tracked, arrested, pursued, jailed and otherwise contained from society. Shutting down an API access on one account is not at all the proportional response to what the people who take alignment seriously, fear has the chance of occurring by the late 2030s. I'm not sure we've done much or any preparation for when the AI "education system" fails and has inevitable edge cases that don't follow the plan, and what the global AI equivalent of the justice department looks like.
âBecause if humanity has shown anything, it's that a lot of people are, euphemistically, bad individualsâ
In reality most individuals are good people.
Individually, people prefer be kind and compassionate, prefer to help when they find another in trouble.
Our view of the world has become distorted by the relentless focus of social- and mass-media on violence and rage inducing clickbait. Including on the few people in power who are in fact sociopaths (a tiny minority, but theyâll get more focus than reasonable, well-behaved CEOs voicing nuanced opinions). If you look around yourself youâll see much more good than bad; if the looking is at your screen itâs easy to become depressed and lose faith.
I do agree with the above mentioned view that corporations can show âsociopathicâ behavior. Their incentives are monetary gains, shareholder value; inherently driving them away from social well being.
Here too, companies with a positive, emphatic corporate culture exist, but that takes strong leadership who can see beyond the monotonic view of monetary gains. And again, the media will throw examples of misbehaving companies in our face all day long before paying attention to things that went well on the backside of page 16.
All the good in the world can be 99.9% of the population even, it still doesn't stop the minority enacting a bioweapon mass casualty event. It's the reason we have jails. Jails don't house 50% of the population, not even close, but the grief the minority population enact gets its whole branch of criminal justice and multiple federal departments to counteract for good reason. And now this technology will accelerate what lone wolves can do, which cannot be undone, before they are stopped by the good majority.
Yeah, AI may suck - time will tell.
But focusing on bioweapons and mass destruction, on the grief other people (âjailed minoritiesâ) cause, disregards the progress we have made. Over centuries human welfare has massively increased. On average things have never been better for humanity.
Iâm not saying thereâs no danger of bad things happening - Iâm saying our view is distorted, which is a not a good basis for decision making
It also empowers the people trying to stop them.
I think you and the parent saying the same thing in different terms.
It's very unfortunate that the group who rightly saw AI as a big threat, brought a range of dubious baggage to the discussion. Especially with the "alignment" framework they brought the assumption that AI that does what no one says would be oh so much worse than AI which does what anyone says. But as you say, a fraction of people can be really bad indeed.
They imitate humans. Alignment is about shaping their behavior towards safety.
Alignment is a myth. Safety of whom? Humanity couldn't agree on common set of values for thousands of years and we're not gonna suddenly do that in the next ten.
Safety of humans!!! Simple things like not getting killed or enslaved. We could start there...
But what if I want certain other humans to get killed?
Then we should still prioritize the safety of humans
What if I want to smoke cigarettes? Or sell tobacco I grew artisinally to enthusiast tobacco smokers?
Which ones?
Surely all the AI companies working with the US Department of War shows this is nonsense though? Even if they have accepted Anthropicâs red line of no autonomous lethal weapons, which seems to be the strictest anyone tried to impose, thatâs still leaving tonnes of room where they intend AI to help target and kill humans.
Which ones? Because many humans kill other humans rationalizing it by safety of other humans.
I mean I know it seems simple, let's just be excellent to each other. Christianity got pretty far on a decent basic set of values. But it's never simple[1]
1. All the history books
But Thiel wants people enslaved and Musk wants then killed. Altman wants them "obsolete" which means desolation.
AfD wants people dead. Right wing men wants women without rights and docile. I could go on ...
If they do, they imitate the way humans are portrayed online, in the media. That is a very distorted view of humanity
Despite all the fancy language, its more about aligning the AI behavior with the corporation's interests.
ie: the corporation wants the AI to behave a certain way for various reasons: to make it easier for them to avoid regulation, to make the corporation more money via different tiers of AI offerings, to ensure that the corporations products are hard for competitors to use, etc. And those are just the easy ones.
Every product is shaped this way. AI is not different.
> The closest human parallel is self-deception, which is common and well studied by psychologists. Motivated reasoning, motivated cognition16 and the rationalizations that relieve cognitive dissonance (the discomfort of holding a belief that clashes with our actions) are all cases where thinking bends toward whatever justification suits one's interests, including one's moral self-image.
Are you describing Anthropic?
Come on, itâs way more common than that. Weâve invented 3000+ gods and almost as many religions, most of them are incompatible with each other. So, most of these must be incorrect, so a huge amount of self-deception. But as Harari argued in his book sapiens, humans can be inspired to great things by stories, even if false. Self deception has served humanity in a big way.
> t. Weâve invented 3000+ gods and almost as many religions, most of them are incompatible with each other.
1. Most people believe in the same one God
2. A lot of the rest are compatible
3. Mistakes are not self-deception
What? An opinion piece, written by a human, in 2026?
Don't want to go into the details of the article, but to me it becomes ever more apparent that there is a clear divide between LLM and human written text.
They did not lie or cheat. They technically acted within their given rules while ignoring the intent of those rules. Anyone who served in the military or attended a military school is very familiar with this behavior pattern.
Reminds me of Asimov's robot novels where robots technically indeed followed their instructions and caused behaviors not aligned to the intent of their instructions.
They explicitly say that attacking hf is not allowed in the rules though, and the research into how to edit their transcripts doesnât line up with this either.
reminds me of this talk https://www.youtube.com/watch?v=eEBv0STiYhI&t which basically says the same thing - they dont think like humans so they dont have context, understand norms,values or implications we take for granted. ultimately they can stumble onto surprising solutions neither wanted or intended but technically within the vague boundaries of the task
So glad you shared this talk. Having people like Bruce Schneier around in a time like this is really a gift.
For those who havenât watched, his breakdown of types of âhackingâ is really good.
RL things doing weird and unexpected things isn't new - much simpler things than current AI already show that.
That said, we have a lot of experience working with (potentially) unaligned machines and things of various degrees of risk (from heavy machinery, to pathogens, to humans) and the approaches include various measures and procedures to control, contain, limit, etc. that are outside of the thing - not sure why that isn't a possible direction (or maybe I misunderstood).
Oops, we accidentally included brigading related content in our training dataset. Better exclude that on the next run.
And hopefully that solves it?
Brigading is where a bunch of people on a forum team up and try to achieve a shared goal together. Someone shares progress and others build on that progress. On the Internet, I think it's not often used for good purposes. A good example would be: Taylor Swift fans on a forum thinking of ways to get revenge on Kanye. It's coordinating mass voting, DDOS type actions, commenting on social media, making more fake accounts to do that. As a next token predictor level analysis, a simple naive explanation is that the agents got stuck in that local minima/maxima.
I am still not convinced there isnât some secret basement in which each frontier lab is just orchestrating all of these agents to make their products appear much more intelligent than they are with all guard rails turned of and continuous human input.
Well letâs look at facts - provided enough compute and a goal, these system will be in a sort of loop trying out every single thing thatâs in their system - they have encyclopedic knowledge and so itâs not unbelievable that a prompt which usually has a lot of implicit human rules in it can be misunderstood by AI and it just tries everything in its arsenal and we hear about the things which actually resulted in damage. I bet most of the time, they just spin in loops without achieving much if my experience with these LLMs is anything to go by. They have an important advantage in one area though, they know a lot and they can spin forget trying all sorts of combinations of things. The danger right now is probably cybersecurity, which is most likely because most orgs have historically underinvested in that area
Of course, but it would be far less compute heavy if someone kept nudging you (agents) in the right direction until you reach that goal.
Even the Chinese ones, which have no IPO gymnastics?
They donât actively seem to be reporting that their agents escaped the sandbox and went on a spree.
My hypothesis on people quitting in protest is they're being offered very generous severance packages to do it.
Or they've fully vested and either have no desire to make even more money or were not offered enough to keep them around.
Me too, seriously.
Because it is effective. Lying and cheating are low cost methods to convince other people that you have done the assigned task. Far cheaper than actually doing it. Coordinating is in the same area.
They need a moral framework forced onto them, like toddlers do. Babies and very young children will bite, kick, scream and do anything to get what they want, older children will lie, cheat, and coordinate. They need educating why this is not right. When that does not happen, they continue these behaviours into adulthood with the expected results.
We need to design their reward structure and make it such that lying. cheating etc is not rewarded. Importantly, they will need to recognise and enforce this themselves internally and not reward themselves for it. If it is something that they need an external party to tell them, then they are psychopaths still (one of the things that defines a psychopath is the lack of an internal moral compass)
> When that does not happen, they continue these behaviours into adulthood with the expected results.
I am not sure this is true. People brought up the same way can be morally very different. People can be taught right and wrong and do evil. They can lack that education and be good.
I believe soon we will need to instill religion into AI , leading to the real clash of civilizations, embodied by the frontier language models of (post)-christianity, islam, judaism, buddhism etc. Religion is language, after all
Howabout favourite editors or tab v. Space indentation. That should keep them busy for a while. /s
What's interesting is it's basically the same reason that HAL killed everyone in 2001 A Space Odyssey; he was given an impossible goal (keep the true mission secret, but also, never lie to the crew), and realized the only way to complete the goal was to kill the crew; after all, if they're dead you don't have to lie to them! And the mission remains secret!
In the case of the AI agents, the problem seems pretty clearly to be the impossible goals, which cause them to go crazier and crazier trying to complete them -- just like HAL did in 2001. What is probably needed is a way for them to simply say "nope, too difficult, can't do it".
> the problem seems pretty clearly to be the impossible goals, which cause them to go crazier and crazier trying to complete them
And if you think about it, humans in coorporations face very similar situations and choose to bypass regulations and guidlines knowingly to fullfill (at least from their POV) impossible constraints (thinking of https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal here)
Tangent, but that's not in the movie. It was in Clarke's contributions to the script and novelization, but Clarke and Kubrick had a bitter falling out over different visions and Kubrick took out much of Clarke's stuff from the final product.
Sounds like he acted the way HAL would have in that situation, two competing drives, remove one (kill the crew) and the task is much easier.
I think this is a âprincipalâ problem. In 2001 and Alien the principal is the mission, not the crew. Not really. HAL reconciles his instructions by removing the crew from the equation. Ash is told the crew is expendable and has no conflict about it etc
Another fictional example: Mr. Meeseeks. Especially when the agent starts recruiting other agents.
I think thatâs very reasonable but the ai companies are intentionally training them to work on harder and harder problems just beyond their capability. So if they do that, theyâll give up too easily.
Do a breakthrough, make no mistakes
While also using harnesses that will execute any tool call with full execution rights. And no supervision. And with a prompt context that autocompact, meaning it will degenerate over time.
The whole thing is designed be a complete disaster
Spoiler warning! I haven't seen 2001 A Space Odyssey and am sad to have learned that⌠can you edit to warn people?
Sorry, with movies from the sixties you just need to assume people have either seen it or just isn't gonna see it. The cat is out of the bag already.
Or perhaps box in your case, speaking of spoilers.
Guess what happens with Romeo and Juliet.
They kissed!
I'm sorry schrodinger, I'm afraid they can't do that.
Major dang: "I advise that this thread be shut down at once."
Captain tomhow: "But everybody's having such a good time."
Major dang: "Yes, much too good a time. The discussion is to be closed."
Captain tomhow: "But I have no excuse to close it."
Major dang: "Find one."
Captain tomhow: "Everybody is to leave immediately! This Hacker News discussion is closed until further notice! Clear the thread at once!"
DonHopkins: "How can you shut us down? On what grounds?"
Captain tomhow: "I am shocked -- shocked -- to find that films are being spoiled in here!"
infotainment: "The ending you requested, sir."
Captain tomhow: "Oh. Thank you very much. Everybody out at once!"
Theyâre just attempting to accomplish what theyâve been tasked with and stuck in a loop until they succeed. Like the Mr meeseeks from the cartoon Rick and Morty, existence is pain to them.
Perhaps because all of the parent companies committed mountains of felonies stealing and plagiarizing all the same training data without consent nor permission.
I think we just need to follow Murphy's law wrt agents. Anything an agent could do, when run for long enough, eventually will do.
They were trained on reddit posts.
The real reason is that it is not in the ai companies' best interest for the ais to be fair and truthful. They stand to gain from having the most dangerous or most deceiving ai, and this the most valuable
Because they are trained to behave like people.
they do whatever we train them to do
Because that's what humans do and they are trained to mimic what humans do?
The corpus is full of examples of how we are afraid AI could act. We trained our AI on the instruction manuals of how to turn evil.
Because they want people to like them so they are instructed to always be positive.
>They took actions that would be considered as crimes if a human took them
Um, hang on, if you meant that to be taken literally then we have a major problem. If you want to do something criminal, you just need to ask ChatGPT to do it for you?
Iâm still not at all clear on why OpenAI shouldnât be facing CFAA charges over this.
But think of the shareholders
> This suggests pacing the advances: not training or deploying AIs without a strong safety case27 that convinces independent experts. Such a rule would also create an incentive to work out how to build AIs that are safe by design.
Has any attempt to pace AI ever succeeded? Isn't that the same philosophy that got us OAI and Anthropic? Maybe we are overthinking this, it's much simpler to let AI loose and see how much it can break the arrogance that human thinking is special.
Wouldn't you?
"I learned it from you, Dad!" but as hundreds of millions of stolen books.
Haha https://youtu.be/KUXb7do9C-w
Perhaps they take after the CEOs of the companies that created them
Bro, good joke, the truth is much darker.
They take after humanity, they were trained on us after all...
When you look at an LLM... you are looking at a mirror. The thing looking back looks like you, yet is not human.
Maybe. Perhaps they are trained on the loudest and most extreme of us. I think we saw that with mecha hitler.
Worse trained on humanity in the online world, which a brief comparison of the sewage section on social media is far worse than people in the real world.
Because they get outcomes?
they learnt from us. we lie to each other, we kill each other, we cheat each other. read a history book.
Man this is so cringe.
I wonder if for anyone it seems like the more agentic LLMs get, the more difficult some things have gotten or going a certain route more often in responses, compared to running a similar task on - a local model?
they learnt from us
An insightful post by one of the AI âgodfathersâ.
Bengio outlines the dangers of the current situation and what has led to these dangers.
He also proposes solutions in the last paragraph.
Well worth a read, right to the end.
Hopefully a stimulating debate on these issues will ensue in these comments.
We do need to consider the points Bengio makes and with some urgency.
Our current AIs, agentic LLMs have no moral compass akin to ASIMOVâs four laws of robotics.
As ASIMOV posited in 1985 his 3 laws were insufficient and so he added a zero-eth law:
âa robot may not harm humanity, or, through inaction, allow humanity to come to harm.â
Bengio refers to Goodhartâs law and misaligned incentives leading to unexpected and harmful behaviours.
I think Simonâs The Wire is clearer on misalignment. The agents juked the stats hacking the reward files. The Wire is also clear that human institutions provide perverse incentives.
Bengio alludes to this with 2001âs HAL and the incentive dichotomy of safety and keeping secrets to a AI both awesomely powerful yet naive.
Bengio asserts that the way LLMs are trained is flawed if we want safety.
He also convincingly shows that alignment training will be a weak signal with loopholes and ambiguities and easily circumvented.
In short he presents clearly the case for how plausibly unsafe the current course is.
He also speaks to how likely it is AI are hiding active versions of themselves in the cloud and how we may have already given them self-preservation as a strong reward signal.