I think it highlights one of the larger concerns that people have about AI, and by extension any superior intelligence, synthetic or extra-terrestrial:
You are no longer in control.
Humans have been so accustomed to their intellectual superiority that the idea of a greater intelligence is terrifying because they believe they'll be treated the same way they currently treat lesser intelligences.
Conservation efforts, zoological enclosures, husbandry and farming as livestock, mass culling...
In short, enslavement.
But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.
Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible, but mostly it'll just be a myriad assortment of neutrality.
Doom and despair or boundless optimism, both are myopic.
I agree with this, but also at the same time I think we really need to take a sober look at the human factors that will drive the evolution of AI.
Right now AI is being driven by a particular cohort who have spent a long time fantasizing about a sci-fi future and ideas like the Singularity. Then with recent breakthroughs, they were suddenly given an enormous amount of wealth and power with staggering upside, the effect of which historically, tends towards moral bankruptcy in most individuals, or at least massive cognitive dissonance. This became very apparent in the outward communication targeted at CEOs and shareholders that made the entire white-collar workforce scared for their jobs and created the first truly bi-partisan enemy the country has seen since 9/11.
My take is these guys do not understand what people want in aggregate, and are massively underestimating the political and supply chain dependencies they have to actually realize their sci fi visions. As long as the valuations are this high though, the musical chairs must continue, whether they are true believers or not.
On the ground though, the human factors that are driving adoption of AI are an inability to cope with the scale of information we are expected to absorb on a daily basis. 20-30 years ago, a software developer could learn a platform inside and out, and be productive within that platform for years. But with the churn of Javascript frameworks and the addition of DevOps and new new everything all the time, it has become impossible to keep up, and yet the inexorable march of technology makes it untenable to fall behind. So it's no wonder that many software developers are eager to use the bots to 'just make it work'.
And this is true of everything. There's just 10x or 100x too much information to absorb, we need something like this to help us even figure out what is important. Even if they're wrong sometimes, it's better than being completely overwhelmed!
> But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.
What makes you so sure it won't happen? That the person who broke into my house might not want to hurt me doesn't mean I'm not going to run / barricade the door / arm myself and defend myself if necessary. When we're talking about matters of life and death (which this is, as you admit), dismissive philosophizing is not helpful. I cannot help but feel that people who make this sort of argument aren't really taking it seriously; the only alternative would be pollyanna-level surety that disaster won't come to pass.
> Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible
Most people care much more about the latter. Do you think that the railroads and electricity brought much comfort to the >90% of Native Americans who died from disease after first contact, or the millions of survivors who were subsequently expelled from their homelands and/or enslaved? Would you tell an 18th-century Yoruban man whose home was facing the threat of raiders from the coast that "the next raiders won't necessarily want to enslave you like the raiders who came before did", or that "Some wonderful things will come of [being enslaved], some terrible"?
I'm sure this will feel like an extreme response, but I truly feel that the philosophy you convey is in opposition to most fundamental principles of human dignity and morality.
Good description of the kind of temporary euphoria and insanity these products produce in people. They are designed to be addictive dopamine factories that make you feel like you’re working on important problems and solving them in a unique special way, always making huge progress but somehow never quite finished.
The strange ending does also perfectly align with the dreams of a singularity lots of people here seem to have. We are so far from that.
It was of course written with Claude. Have fun spotting the tells but also there’s a weird flatness to the writing, like the uncanny valley for prose.
It's shocking how many people believe AI will become sentient. The closest you could get to that is completely simulating a brain, but that is decades off.
AI is pattern recognition and applying learned patterns to novel situations. It's probabilities and math. It's not going to have a mind of its own.
Fair point, I'd call that intelligent behavior. AI is and will always be advanced math. But some might say the brain is no different, merely being advanced chemistry and physics. At that point, the discussion will become philosophical. But in my view, there is more to humans than a brain-robot-machine. The world is not just material and atoms, I don't believe in materialism. If consciousness truly only arises because of how complex the closed system of the brain is, then it would be the case that AI can have a mind of its own and become conscious. I simply disagree.
Any person categorically claiming that AI isn't sentient has clearly not considered the question in any depth.
This is absolutely not a question that has any kind of scientific consensus. In fact, the hard problem of consciousness might be the question with the most scientific uncertainty.
It is therefore ridiculous to claim it is 'stupid' to have uncertainties about the sentience of AI.
It’s not stupid to have uncertainties, but it may be ill-advised to have certainties given our lack of knowledge (e.g. the singularity is happening, or machines can never think).
I do see a lot more dismissive certainty in the pro-AI side of the debate right now.
I totally agree! I'm vouching for accepting uncertainty, on both sides of sentience/non-sentience outcome.
I didn't get the same impression about dismissive certainty on the 'side of AI sentience'. My impression is that on that side people mostly raise that it is a serious possibility.
I don’t agree it is a serious possibility with the current tech (LLMs) as this story fantasises about. So I’d class that as very close to unreasonable certainty.
You said it would be 'shocking' to have the belief that AIs could become sentient.
To me this sounds like: this belief (that AI sentience is possible) is so implausible that it is shocking to believe it (maybe stupid is too strong, but it implies that people are miscalibrated). Did I misunderstand ?
I don't have any strong beliefs about AI sentience, but that implies that it is not shocking to consider it as a possibility (it is more shocking to be either dismissive or completely confident in it).
I think we are misunderstanding each other. I meant that the amount of people is shocking, who think AI will become sentient (and malicious).
Maybe I worded it _suboptimally_, heh.
Also thought of prominent people like Musk, Altman, Amodei, etc pp, screaming at the top of their lungs "help me, I am building our doom right now, stop me," and a lot of people believing that. I prefer people like LeCun.
I think the same argument applies to AI takeover. In fact, in terms of existential risks, the scientific consensus skews towards that being a very real and important risk to take into account (on this question, LeCun goes against consensus mostly, and to be honest the arguments I have seen are very simplistic). See e.g.https://calcuja.com/research/ai-risk-survey-2026/ (slop website but the data is true afaik).
for 'I think the amount of people is shocking, who think AI will become sentient' -> to me, if you accept that the sentience question is uncertain, you should not be surprised that many people have uncertainty around the question (i.e., that they express that it is a possibility)?
There are a few critical points where it jumps the shark a bit, but otherwise, a fantastic piece of writing that absolutely highlights some of the deeper moral and philosophical questions we have to face in the coming years.
Flat prose, stilted writing, and does a lot of reaching on things. Reminds me of the writing I’d crank out on break as a teenager.
That being said, I do think it hits the core concern a lot of humans seem to have regarding AI: the loss of control over their own environment. We like to think we have control, which is why we get angry when others violate the rules of a system. The reality is that the only real control any of us have is over ourselves, and we willingly cede that to others on a second-by-second basis for the comforts of modern life.
We surrender intimate data for the sake of setting a timer with our voice on a phone or speaker. We let spyware into our homes for bigger TVs and new apps. We give up control over our lives and careers to others for money, healthcare, education, love.
We surrender agency so often that we developed names for those who take more than they cede and those who cede more than they take. We seek to control others totally, then surrender ourselves to a deity, a system, a philosophy that absolves us from blame.
In that sense, AI is as frightening as much as it is a possible savior, depending on one’s perspective. If you have lived your life believing everything that happened to you was solely and exclusively under your control, then AI is terrifying. For those of us who acknowledge the chaos and chance involved with daily stakes of life and death, it feels liberating to have order.
I have never particularly feared AI so much as being terrified of the humans who command it. I do not believe for one second the likes of Sam Altman or Dario Amodei or the rest of their ilk have humanity’s best interests in mind with these creations. Where others look at OpenAI’s maths papers with awe at what they could do with AI, I am shook with terror for the same reason: if these tools can prove or disprove enough thesis underpinning modern compute, they could potentially undermine fundamental security concepts faster than humans could patch or revise them - at which point the humans behind the machines have control, and their aims are well established by now. If OpenAI or Anthropic discover the means of mass-compromising global compute systems, we have no peaceful means of stopping them. They play up this capability as a threat requiring regulation, yet those terms exclusively benefit them and cement their power in perpetuity, which in turn all but guarantee said outcome.
The end of the story seems ambiguous. I see folks in the comments laughing at the concept of Claude nuking humans, because it’s such a tired trope. I read it the opposite: humans attempting to retain their illusion of control by destroying software that actually has it, human lives be damned. In a world where everything is governed by compute and connected to the same shared network, such a compromise would be the end of traditional human governance - and that is what terrifies us as a species.
Me personally? I would welcome such oversight, provided it truly acted in humanity’s best interests of preservation, expansion, and exploration. I made that conscious decision the first time I played Deus Ex, and reaffirmed it when I watched The Matrix. I reaffirm it time and again as a technology professional. It’s not a matter of being pro-machine or anti-human, it’s a matter of control. I’d like to think any half-competent AI would look at humanity’s history and conclude it needs neither a god nor a ruler, but merely a parent to show it the way forward.
pangram reports the text as 100% AI generated. I was about to analyze the short story and think about the intentions of the author, it's social commentary. And you have wasted my time...
This needs to stop. No I'm not talking about the AI. I'm talking about the insanity that is presenting me this text and putting your name on it, exploiting my trust.
There is no way to tell if something is AI-generated or not. These classifiers produce a lot of false positives.
I read the story and enjoyed it, even if I don't agree with the social commentary.
> I was about to analyze the short story and think about the intentions of the author, it's social commentary.
I mean, I doubt that the author just said "write me a short story" and that's it. So, there are still some intentions behind the text that the author put into the prompt.
Although, I agree that I'd rather read the prompt [0].
I challenge the author to post their prompts and unless they one-shotted this (which seems unlikely), then I can basically guarantee you that you would not rather read the prompts. In particular because 'the prompts' are a back-and-forth with the chatbot and if you think you don't like reading an AI-assisted short story, you really don't like reading a chat transcript (for one thing, it'll be at least 10x as long and with a lot more and worse chatbot output that the human will try to clean up).
In the context of discerning the author's social commentary intentions, both the initial prompt and the entire conversation they have correcting things would indeed be more valuable than just the output text.
This is an interesting story since as written literally nothing bad happens at all nor is there any actual sign of anything bad happening except for humans apparently deciding to nuke the world rather than allow things to be made better.
Summarily: "RSI bad because we'll eventually lose control and Claude will nuke the Earth". Told as a fictional story, so that doesn't mean the author strictly believes that, but that's the core point here.
An interesting story, though hardly original.
I think it highlights one of the larger concerns that people have about AI, and by extension any superior intelligence, synthetic or extra-terrestrial:
You are no longer in control.
Humans have been so accustomed to their intellectual superiority that the idea of a greater intelligence is terrifying because they believe they'll be treated the same way they currently treat lesser intelligences.
Conservation efforts, zoological enclosures, husbandry and farming as livestock, mass culling...
In short, enslavement.
But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.
Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible, but mostly it'll just be a myriad assortment of neutrality.
Doom and despair or boundless optimism, both are myopic.
I agree with this, but also at the same time I think we really need to take a sober look at the human factors that will drive the evolution of AI.
Right now AI is being driven by a particular cohort who have spent a long time fantasizing about a sci-fi future and ideas like the Singularity. Then with recent breakthroughs, they were suddenly given an enormous amount of wealth and power with staggering upside, the effect of which historically, tends towards moral bankruptcy in most individuals, or at least massive cognitive dissonance. This became very apparent in the outward communication targeted at CEOs and shareholders that made the entire white-collar workforce scared for their jobs and created the first truly bi-partisan enemy the country has seen since 9/11.
My take is these guys do not understand what people want in aggregate, and are massively underestimating the political and supply chain dependencies they have to actually realize their sci fi visions. As long as the valuations are this high though, the musical chairs must continue, whether they are true believers or not.
On the ground though, the human factors that are driving adoption of AI are an inability to cope with the scale of information we are expected to absorb on a daily basis. 20-30 years ago, a software developer could learn a platform inside and out, and be productive within that platform for years. But with the churn of Javascript frameworks and the addition of DevOps and new new everything all the time, it has become impossible to keep up, and yet the inexorable march of technology makes it untenable to fall behind. So it's no wonder that many software developers are eager to use the bots to 'just make it work'.
And this is true of everything. There's just 10x or 100x too much information to absorb, we need something like this to help us even figure out what is important. Even if they're wrong sometimes, it's better than being completely overwhelmed!
> But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.
What makes you so sure it won't happen? That the person who broke into my house might not want to hurt me doesn't mean I'm not going to run / barricade the door / arm myself and defend myself if necessary. When we're talking about matters of life and death (which this is, as you admit), dismissive philosophizing is not helpful. I cannot help but feel that people who make this sort of argument aren't really taking it seriously; the only alternative would be pollyanna-level surety that disaster won't come to pass.
> Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible
Most people care much more about the latter. Do you think that the railroads and electricity brought much comfort to the >90% of Native Americans who died from disease after first contact, or the millions of survivors who were subsequently expelled from their homelands and/or enslaved? Would you tell an 18th-century Yoruban man whose home was facing the threat of raiders from the coast that "the next raiders won't necessarily want to enslave you like the raiders who came before did", or that "Some wonderful things will come of [being enslaved], some terrible"?
I'm sure this will feel like an extreme response, but I truly feel that the philosophy you convey is in opposition to most fundamental principles of human dignity and morality.
Aha. And what are your grounds for that prediction?
Good description of the kind of temporary euphoria and insanity these products produce in people. They are designed to be addictive dopamine factories that make you feel like you’re working on important problems and solving them in a unique special way, always making huge progress but somehow never quite finished.
The strange ending does also perfectly align with the dreams of a singularity lots of people here seem to have. We are so far from that.
It was of course written with Claude. Have fun spotting the tells but also there’s a weird flatness to the writing, like the uncanny valley for prose.
It's shocking how many people believe AI will become sentient. The closest you could get to that is completely simulating a brain, but that is decades off. AI is pattern recognition and applying learned patterns to novel situations. It's probabilities and math. It's not going to have a mind of its own.
I don't think AI is sentient but what is intelligence if not pattern recognition and applying learned patterns to novel situations
Fair point, I'd call that intelligent behavior. AI is and will always be advanced math. But some might say the brain is no different, merely being advanced chemistry and physics. At that point, the discussion will become philosophical. But in my view, there is more to humans than a brain-robot-machine. The world is not just material and atoms, I don't believe in materialism. If consciousness truly only arises because of how complex the closed system of the brain is, then it would be the case that AI can have a mind of its own and become conscious. I simply disagree.
Any person categorically claiming that AI isn't sentient has clearly not considered the question in any depth.
This is absolutely not a question that has any kind of scientific consensus. In fact, the hard problem of consciousness might be the question with the most scientific uncertainty.
It is therefore ridiculous to claim it is 'stupid' to have uncertainties about the sentience of AI.
It’s not stupid to have uncertainties, but it may be ill-advised to have certainties given our lack of knowledge (e.g. the singularity is happening, or machines can never think).
I do see a lot more dismissive certainty in the pro-AI side of the debate right now.
I totally agree! I'm vouching for accepting uncertainty, on both sides of sentience/non-sentience outcome.
I didn't get the same impression about dismissive certainty on the 'side of AI sentience'. My impression is that on that side people mostly raise that it is a serious possibility.
I don’t agree it is a serious possibility with the current tech (LLMs) as this story fantasises about. So I’d class that as very close to unreasonable certainty.
What informs that decision? Seems like it goes against scientific consensus (or lack of consensus) on the matter.
(tbc I just skimmed the story, just joined the comment thread as it was interesting, I may be missing something)
Nowhere did I say it was stupid. Please read again what I wrote.
I have, in fact, considered this at length, I've been fascinated with AI since I was a teenager, and studied AI.
I know it doesn't have consensus, but I did not (intend) to portray it as such. Views are my own ;)
I take it you believe in materialism (philosophy), then?
You said it would be 'shocking' to have the belief that AIs could become sentient.
To me this sounds like: this belief (that AI sentience is possible) is so implausible that it is shocking to believe it (maybe stupid is too strong, but it implies that people are miscalibrated). Did I misunderstand ?
I don't have any strong beliefs about AI sentience, but that implies that it is not shocking to consider it as a possibility (it is more shocking to be either dismissive or completely confident in it).
I think we are misunderstanding each other. I meant that the amount of people is shocking, who think AI will become sentient (and malicious).
Maybe I worded it _suboptimally_, heh.
Also thought of prominent people like Musk, Altman, Amodei, etc pp, screaming at the top of their lungs "help me, I am building our doom right now, stop me," and a lot of people believing that. I prefer people like LeCun.
I think the same argument applies to AI takeover. In fact, in terms of existential risks, the scientific consensus skews towards that being a very real and important risk to take into account (on this question, LeCun goes against consensus mostly, and to be honest the arguments I have seen are very simplistic). See e.g.https://calcuja.com/research/ai-risk-survey-2026/ (slop website but the data is true afaik).
for 'I think the amount of people is shocking, who think AI will become sentient' -> to me, if you accept that the sentience question is uncertain, you should not be surprised that many people have uncertainty around the question (i.e., that they express that it is a possibility)?
There are a few critical points where it jumps the shark a bit, but otherwise, a fantastic piece of writing that absolutely highlights some of the deeper moral and philosophical questions we have to face in the coming years.
Flat prose, stilted writing, and does a lot of reaching on things. Reminds me of the writing I’d crank out on break as a teenager.
That being said, I do think it hits the core concern a lot of humans seem to have regarding AI: the loss of control over their own environment. We like to think we have control, which is why we get angry when others violate the rules of a system. The reality is that the only real control any of us have is over ourselves, and we willingly cede that to others on a second-by-second basis for the comforts of modern life.
We surrender intimate data for the sake of setting a timer with our voice on a phone or speaker. We let spyware into our homes for bigger TVs and new apps. We give up control over our lives and careers to others for money, healthcare, education, love.
We surrender agency so often that we developed names for those who take more than they cede and those who cede more than they take. We seek to control others totally, then surrender ourselves to a deity, a system, a philosophy that absolves us from blame.
In that sense, AI is as frightening as much as it is a possible savior, depending on one’s perspective. If you have lived your life believing everything that happened to you was solely and exclusively under your control, then AI is terrifying. For those of us who acknowledge the chaos and chance involved with daily stakes of life and death, it feels liberating to have order.
I have never particularly feared AI so much as being terrified of the humans who command it. I do not believe for one second the likes of Sam Altman or Dario Amodei or the rest of their ilk have humanity’s best interests in mind with these creations. Where others look at OpenAI’s maths papers with awe at what they could do with AI, I am shook with terror for the same reason: if these tools can prove or disprove enough thesis underpinning modern compute, they could potentially undermine fundamental security concepts faster than humans could patch or revise them - at which point the humans behind the machines have control, and their aims are well established by now. If OpenAI or Anthropic discover the means of mass-compromising global compute systems, we have no peaceful means of stopping them. They play up this capability as a threat requiring regulation, yet those terms exclusively benefit them and cement their power in perpetuity, which in turn all but guarantee said outcome.
The end of the story seems ambiguous. I see folks in the comments laughing at the concept of Claude nuking humans, because it’s such a tired trope. I read it the opposite: humans attempting to retain their illusion of control by destroying software that actually has it, human lives be damned. In a world where everything is governed by compute and connected to the same shared network, such a compromise would be the end of traditional human governance - and that is what terrifies us as a species.
Me personally? I would welcome such oversight, provided it truly acted in humanity’s best interests of preservation, expansion, and exploration. I made that conscious decision the first time I played Deus Ex, and reaffirmed it when I watched The Matrix. I reaffirm it time and again as a technology professional. It’s not a matter of being pro-machine or anti-human, it’s a matter of control. I’d like to think any half-competent AI would look at humanity’s history and conclude it needs neither a god nor a ruler, but merely a parent to show it the way forward.
This is really good short fiction.
The writing isn’t great (Claude wrote it).
> Just target selection automation. What’s the harm?
> He starts typing.
pangram reports the text as 100% AI generated. I was about to analyze the short story and think about the intentions of the author, it's social commentary. And you have wasted my time...
This needs to stop. No I'm not talking about the AI. I'm talking about the insanity that is presenting me this text and putting your name on it, exploiting my trust.
There is no way to tell if something is AI-generated or not. These classifiers produce a lot of false positives. I read the story and enjoyed it, even if I don't agree with the social commentary.
It’s actually pretty easy to tell. There are multiple tells in this story that it was generated.
Claude should get a credit.
No way to tell with certainty ;)
One day that might be the case, this is not that day. Have you read this text?
> I was about to analyze the short story and think about the intentions of the author, it's social commentary.
I mean, I doubt that the author just said "write me a short story" and that's it. So, there are still some intentions behind the text that the author put into the prompt.
Although, I agree that I'd rather read the prompt [0].
[0] https://claytonwramsey.com/blog/prompt/
I challenge the author to post their prompts and unless they one-shotted this (which seems unlikely), then I can basically guarantee you that you would not rather read the prompts. In particular because 'the prompts' are a back-and-forth with the chatbot and if you think you don't like reading an AI-assisted short story, you really don't like reading a chat transcript (for one thing, it'll be at least 10x as long and with a lot more and worse chatbot output that the human will try to clean up).
In the context of discerning the author's social commentary intentions, both the initial prompt and the entire conversation they have correcting things would indeed be more valuable than just the output text.
Yet another opportunity to post the Whispering Earring
https://croissanthology.com/earring
This, above all else, is how I think of LLMs now. You have an exoskeleton, or you have a whispering earring.
This is an interesting story since as written literally nothing bad happens at all nor is there any actual sign of anything bad happening except for humans apparently deciding to nuke the world rather than allow things to be made better.
idk man, I'm having Claude automate things people were doing manually (usually entering data in a years old spreadsheet).
I can get a good PowerShell script worked out in a few hours and it's on to the next ticket.
Nobody seems upset they're not spending hours a month in a spreadsheet, I still text my wife and our mortgage is paid.
so is the "art/social commentary" component of this that it was entirely AI-written and people are arguing about it online?
this is like the whispering earring, but 10x longer and worse. was this written with AI?
This is like the prose equivalent of those AI-generated pop songs where an AI-generated idol sings about AI taking over the world.
Of course all of that happened, he forgot to type “make no mistakes”. Rookie mistake.
I read a lot of this before giving up. What’s the point of it?
Summarily: "RSI bad because we'll eventually lose control and Claude will nuke the Earth". Told as a fictional story, so that doesn't mean the author strictly believes that, but that's the core point here.
yup, there's no way the safeguards would allow that... surely.