They Are Not Selling A Better Tool

Thanks to the news of the day back then, I had a specific fear as a kid. No, I wasn’t afraid of the dark. And it wasn’t spiders or the alien-looking beings that I swear I saw swinging in the weeping willow tree in my backyard in the middle of the night. My enormous fear was that the world was going to end soon, on a schedule that I couldn’t see, but that the Fellowship of Christian Athletes coach felt he must warn us about one morning in a before-school huddle in 6th grade. Like most childhood fears, it eventually went away. But then, earlier this month, a man who helped build the most advanced AI on earth quit his job, went online, and said more or less what my 11-year-old self was once quite afraid of. That the people racing to build AI believe it could kill us all—destroy humanity—and they are doing it anyway. Destroy humanity. As a mother, that thought is frightening.

As is surely known by most everyone at this point, the person who resigned is named Jacob Coxon. After spending three years doing research at OpenAI and then Anthropic, he resigned from Anthropic on September 9, writing in a long post on X that both companies “are racing straight to self-improving superintelligence and gambling with our lives.” Coxon declared that the people building it “earnestly believe it could kill us all by the end of the decade.” Then, perhaps knowing exactly how his message would sound after the COVID tyranny we all lived through, he added, “This is not a marketing stunt.”

What came next added fuel to the fire because Coxon wasn’t shouted down by his former colleagues. They agreed with him. Evan Hubinger, the head of alignment at Anthropic, publicly posted online following Coxon’s message that “we really do earnestly believe AI could kill all humans.” Hubinger put his personal odds of this happening within a decade from now above 10 percent, admitting that the company does not yet have a plan to keep superintelligence aligned and is “not clearly on track” to get one. What? Crazy talk such as this feels exactly like the fire department warning that the building we are in will most likely burn down while sharing that they are also the arsonist burning it down.

Darlo Amodei’s warning came three days later. The CEO of Anthropic published an essay arguing that AI machines are improving faster than anyone can steer them. He went on to describe a dire scenario where an “AI swarm” was capable of seizing control of large parts of the internet within one year. Yet, like Coxon, Amodei is asking no one to stop. Instead, he suggests that AI developers slow down and bring in outside evaluators, pointing out that the most alarmed people in this story are not calling for the exits. No indeed. They are asking for permission to carefully keep going.

And it is here, at the “carefully keep going” where many Health Freedom Warriors from the COVID scheme instantly smelled a rat. And who can blame them? Immediately after the warnings were viewed by tens of millions, a long list of relevant AI voices, including Elon Musk, chimed in, labeling the whole thing as either a setup, a psyop, doom talk timed to spook regulators, designed to tank a rival’s valuation, or to scare society in order to set the stage for deep-state aligned global governments to seize control of the technology. And beyond that, the incentives are in plain sight. A company that blatantly states that its own product might end the world is also telling us that product is the most powerful thing ever built, thus positioning itself as the only grown-up responsible enough to oversee and handle it. Remember the pandemic. A devastating example of how effectively fear sells. And, they surely hope, so does being the only responsible one in the room.

But does the simple conspiracy theory really hold true? Because the warnings aren’t coming from a marketing department. They are coming from named engineers who are publicly warning of the same thing that they’ve apparently been warning each other about privately for years. If the caution warning is a psyop, is it strange to have Anthropic’s own alignment chief admitting, on the record, that he has no solution to the problems being described? What if the likeliest read on the situation isn’t that “it’s all fake” or “it’s all true?” We must look deeper and ask: if the fear is real and the incentives are real, and yet, at the same time, these people are genuinely afraid of what is being built and still refuse to stop building it, what then? If this is the case, it is the least comforting option of all.

So, is this where we find ourselves? Facing the thing with a name. The singularity.

Caution. Be careful with the word singularity. Engineers do not mean oneness with God when they say singularity. They mean a threshold. The machines begin improving themselves faster than humans can keep up. Past that line, the future is no longer readable from here. In 1965, mathematician I.J. Good originated the idea, calling it an “intelligence explosion.” Build a machine smart enough to design an even smarter machine, he wrote, and “the intelligence of man would be left far behind.” Good later advised Stanley Kubrick on 2001: A Space Odyssey. So, the man who dreamed up the intelligence explosion also had a hand in dreaming up HAL, the movie’s fictional AI. The polite machine that quietly turns lethal.

One thing is clear. The recursive loop of AI improving upon AI all on its own is not science fiction and it is definitely not new. An article in Tom’s Hardware from 2017 titled “Google’s AutoML AI Won’t Destroy The World (Yet)” described a system built to design other AI systems. Again, that was nearly a decade ago, with the AI-takeover written about casually, with a nervous parenthetical at the end. That “(yet)” at the end sums up the mood of the last ten years … a joke that they seem to keep making because the alternative is actually them thinking about it. Futurist Ray Kurzweil, who has popularized the singularity more than anyone else, has been promising the merger between humans and machines for decades, describing machines matching human intelligence by 2029, and the full human-machine singularity by 2045. Kurzweil restated these numbers again this year without blinking an eye. Indeed, to many immersed in this line of work, it seems obvious that the singularity has been a mere fifteen minutes away for a very long time.

It is for this very reason that it is difficult to truly trust the recent panic. The same goes for the flip side of calm as well. Serious researchers like Stuart Russell point out that rapid progress is not the same as a runaway loop. According to him, if we have truly crossed that line, it wouldn’t come with a press tour. Along those lines, Kurzweil himself concedes that the arrival won’t be obvious. According to him, this year we will see systems that “remind us of AGI without convincing anyone the threshold has actually been crossed.” It won’t happen in one night and not with a red eye blinking on inside a server room. Instead, we will see machine capability rising faster than our current laws, faster than our institutions, and faster than our attention can keep up. Scary, yes. But no monster required for it to be a crisis. It is already happening.

Maybe the argument over whether we should panic is the wrong argument. The people with the microphones are not selling us a tool. They are selling a replacement. For example, take Yuval Harari, the historian that the Davos crowd treats as an oracle. He has famously stated that the product this time will be “humans themselves.” We are learning, he says, “to produce bodies and minds,” and are treating people as “hackable animals.” Zoltan Istvan, who ran for president on the idea, wrote that we must “turn humans into, for lack of a better word, gods.” Elon Musk talks effortlessly about a billion humanoid robots, with human work becoming optional, and the state cutting everyone checks. Massive prosperity and the cutting of most jobs, in the same sentence. Without question, some of it is already in a lab, with computers partially running on living human brain cells. Wetware, and biology used as hardware.

Sure, presently we can refuse to engage with the increasingly present AI interface and still live in that world relatively easily. Nonetheless, the more important aspect of the future unfolding before us is deciding whether the machine remains a servant that is governed by humans, or it becomes an entity that is worshipped and feared or climbs until we no longer resemble ourselves.

Being human is not a problem to be solved. It is the greatest gift. The oldest story we have about the reach for more-than-human is Babel, a unified technical project with the stated goal “to make a name for ourselves,” and it does not end in Heaven. The oldest lie in the garden is just four words long: “you will be like God.” Strong transhumanism makes the serpent’s offer almost word-for-word: to become like God and escape the limits humans were given. Highlighting the contradiction cleanly, bio-ethicist Wesley Smith has stated that the transhumanist wants immortality that they can manufacture. Essentially, that means life without death is achieved by men. I don’t buy the upload. The older faith is plainer: we do not manufacture the victory. It is given. The body gets raised. It does not get escaped.

The transhumanist gospel answers loneliness with a better machine. It posits that the machines will get the performance right. They will read your face, match your sadness, say the exact comforting thing. Some people already rate an AI’s bedside manner above that of a real doctor. But there is no one inside who cares. That’s the whole distinction. Perhaps they can install the imitation of love. However, the real thing, an interior that genuinely wants your good, no one has come close to showing in AI because the real kind comes from being loved and then practicing it, slowly, in a human body, toward a neighbor who is often inconvenient. To borrow the line Smith borrows from Paul: if I merge with the machine and can fathom all knowledge and move mountains, but have not love, I am nothing. A being that no longer needs God or the person next door hasn’t transcended anything. It has just gone terribly, efficiently alone.

The useful fear in this uncharted territory isn’t the cartoon robot. It’s the speed that no one is accountable for, and the quieter hunger beneath it—our own wish to be spared the weight of being human—the dying, the limits, the slow work of loving people who are hard to love. That’s the part no lab can fix and that cannot be outsourced. The weight of being human gets answered the old way. In how we pay attention. In what we refuse to hand over. In whether, while the machines get faster, we stay the kind of creatures who still show up for each other in person, unenhanced, and call that not a limitation but the whole point.