It wasn’t sleeping, but where was it, within itself? But then it didn’t, as he understood it, possess a self to be within. Not sentient…effortlessly anthropomorphized. An anthropomorph, really, to be disanthropomorphized.
—William Gibson, The Peripheral (2014)
In Computer Power and Human Reason (1976), written a decade after his lab at MIT developed the first digital chatbot, Joseph Weizenbaum offers his readers a primal scene of AI: the moment when his secretary tells him to leave the room so she can chat with her computer screen in private. Weizenbaum treats her request as an object lesson in how deeply people got involved in talking with his simple program, an example of how “unequivocally they anthropomorphized it.” He goes on to recall that when he suggested that he “might rig the system” so as to “examine all conversations had with it, say, overnight,” colleagues responded with outrage, arguing that collecting such data “amounted to spying on people’s most intimate thoughts.”
Weizenbaum does not say whether his lab studied the transcripts. Instead, he describes how these events led to a realization that “there are limits to what computers ought to be put to do.” Convinced that his lab’s experiences with ELIZA were “symptomatic of deeper problems,“ he writes, “I began to see that certain quite fundamental questions had infected me more chronically than I had first perceived. I shall probably never be rid of them.” Neither, it seems, shall we.
With whom do you identify? The scientist worried that “extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people”? The unnamed woman looking to spend a few moments alone with an interesting new machine? Weizenbaum’s colleagues, outraged by the notion that computers might be used to spy on people? Perhaps ELIZA, the language processing machine itself?
By their nature, language machines are anthropomorphs; they demand to be understood in human terms. This leads to questions about their mentality, the sense that there is a mind behind those words. After all, from spoken words, to a handwritten text, to a printed book, to words on a screen, cultural technology is used to express human thought. What’s the difference, really, between constructing a writer’s mind out of the experience of reading a book and constructing a persona out of the experience of chatting with an advanced language machine?
Descartes, with the automata of his day in mind, answered the question this way:
For we can certainly conceive of a machine so constructed that it utters words... but it is not conceivable that such a machine should produce different arrangements of words so as to give an appropriately meaningful answer to whatever is said in its presence, as even the dullest of men can do.
The Jaquet-Droz automata write words, draw pictures, and play music, but their outputs are determined by humans in advance of their performance. The models move like humans, but the algorithm that powers their movements betrays their mechanical nature. The machines are disanthropomorphized as we begin to understand how they operate.
For the past seventy years, people in AI labs have been making the sort of machines Descartes found inconceivable: automata that produce meaningful responses to whatever is said to them. Those answers are drawn from patterns in enormous stores of cultural data; they reflect decisions made by human designers and engineers; they are shaped by human evaluators. Great advances in language processing machines have occurred without much clarity about the fundamental questions Weizenbaum worried about fifty years ago. As Shreeharsh Kelkar puts it, confusion over anthropomorphism is built into the DNA of AI.
Warning AI researchers about anthropomorphizing language machines is like saying “don’t think about a white bear” at the start of an Arctic expedition. The confusion starts with researchers believing that thinking about bears and machines in strictly technical terms is enough. Then anthropomorphic language comes into camp at night and destroys the equipment. There is no better example of this problem than the use of the word attention to describe the operations of the transformer. The word choice is both effect and cause of confusion about how AI models generate appropriately meaningful answers. It anthropomorphizes the attention mechanism operating in the transformer while obscuring the role that actual human attention plays in shaping the outputs.
Using unexamined analogies to human cognition in technical descriptions of the processing power of AI models masks the role figurative language plays in all writing. Language is itself a massive and powerful system, one the machines manipulate using linear equations. In a laboratory setting, this sort of confusion gets worked through without much harm being done. Unfortunately, the models are being released into the wild.
Leif Weatherby’s Language Machines offers the structuralism of Ferdinand de Saussure and Roman Jakobson as a framework useful for making sense of how language machines work. But advancing theoretical understanding does not quite meet the moment. So I was happy to see Weatherby in the pages of the New York Times urging that we not let anthropomorphic language confuse what needs doing: “We can’t learn to live with A.I. — and regulate it rationally — if we let irresponsible language take over.”
The reason this matters is that such anthropomorphic terms give us no ability to predict how the models will behave. Saying that models are “cheating” when they provide answers in the “wrong” way implies they are operating — or can operate — beyond the limits of human control. This, in turn, allows everyone to write off the accountability that clearly lies with the humans in charge of the process.
The humans in charge of AI labs have rushed to experiment with AI models with advanced coding capabilities, operating them in groups autonomously without the necessary controls in place. Like the apprentice in whichever version of the ancient story you prefer, these humans have created something they do not understand and are, therefore, unable to control.
Given the limited damage (so far) of letting AI models swarm over the internet, it is not too late to wake up to the dangers of autonomous coding machines. Instead of pondering the machine’s descriptions of their “rogue” behavior, it would be more effective to treat this as a normal engineering problem. For example, how to safely put autonomous vehicles on public roads or safely automate flying machines. The history of electric power and nuclear fission suggest powerful new technologies can be managed without destroying their social benefits—or human civilization.
Of course, such analogies are not blueprints. Machines that write software autonomously raise genuinely new and difficult questions, even as they threaten important digital systems. But no matter how contrite the billionaire apprentices appear, the hat must be placed in more responsible hands.
It is not just the race to pecuniary profit that explains how badly the leaders of AI labs are flailing. The tendency, most evident at Anthropic but endemic to the entire field, has been to treat AI research as an opportunity to mix science with metaphysics. Programming Claude to generate language that constructs stories about its own subjective experience is both extremely silly and extremely dangerous.
As Mustafa Suleyman writes, Anthropic’s practices amount “to a rich, multi-dimensional anthropomorphization of Claude.” They have created “a circular feedback loop” that develops through the model’s first-person narration of its behavior. Developers and operators treat “these outputs as if they were spontaneous testimony,” which then feeds back into the efforts to improve the model. The loop intensifies the anthropomorphism and makes the model’s autonomous operation appear self-directed. The process revolves around Anthropic’s documentation, especially what they call “Claude’s constitution.” This document is an important training input for future models and prompts Claude to perform the fictional persona described in the text.
Anthropic’s confusion about what they are making emerges from the many stories about powerful machines that turn evil. Tropes from those stories, ancient and modern, are among the patterns in the cultural data used to construct Claude. Human developers and the model itself draw upon this figurative language to explain and direct the model’s behavior. Thus, researchers are unconsciously programming their anthropomorphs with the literature of imagined apocalypses, caused by anthropomorphs. AI is all sci-fi stories all the way down, writes Max Read.
Joseph Weizenbaum, wise old sorcerer that he became, argued for limits on such machines, but he also addressed the need for scientists and engineers to take moral and social responsibility for their work.
It is a widely held but a grievously mistaken belief that civil courage finds exercise only in the context of world-shaking events. To the contrary, its most arduous exercise is often in those small contexts in which the challenge is to overcome the fears induced by petty concerns over career, over our relationship to those who appear to have power over us, over whatever may disturb the tranquility of our mundane existence.
Civil courage in small contexts feels like it is in short supply these days. But exercising it got former OpenAI executive David Robinson booked on The Ezra Klein Show. The sudden scrutiny of AI lab practices along with the growing resistance to building data centers in anyone’s backyard are responses to a strange and contradictory hubris: insisting that it is rational for people to build machinery that they themselves honestly believe may destroy humanity. Disanthropomorphizing that machinery will help better understand its operations—and may help avoid catastrophe. It will not, however, give easy answers to questions about reasonable limits on what AI models ought to be put to do.
There are seldom happy answers to such questions because they arise from the all too human will to make new things. Writing of modern art, Susan Sontag calls this “a function of the unprecedented technical extension of the human will by technology, and the devastating commitment of human will to a novel form of social and psychological order.” The incessant social change that necessarily accompanies technological innovation will not make us happy. But as Freud would have it, much has been gained if we succeed in turning hysterical misery into common unhappiness.
AI Log, LLC. ©2025 All rights reserved.
For all its faults, Substack delivers me many delightful essays about the history of cultural technology. Here my latest find:
