Orwell’s Warning for the Age of AI

I.

In the introduction to his 1991 book Orwell: The Authorised Biography, Michael Shelden distinguishes his approach from that of Bernard Crick’s George Orwell: A Life, published a decade earlier. While Crick’s volume offered the most complete portrait of Orwell available at that point, Shelden argues that it’s too dull and impersonal—a flood of facts that bury Orwell’s singular, idiosyncratic personality. Shelden observes that Crick “relies heavily on the notion that facts speak for themselves if presented in enough detail.” So he attempts to provide a more intimate account of Orwell’s life: “A writer’s character and personal history influence what he writes and how he writes it. And the more we know about him, the better we are able to appreciate his work.” After all, “Books are not written by machines in sealed compartments.”

But we have now entered an era in which books can, in fact, be written by machines in sealed compartments. Large language models (LLMs) generate billions of words a day and are increasingly capable of producing long, structured, and sophisticated texts. While Orwell could not have foreseen the AI revolution, he predicted that synthetic text could someday replace human writing. In his 1946 essay “The Prevention of Literature,” he observes: “It would probably not be beyond human ingenuity to write books by machinery.” Although he doesn’t linger on this possibility, he laments the depersonalisation and mass production of writing already underway in the 1940s, and these arguments are just as applicable to AI-generated writing today.

Orwell expressed an almost eerie sensitivity to the ways in which literary ability—and even the quality of thought—can decline alongside a growing reliance on automated writing processes. For example, he cites radio features “commonly written by tired hacks to whom the subject and the manner of treatment are dictated beforehand.” The writing itself was “merely a kind of raw material to be chopped into shape by producers and censors.” His experience dealing with the pressures of working in a strictly controlled corporate environment at the BBC during wartime undoubtedly left him with this impression. He also cites “innumerable books and pamphlets commissioned by government departments” created in the same industrial manner.

Orwell’s scrutiny of the “machine-like” creation of “short stories, serials, and poems for the very cheap magazines” holds up particularly well today. In an uncanny anticipation of the process by which millions of users now produce creative content with AI, he writes:

Papers such as the Writer abound with advertisements of Literary Schools, all of them offering you readymade plots at a few shillings a time. Some, together with the plot, supply the opening and closing sentences of each chapter. Others furnish you with a sort of algebraical formula by the use of which you can construct your plots for yourself. Others offer packs of cards marked with characters and situations, which have only to be shuffled and dealt in order to produce ingenious stories automatically.

“The Prevention of Literature” was published around the time Orwell began work on Nineteen Eighty-Four, and it shows. Winston Smith’s job in the Ministry of Truth is to rewrite historical documents to match Party propaganda. He deletes “unpersons” from old news stories and ensures that recorded events always line up with the latest party line, all with the help of his speakwrite dictation machine. He dumps original documents into the Memory Hole for incineration. In the essay, Orwell moves from a discussion of increasingly robotic forms of literary production to the role this shift could play in a totalitarian state:

It is probably in some such way that the literature of a totalitarian society would be produced, if literature were still felt to be necessary. Imagination—even consciousness, so far as possible—would be eliminated from the process of writing. Books would be planned in their broad lines by bureaucrats, and would pass through so many hands that when finished they would be no more an individual product than a Ford car at the end of the assembly line.

In some ways, Orwell’s bleak prophecies would turn out to be more accurate than he could have imagined. The idea that human thought would be replaced by an “algebraical formula” and that consciousness would be eliminated from the writing process is now a reality on a vast scale (though the question of whether consciousness will emerge from AI systems remains open). But Orwell filtered his predictions about the future of writing through his fixation on state power and the possible emergence of a “rigidly totalitarian society,” and this led him astray. In such a society, Orwell assumed that “novels and stories will be completely superseded by film and radio productions.” To the extent that people would want to keep reading, “perhaps some kind of low-grade sensational fiction will survive, produced by a sort of conveyor-belt process that reduces human initiative to the minimum.” He concluded: “It goes without saying that anything so produced would be rubbish.”

Orwell was wrong that assembly-line modes of writing are incapable of creating anything worthwhile. In a way, it would be comforting if his prediction about state-enforced rubbish-production turned out to be true—in that case, the replacement of human writing with synthetic content would be a product of mere coercion. But it is a choice. Individuality isn’t being drained from writing by faceless bureaucrats, but rather by faceless algorithms. ChatGPT was the fastest-growing consumer app in history, and usage rates for AI are surging. As I write this essay, I’m receiving constant prompts from AI to “help me write.” When I type an email, words and phrases are now highlighted for AI revision. AI-produced content has become a scourge in universities, and it is rapidly displacing human writing in new articles online. Elon Musk recently declared: “AI content will vastly exceed all human content.”

None of this is to say we should spurn AI, which is a powerful tool for synthesising information and developing ideas. Issues like hallucination and the spread of misinformation strike me as solvable technical problems, and the emphasis on what AI can’t do seems absurd when its abilities across diverse cognitive tasks have exploded in just a few years. At this stage, we don’t know what the overall impact of AI will be on creative content, the job market, or human cognitive capabilities. But we can make two fairly safe assumptions: humans will write less, and more of the content they consume will be generated by AI. Beyond Orwell’s prescience about the possibility of machine-generated writing, there’s a deeper sense in which a review of his life and work offers a warning about the AI era.

II.

“Literature is an attempt to influence the views of one’s contemporaries by recording experience,” Orwell writes in “The Prevention of Literature.” The transmutation of experience into prose was what he did best—his hatred of imperialism and oppression was catalysed by five years as a colonial policeman in Burma. His socialism was initially formed in the streets of Paris and London, where he went on “tramping expeditions” to develop a greater understanding of poverty. Those socialist convictions were shaped and hardened in Barcelona amid an anarchist revolution in the early days of the Spanish Civil War. In Homage to Catalonia, Orwell writes that he immediately recognised what he saw in revolutionary Spain as a “state of affairs worth fighting for,” and he did fight for it, taking a bullet in the neck from the rifle of a fascist sniper.

Then Orwell experienced a Stalinist purge. In Spain, he fought under the banner of the Partido Obrero de Unificación Marxista (Workers’ Party of Marxist Unification, or POUM), which had been designated a Trotskyist organisation by Soviet agents and targeted for liquidation. Orwell’s friends were thrown in prison while Barcelona—which had so inspired him just months earlier—was inundated with Stalinist propaganda declaring the POUM to be a covert fascist group. Orwell and his wife Eileen barely escaped the country without being arrested. The police searched their room at the Hotel Continental, and Soviet archives later revealed that they were under surveillance and slated for arrest for the crime of “rabid Trotskyism.” After they left the country with their friend John McNair, they saw a newspaper at a French train station declaring that McNair had been arrested for espionage. “The Spanish authorities had been a little premature in announcing this,” Orwell observes. “Fortunately, ‘Trotskyism’ is not extraditable.”

“The Spanish war and other events in 1936–7 turned the scale and thereafter I knew where I stood,” Orwell wrote in 1946. “Every line of serious work that I have written since 1936 has been written, directly or indirectly, against totalitarianism and for democratic socialism, as I understand it.” Orwell’s firsthand experience of Soviet repression gave him unique insight into the mechanics of totalitarianism, which he drew upon for his greatest works: Animal Farm and Nineteen Eighty-Four. “You have got to realise what was the feeling of the time,” Orwell writes in Homage to Catalonia, “the horrible atmosphere of suspicion and hatred, the lies and rumours circulating everywhere, the posters screaming from the hoardings that I and everyone like me was a Fascist spy.”