America's smartest AI optimist

A single page of fiction can cover 1,000 years of in-story time; 1,000 pages of story can take place in an instant. That's a neat bit of magic, and it profoundly bothers the kind of people who study literature. Experts have spent years — decades, even — trying to gauge how fast most in-story clocks tick. They tediously counted the words in thousands of books; they laboriously hand-coded computer programs to measure the passage of fictional time. Yet for all their brute-force efforts, they couldn't agree on something as simple as how much time the average page of fiction covers.

So it was kind of cool when, last year, ChatGPT did it.

Given a well-designed prompt and a passage of fiction, ChatGPT could ingest the text and spit back a fast, precise estimate of how much time had elapsed in the passage. A chunk of "Jane Eyre"? About a week. A passage of the same length from "The Big Sleep"? Seventy-five minutes. Over the past few hundred years, the bot calculated, literary time has been slowing down. The average page of literature used to cover an entire day of time; now it barely gets through an hour.

The well-designed prompt came from Ted Underwood, an English professor at the University of Illinois. In a world filled with AI skeptics and chatbot alarmists, Underwood is making one of the strongest and most compelling cases for the value of artificial intelligence. While some (me among them) fret that AI is a fabulizing, plagiarizing, bias-propagating bullshit engine that threatens to bring about the end of civilization as we know it, Underwood is pretty sure that artificial intelligence will help us all think more deeply, and help scholars uncover exciting new truths about the grand sweep of human culture. Working with large language models — the software under a chatbot's hood — has made him that rarest of things in the humanities: an AI optimist.

To be clear, chatbots don't read, and Underwood knows it. They don't have opinions on how good a detective Philip Marlowe is. But a bot can do all sorts of interpretive tasks that used to be thesis fodder for overworked literature students. And from that data, Underwood believes we'll finally be able to see the bigger picture — one that can be grasped only by surveying and analyzing centuries of literature, in hundreds of languages.

"The stuff we legitimately really didn't know, that was important, was often on this bigger scale," Underwood says. "We couldn't see it because it was like the curve of the horizon. You need to get some distance on it." And the best way to get that distance, Underwood believes, is to train a digital model of language on terabytes of human writing.

In other words, to use AI.

Underwood's father was a computer scientist, and as a kid Underwood learned to code. (Back then it was called "programming.") But in the 1990s, just as personal computers were beginning to transform the world, Underwood decided — with "impeccable timing," as he puts it — to go to grad school in English. 

As a student, Underwood tried to use digital tools to analyze literature. But in those days, there weren't any databases with enough texts to make it practical. The earliest computers, in essence, weren't as well-read as the average grad student in literature.

Then came Google Books. Google's effort to ingest the entirety of the world's published material into its insatiable informational maw may not have been a great development for libraries or writers, but it was supercool for data scientists — and for data-minded literary analysts like Underwood.

Before Google, digital literature analysis had been a lot like the analog kind: read, react, maybe count the occurrences of something you were studying (places, pronouns, money, etc.). But now, with Google Books, Underwood could create statistical models not of passages or books, but of entire genres. Science fiction, mystery, romance — what, in concrete terms, made them different from one another? His book "Distant Horizons" set out to answer that question. Are books like "Frankenstein" and "The War of the Worlds" science fiction, even though they were written before the editor Hugo Gernsback coined the term? Turns out they are. By measuring the occurrences of sublime, large-scale stuff like vastness and infinity — as well as mundane stuff like human pronouns and big numbers — Underwood was able to tell which fi is the most sci.

Soon after "Distant Horizons" came out in 2019, large language models like ChatGPT emerged on the scene. That changed the literary-analysis game even more than Google Books had. LLMs, at their most basic level, operate by figuring out the statistical probabilities of which words are most likely to come after which. They don't "understand" or "know" anything. They're just converting words to numbers and solving equations. But in the course of their massive and mindless computations, they also calculate how closely words are related to one another, based on their context.

In linguistics, the idea that words get meaning from context is called distributional semantics. That concept might explain why LLMs have demonstrated some seemingly surprising abilities, like being able to tell where and when a whole bunch of famous events took place, or infer the relationships among colors (orange is more like red than blue, say). Language encodes all kinds of knowledge and cultural wisdom — and LLMs encode language.

This is some heavy-duty philosophy. Language isn't just communication; it's a substrate for thought and a carrier wave for culture. And Underwood thinks LLMs are tuned into that wave. For scholars, the point of reading, of writing, of studying language isn't just to generate an essay or critique a poem, but to figure out what we think, and how best to express it. Underwood hopes that LLMs, with their sophisticated statistical models of language, might help us uncover new insights into our own minds. In his view, they do more than simply parrot sources stochastically, not because they're "thinking" — but because we're there to listen to them. 

"I'm not one of those people who thinks these models might be slightly conscious," Underwood says. "But I do think their statements have meaning. I don't mean there's a conscious mind in the machine. I mean, I'm describing the interaction between me and the computer."

Underwood is palpably excited about the kind of insights that this AI-human partnership can offer into literature. "We're going to be able to talk about things like plot and character motivation," he says. "Not just 'count the instances that money appears in a text,' but asking what the money is doing in the plot." AI might even be able to model something as ineffable as what keeps people turning pages — what makes a Stephen King novel unputdownable. It might unpack the structures best at generating … 

…

suspense! AI trained on large language models will be able to score new insights on the most fun parts of the stories that we love to read and tell. Maybe that sounds like something only the resident of an ivory tower could love. Who cares about stuff like quantifying suspense, or clocking the passage of literary time? But Underwood's point is that AI can help bring the far-too-rarefied world of literary criticism back down to earth. "We care about this stuff because people enjoy stories," he says. "And we should keep that central to what we're doing."

Still, even a well-intentioned alliance with our robot overlords is controversial. Distributional semantics isn't the only way to think about language and meaning. There's also a "denotational" approach, which basically says that words mean what they're talking about — the actual thing. And if that's true, well, LLMs are too stupid to live.

I lived in Tokyo for a semester in college, and I spent a week with a local family. One day, while we were out driving, the son and daughter offered me a piece of candy. But be careful, they warned me, because the candy was very [a word I didn't know]. I asked them to say it again, and what it meant, but my Japanese wasn't good enough for me to understand them. 

Finally, I just ate the candy, which turned out to be the flavor of toxic-acid-strip-the-finish-off-a-cabinet lemon flavor. That's how I learned, indelibly, that "suppai" means "sour."

Today, almost 35 years later, suppai isn't just "sour" for me. It's "that ultralemon candy I ate in the back of a car in a Tokyo suburb." That's heavier than anything I might look up in a dictionary. 

No AI can learn like that. That's why it's hard to trust them. Humans bring more to language than mere vocabulary. "If all you have is the distribution of words, it's very flat," says Emily M. Bender, a computational linguist at the University of Washington who is a cautionary voice on the dangers of LLMs. "Nobody denies that distributional semantics is a thing, that words with similar meanings appear in similar contexts. It's that next step — to 'therefore this thing has a model of the world' — that becomes problematic."

The key is what's actually in the training data — the "large" in the language model. If researchers are just using a bot like ChatGPT to draw conclusions about "Anna Karenina" or "Things Fall Apart," that's a big problem. Because the companies behind those bots keep their data mostly secret, which makes any research based on them suspect. What texts did the bots learn from? Which cultural assumptions feed their analysis? What are their blind spots? Researchers have no way to know.

But if researchers train a chatbot on the particular texts that interest them — if they curate the bot's diet — well, then you might have an academic power tool. Underwood says universities and scholars should pool resources to create their own large language models, built from materials specific to their research needs. One new LLM, MonadGPT, was trained on 11,000 texts from the 17th century or earlier. Theoretically it can proxy a 17th-century mindset in a way no living human could.

"I think it's really critical that we train language models that are capable of modeling multiple perspectives," Underwood says. "If these get used in education, we absolutely do not want students coming away with the view that there's one received wisdom about the world."

There are plenty of ways an approach like Underwood's could go horribly wrong. If an LLM can be trained on 17th-century texts, it can just as easily be trained on QAnon forums, or a dataset that presupposes the superiority of one religion or political system. Use a deeply skewed bubble machine like that to try to understand a book, a movie, or someone's medical records and the results will be inherently biased against whatever — or whoever — got left out of the training material.

But that danger is precisely why Underwood believes we need to learn to use AI to explore deeper questions of culture and knowledge. Left to the devices of Silicon Valley and corporate America, LLMs will inevitably tend toward trade secrecy. But in the humanities, their biases will be made visible, and maybe even useful.

"The next stage of pushing these things forward is going to involve choices of a kind we think of as belonging to the humanities," Underwood says. "If the humanities don't want to be left in the dust, we have got to be sitting at that table, talking about alternate shapes for this technology. Nothing against Google and Microsoft, but they should not completely determine how we think and write." For an AI optimist like Underwood, it's not the AI we need to distrust. 

Adam Rogers is a senior correspondent at Insider.