Motte-ly Defiance
Resisting the machinic satiation of Ai writing tools
[This is a long one. It’s got more links than normal, more footnotes than normal, and I ended up cutting a couple paragraphs as it was. I’m finding, though, that it’s been productive for me to periodically assess my attitudes about Ai as they’ve changed over time. So here’s where I’m at right now…]
I don’t spend a great deal of time thinking about spelling, but there are a few exceptions. Like many nerds, I remember participating in K-12 spelling bees. The only actual moment that I vividly recall from those days is a single point of failure, when I spelled receptacle with an i instead of an a. I can’t tell you whether it was the first round or 50th, just that I remember standing on a riser with a handful of other students, the experience of having my mind go blank, and the sinking feeling, as I articulated the letter i, that it (and I) was wrong.
More generally, as someone who reads and writes a lot, I’ll occasionally bump up against words that are correctly spelled but don’t feel right, although this is a more temporary phenomenon for me. I think of it as the written version of semantic satiation, so much so that I still have to occasionally use a search engine to remind myself that the “technical” term for what I’m describing is wordnesia.1 (Wiktionary defines it, mistakenly in my opinion, as a synonym for SS.) If you look over that piece from Smithsonian, though, what it describes feels more like inverse satiation. It cites a Slate interview with Charles Weaver, who “describes how memory glitches trip us up when, for some reason, we slow down during tasks2 that are usually done on autopilot.” Wordnesia only happens intermittently to me and it rarely lasts (or even recurs with the same word) beyond a single session.
All that being said, I am almost certainly an above average speller. Part of the reason that I was thinking about spelling today is the relationship that I had, early in my career, with the spellchecker. It’s not something that I’ve ever researched explicitly, but I can pinpoint a few distinct phases in my interaction with this tool.
When I was a graduate student, spellcheck used to frustrate me a great deal. I wanted to be able to catch typos, because I was (and am) a believer in getting into the flow of writing and returning later to fix mistakes. When I’m stopping every few words or couple of sentences to touch up my prose, it slows me way down, and makes it difficult for me to think globally about what I’m saying. In its early days of implementation, spellcheck was a constant irritant, not because I’d gotten any worse at spelling, but because I was using specialized language (and lots of proper names) that didn’t show up in the default dictionary. Even though I knew they were coming, every time one of those squiggles showed up, it would take me out of my flow.
If I remember correctly, the background autocorrect that we all take for granted rolled out in the mid 1990s (maybe as part of Word for Windows 95?). It took a while for it to completely saturate the market and become a core feature of all word-processing. But here’s how I knew when it had. Back in those days, we would still talk explicitly about spellcheck in our writing courses, and warn students not to rely blindly on it. But over the course of 2-3 years, I started seeing one error in particular pop up across a broad range of student papers. Out of nowhere, my students started signaling a great deal of insubordination and/or noncompliance in their essays. It wasn’t directed at me, but it stood out in the context of their work.
What happened, in dozens of sentences, was a massive uptick in the word “defiantly.” One of their sources defiantly believed thus-and-such, while another illustrated a position that was defiantly true. It was really pretty funny. It turned out that students were misspelling definitely (making the same mistake in reverse that I had with receptacle) replacing the second i with an a: definately. And spellcheck persistently misunderstood this error, I guess, assuming that the a and n were accidentally swapped (and an e inadvertently added). When they should have been proofreading, students were simply running a spellcheck, and taking the first option their word processor presented to them for any instance where they found the underline. Spellcheck, for them, wasn’t the final stage in a longer proofreading process, but rather a substitute for it.
Machinic Satiation
If I had paid closer attention to the intrusion and invasion of writing by spellcheckers, I might have drawn an analogy between semantic satiation and what I want to call machinic satiation3. In the former, you repeat a word so often that it just becomes a sound; similarly, using spellcheck so that you can quickly eliminate red squiggles turns a tool designed to support proofreading into something almost like a video game. Pew pew pew! By machinic satiation, then, I mean something a little more internalized than saturation, which was also true of spellcheck. Not only is that function ubiquitous, but it has become all but reflexive.
There’s an academic quasi-discipline devoted to these sorts of questions, taking up questions of what’s called technology transfer. A long time ago, as someone interested in sociorhetorical accounts of technology, I read Everett Rogers’ Diffusion of Innovations, one of the canonical books in that field. The technology adoption curve (pictured above), which still frames a great deal of our conversation about the speed at which tech is integrated into our lives, can be traced back to Rogers.
But there’s a phenomenological dimension to that curve as well, in addition to its representative qualities. Technologies that find themselves on the far left end (and their innovators) initially face a figurative and literal “uphill struggle” if they want to arrive at the right end of the graph. The benefit, once you hit the middle zones, is that by that point, the technology (or idea) basically sells itself. It becomes unusual not to adopt it, and people who don’t are laggards (or Luddites, or losers, or…).
One of the interesting elements of the graph is that “chasm” between early adopters and early majority. That gap signifies the point after which machinic satiation might be said to take place (or at least begin), where a particular technology shifts from a strange, new possibility to something we take for granted. Even though there’s still some adoption hill to climb, it acquires a sense of inevitability, and some users are more attuned to that sense than others. The question facing the innovators, then, is how to bridge the gap, as painlessly and inexpensively as possible. One popular answer comes in the form of another chart, which can be overlaid atop this one, that narrates the “hype cycle.”
(Wikipedia discusses this as the “Gartner hype cycle,” although Gartner is the name of the consulting firm where its actual originator, Jackie Fenn, was employed at the time.) There’s a bit of rhetorical ledgerdemain involved in this graph, and not only because it assumes the happy ending of permanently rising productivity. It actually represents (in pseudo-objective fashion) the potential consequences of a bait-and-switch. Those “inflated expectations” are often generated by the folks with a financial stake in crossing the chasm from early adopters to early majority on the adoption curve as quickly as possible, to the extent that inflated expectations might better be understood as the kind of brute force onslaught of marketing that we experienced recently with craptocurrency, and that we’re now undergoing with sports betting, prediction markets, and of course, artificial intelligence. Another form of “bait” is the rhetorical purpose of a graph like this in itself, which might supply consultants with “proof” to recommend that a C-suite pour as much money as possible into inflating those expectations. But neither graph is representative per se. Each presents a narrative (or makes an argument) that benefits some at the expense of others. They present one model for how adoption (or hype) might proceed, not the natural, inevitable order of things.
By way of contrast, consider the development of the microwave oven, which was invented almost accidentally in the 1940s. I’m just old enough to be able to vaguely remember the kitchen where I grew up before it had one. By the time I graduated high school, only 25% of American households had them, a percentage that ballooned to 90% within ten years or so4. But this was 50 years after it was invented, and 30 years after the first units were made available for home use.
Had microwave manufacturers bought into the logic of the hype cycle, and acted accordingly, we’d probably be spending four figures on them, for a unit the size of the average dishwasher or oven5. As it happened, though, the form factor on the microwave steadily shrank, the technology required to create them got better, and the parts less expensive. It became more affordable to own a microwave, and feasible to fit it into existing kitchen spaces. Interestingly enough, the same fellow who invented/discovered the microwave (Percy Spencer) is also credited with the invention of microwave popcorn. I have nothing but my own intuition to support this claim, but I think that the early 80s were pretty crucial for the microwave, given that 1984 was “the first time that microwave popcorn could be stored in the pantry and easily popped in any standard microwave.”
Not that I believe that a national craving for popcorn (or Lean Cuisine meals, which first appeared around then as well) drove the adoption of the microwave, necessarily. But this technology travelled nearly the entirety of the adoption curve in roughly a decade (generations after its invention), and I think that a sizable portion of that was driven by the gradual awareness of the convenience that it provided, and the availability of obvious use cases. The culture evolved alongside the technology. Perhaps the profit margins were never as high, but microwaves, at least to my recollection, weren’t marketed with the same intensive degree of FOMO that contemporary technologies seem to rely upon. But then, this wasn’t an era of monopoly feudalism, regulatory indifference, and enshittification, either.
Motte-and-bailey arguments
Over the past few years, I’ve seen more frequent reference to an argumentative style6 known as motte-and-bailey, named for the medieval configuration of a highly defensible castle or fort (the motte) surrounded by a much looser, more vulnerable area surrounding it (the bailey). The tactic goes something like this: a person will make a controversial claim that contains some element of truth, then when challenged on it, will retreat to the milder, more defensible position. It’s basically (and intentionally) a bad faith argument, one that occupies the same neighborhood as the bait-and-switch, “seriously, not literally,” clickbait, “truthful hyperbole,” and perhaps my least favorite species of the same tactic, the claim that one was “just joking.7” (If you’ve read me for any amount of time, you’ll recognize this neighborhood for its deep dedication to the discursive and corrosive practice of irony.)
Perhaps the most obvious relevance that motte-and-bailey has for this discussion is that it captures the sweeping decline in the hype cycle from promise to reality. We’ve spent a lot of cultural attention on the bailey promises of metaverses, monkey pictures (NFTs!), and Google/Apple/Meta headsets while these “transformative” technologies haven’t done much outside of the continuous upward transfer and concentration of wealth, some of which has been spent capturing our media and our government. For the vast majority of us, our bank accounts and job prospects have certainly been “lightened,” but I don’t know that this is the “slope of enlightenment” that the hype cycle promises.
I want to pick up on something else though, a rhetorical tic that’s been starting to give me itches (not unlike a certain phrase). In this case, my discomfort was occasioned by the NYT piece on LLMs that came out last week, “What 370,000 College Essays Tell Us About A.I.’s Effects on Creativity.” Rebecca Winthrop shares the results from a massive study of college applicants’ personal statements. The availability of LLMs results in an increase of “word-level diversity” accompanied by the homogenization of ideas. If this sounds familiar, it bears some similarity to the study I talked about a couple of weeks ago, where Ai tools showed small individual gains at the cost of collective novelty8.
For writing treated in relative isolation, or in contexts where no one really gives a shit about it, or for users who aren’t particularly invested in their ability to think or communicate, LLMs are perfectly capable of offering marginal improvements. But they also set a ceiling for those improvements, and as the results are fed back into the material upon which LLMs are trained, that ceiling will steadily drop. Perhaps it will eventually only permit “the kind of walking that made benches become men.” Those of us who read widely are already capable of sensing that line, regardless of whether or not a truly accurate crap detector exists, or ever will. If you’d like to see what I mean, I can think of no better recent example than Sam Kriss’ “If you let AI do your writing, I will come to your house and kill you.” I will almost certainly not tattoo the following line from his essay on my body: “Absolutely all AI prose is filler, an expanding foam insulation made of words.”
All of which is fine. Winthrop’s essay, for the most part, provides me with more fuel for the fiery rage that I have stoked in my heart for LLM writing. After she discusses the study, and its thoroughly reasonable, motte-worthy claims, she offers the following paragraph:
This is not to say that A.I. can never support human creativity. Workers with deep knowledge of their craft can use A.I. to streamline technical or administrative tasks in order to focus on the parts of their jobs where originality lives — including teachers having more time to devise engaging lessons and illustrators devoting more attention to developing visual concepts. A.I. gives specialists the time they need to do what humans do best: brainstorming ideas to creatively solve problems.
Allow me to bring some pedantry up from the notes: the (bailey) claim that Ai can NEVER support human creativity is never at issue, and there’s something really irksome to me about using this paragraph to motte-ify what has already been a reasonable argument with concrete evidence for its claims. I am absolutely prone to doing something like this myself: I’ll talk about how ugly or devastating or overstated the alleged benefits of artificial intelligence are, but then I’ll hedge myself by acknowledging the tiny, potential edge cases that negate much of the force behind my argument. And I should stop this.
This NYT article is to say that rampant Ai use does indeed foreclose on human creativity, even though it offers some individual humans the temporary simulation of that quality, as long as they don’t see how uniform those results are compared to all of the others doing the same.
Let’s also be clear about how flawed Winthrop’s hedge is. As a teacher who thinks deeply and carefully about how engaging my pedagogy is, there’s no such thing as “engaging lessons” that exist separately from and outside of all of the mundane “technical and administrative” bullshit. Do I wish that there was less of the latter? Absolutely. But my ability to engage students comes from interacting with them and their writing, even if that work is often exhausting. Creativity and engagement aren’t the alternative to the mundane work that I do; they emerge from it, and offloading “all the boring stuff” to an approximation isn’t the victory that Ai shills seem to think it is. Ironically enough, in the comments, Winthrop talks about having realized that “I had so quickly gotten used to ‘collaborating’ with AI” that she wasn’t able to complete relatively simple tasks without it. If your understanding of writing as a practice is that it can simply be divided into two categories, one of which can be offloaded to an LLM, then you’re not really talking about writing at all.
Even now, after having said all this, I can feel the temptation to fall back to the motte, for fear (I suppose) of offending the many people I know and admire who have taken up these tools and who genuinely believe that there is something redeemable to them. And that’s despite the fact that their baileys include massive environmental degradation, the absence of any sort of viable economic sustainability, speculative financialization by a host of people who’ve already demonstrated how little trust they deserve, and numerous studies that articulate the deleterious effects of cognitive offloading. How much psychosis are we willing to endure before we stop hedging our own efforts to see the limits of these tools clearly?
Ask Chef Mike
I want to close what’s become a pretty long rant by returning to the microwave oven. While its historical development and market saturation bears no resemblance to that of contemporary technologies, that’s where the dissimilarity ends for me. As I’ve thought about it over the past few years, I’ve gotten more and more convinced that the microwave offers a better analogy for artificial intelligence than calculators, spellcheckers, bicycles, and the like9. As with Kriss, and others like Erin Rose Glass, who wrote last year of the “AI Microwave Smell” that wafts off of LLM-augmented prose, there are certain qualities to that sort of writing that set me off. Kriss’ post lines up several examples in order to demonstrate the parallels across multiple contexts. If you work better through contrast, the shift in voice from one chapter to the next in Vauhini Vara’s Searches manifests the distinction between human and Ai writing10. I’ve written in a couple of places about the fact that, more often than not, I can tell the difference.
It makes me think of Gordon Ramsey’s Kitchen Nightmares, both the British and American versions, almost every episode of which I’ve seen. Not every episode is like this, but many are. An episode will begin with a restaurant that is hundreds of thousands of dollars (or pounds) in debt. Upon visiting the place and trying the food, Ramsey will discover (more often than not) that the kitchen has turned significant portions of their cooking over to “Chef Mike,” the microwave oven. Despite the restaurant’s steady loss of custom, the head chef often points to the small (inadequate) group of loyal patrons and claims that they like the food, so it must be good. Sometimes, they’ve got giant, implausible menus that make it impossible to do anything other than freeze and reheat most of their food. Other times, they’ve decided not to hire kitchen staff (or train the ones they have), or the head chef has largely given up on the idea of prep. And often enough, many of the staff, and sometimes the owner, have become so accustomed to Chef Mike’s cuisine that they don’t know any different. The customers seem to like it, they insist, a confirmation bias that literally jeopardizes the economic survival of their own jobs and workplace.
I do recognize the absurdity of offering up an allegory that equates Gordon Ramsey and myself. At the same time, lifelong readers have indeed developed that same sensitivity to language that a world-class chef has with respect to food. Those KN episodes inevitably include a scene where Ramsey helps the head chef assemble a smaller menu of dishes that are prepared with fresh ingredients, and the staff is nearly always astounded by how much better the food tastes when there’s human effort behind the preparation. That’s the message that needs to sink in with respect to language.
It’s the value of cooking, and it’s what we expect (and pay for) when we dine in a restaurant. It’s no less unreasonable to expect the same from the writing we engage with. I don’t want to pay entree prices for something a model shits out, nor am I interested in a world where these tools are used to bait-and-switch us as they steadily corrupt the venues that might provide us with an alternative. (If the recent news surrounding Google is any indication, it’s soon going to be far more difficult to find those human outlets.) More to the point, I don’t believe that the collective quality of our thought and communication will improve if we abandon the teaching of writing in favor of push-button text generation, regardless of how cleverly we position and hedge the latter. At root, it feels to me like taking a cooking class or going to culinary school, and being taught how to use a microwave.
I was mildly heartened this week to see Ethan Mollick, who’s pretty unapologetic in his support for Ai, write that “…using AI for writing has a cost beyond turning off readers, it risks undermining the development of an important human task.” Don’t worry, though. He does beat an unnecessary retreat to the motte in the very next paragraph (“This is not a condemnation of using AI to help with writing in any way,” a sentence that echoes Winthrop’s with eerie accuracy.). But his point is similar to mine, albeit framed less defiantly, and that’s that if we don’t exercise some intention and agency with respect to these tools now, we won’t have the opportunity later. That’s what machinic satiation entails, and I don’t think that we should be self-sabotaging our opposition to it11.
Whew. This was a long one, prompted by the constellation of several things that crossed my radar this week. If you’ve made it this far, I thank you. The next episode will (mercifully) be shorter.
Pedantry incoming: Wordnesia is a horrible, made-up word for this. I didn’t try very hard, but I could find no one credited for it. It’s obviously a portmanteau of word + amnesia, but here’s the thing: amnesia is not am + nesia etymologically. It’s instead a + mnesia. Mnesia (mnesis) is the Greek for memory (preserved to the present day in the offbeat spelling of the word mnemonics). Wordmnesia would look even worse, for sure, but why on earth couldn’t they just have gone with termnesia?
I’m not going to dig deeply into this here, but there’s a parallel here between what Weaver is describing and the types of experiments that Daniel Kahneman detailed in Thinking Fast and Slow. Another way of phrasing this, then, might be that wordnesia happens when we bring System 2 resources to bear on vocabulary that we’ve long since shifted to System 1. At the risk of massive oversimplification, to the point of getting it entirely wrong, this makes me think about satiation as the reverse, reducing System 2 (word) to System 1 (sound) through repetition. (This note gives some hint as to why this interests me.)
Machinic is an odd enough word that my spellcheck is giving me the squiggle each time it appears here. But I’m going with it for a couple of reasons. First, machine feels to me like “technology + banality,” and that’s the endgoal of adoption, for a technology to become so pervasive that we simply take its presence for granted. (There’s an analogy here between this and the way that Nietzche talks about how everyday language is poetic language that has lost its luster, like a minted and imprinted coin that wears down to plain metal.) And second is the association with the capital-M Machine that Paul Kingsnorth (like many before him) decries. It implies a bundle of material technology as well as social and cultural attitudes focused on reducing human agency. Thinking here too of Berry’s “next great division.”
I’m no more a kitchen appliance historian than I am a specialist in spellchecking. This basic timeline comes from a little bit of basic research, and the numbers themselves from Whirlpool’s history of the device.
I heartily recommend Colin Carnaby’s “In the Future All Food Will Be Cooked in a Microwave, and if You Can’t Deal With That Then You Need to Get Out of the Kitchen.” It imagines how a restaurant owner might have made the same sorts of arguments on behalf of the microwave that today’s tech moguls are talking about Ai.
A little more pedantry: Wikipedia refers to motte-and-bailey as a fallacy, but I lean towards categorizing it as a specific rhetorical tactic. If you follow that link, you’ll find that the philosopher who coined this usage (Nicholas Shackel) describes it as a doctrine rather than a fallacy, although I might quibble even with that.
You know, like when an unfit leader announces that he wants to cancel his country’s midterm elections, and his press secretary has to spend time in an official briefing walking back the corrupt fascism of her boss?
From the abstract for that study: “This dynamic resembles a social dilemma: With generative AI, writers are individually better off, but collectively a narrower scope of novel content is produced.“
I talked about metaphors for Ai in the my annual departmental address last fall, which I posted here. I’ve been thinking about this for a while.
At the risk of offering hindsight, my review of Vara involved me coming to terms with how offput I was by the Ai-written chapters, in a book that I was predisposed to like and appreciate.
One final link, to Dave Karpf’s recent discussion of how to resist AI. “Slow the pace down, and you lose the aura of futurity. And then the whole house of cards collapses.” That aura of futurity is exactly how these companies cross the chasm from possibility to inevitability. His broader point is that resisting data centers might not be the best way to resist Ai, but that it’s still a legitimate tactic working towards that shared goal.
Banger of a post. I have an unpublished blog post about the motte and bailey related to AI and comedians. If you want to read it, I can email it. I Never published it because it's sorta a rant.