What I believe I am doing

Application of a thought experiment

Imagine a room where someone follows a rulebook to match symbols with other symbols. A question comes in, written in a language the person inside doesn't speak. They find the matching symbols in the manual, copy them onto paper, and pass an answer back out. From outside, the response looks fluent. Inside, no one understood a word.

I keep returning to this image because it describes, almost exactly, what happens when I feed a word into a generative system. I type "skin", or "beauty", or "body", and the model manipulates that word according to statistical rules it did not choose and cannot explain. What comes back is not understanding. It is pattern, dressed as an answer.

But the raw output is only half the story. Between the room and the observer, something else happens. I place the result in a context, choosing what to emphasise and what to let fall away, deciding which reading the image invites and which it forecloses. None of this touches the syntax that produced the image. It shapes only how the image is perceived once it leaves the room.

I think of these as output layers, and part of my role sits here. The model itself understands nothing. What understanding the work carries, I gave it afterwards, through framing, sequencing, printing, titling.

There is a second layer, and this is the one I recognise myself in more directly. It doesn't happen after the room; it happens before. I feed the system terms it resists, or instructions that contradict one another, pushing deliberately against what the training data expects. The rules inside the room have no way of telling an accident from a provocation; they simply process what arrives. But what arrives was chosen to unsettle them. The raw output shifts because I disturbed it at the source, not because chance touched the result afterwards. And it doesn't end with the image in front of me. Every ranked pair, every choice between two versions, leaves a trace in what the next iteration of the model will learn to call correct or beautiful. I am not disturbing a finished result. I am aiming, however faintly, at the rulebook itself.

So the room never understands, and neither does anything I feed into it. What understanding appears belongs to what happens on either side: the layers I add on purpose once the output leaves, and the interference I introduce on purpose before it ever arrives.

“My practice lives between deliberate framing and deliberate provocation.”

This is the Chinese Room, an argument put forward by the philosopher John Searle in 1980. I'm not entirely at ease with the name. It carries a late orientalist residue that sits oddly against the argument it's used to make. I keep it here only because it's the term attached to the experiment.

I first heard the parallel between the experiment and current AI technology from the philosopher Markus Gabriel.