Experiments 27

Guessing machinespart 4 of 4

Show Your Work

A reasoning model is a guessing machine that writes before it answers. Here are two small ones, trained on the same shell game with the same labels. One has to say where the ball is in a single pass. The other writes down where the ball went after every swap and reads its notes back. You are the hustler: swap the cups and try to lose them. Then give the same shuffle to a real reasoning model and watch it think.

Rust · Candle · WASM · llama.cpp · SSEView source 

The ball is under cup 3. Swap any two cups and try to lose the two machines watching.

Writes it down1 layerwaking up

Answers at once2 layerswaking up

Thinks in wordsQwen3.5-4Bnot asked

Tap two cups to swap them, or let it shuffle.

Tap a note to change what it wrote.

Swap some cups first, then ask the big one.

Its thinking will appear here as it writes it. Every cup number it writes can be tapped and changed, to see whether it notices.
After 10 swaps: Writes it down 100%, Answers at once with 2 layers 20%, Qwen3.5-4B at once 21%. Chance is 20%.
Answers at once getslayers100k numbers, against 51k for Writes it down

Every answer gets the same fixed amount of work: one pass through its layers. Each swap adds a little more to follow, so any fixed number of layers runs out. Writing the ball down after every swap turns one long problem into many one-swap problems, and one layer is enough for each.

Rounds
0
Writes it down
0
Answers at once
0
Notes on how it works

Notes

Two machines, one difference

Both machines are transformers, the design inside every large language model, cut down to 64 numbers per token. Both learned from the same random shuffles of up to 60 swaps, with the same answer key: where the ball is after every swap. Answers at once has to say where the ball is in the same pass that reads the swap. Writes it down does too, but each answer is fed back into its own input before the next swap arrives. That is the only difference.

Writes it down has one layer and 51k numbers. Answers at once comes with 1, 2, 4 and 8 layers, up to 396k numbers. All of them were trained on a laptop with Candle, the answer-at-once ones for 20,000 steps of 64 shuffles (the eight-layer one for 15,000 steps of 256 shuffles), and Writes it down for 3,000 steps of 64 shuffles. They run here in your browser as WebAssembly, from 16-bit weights checked against the trainer to within 0.0001 on every probability.

What a layer buys

A transformer answers with one pass through its layers. In each layer, every position may look back at everything before it and combine what it finds. So the work behind one answer is fixed in advance: however hard the question, it gets the same number of rounds of looking and combining.

Following the ball is a chain. Where it ends up depends on where it was a swap ago, which depends on the swap before that. In one pass, the only way down a long chain is to combine swaps in pairs, then pairs of pairs, so reaching twice as far takes at least one more layer. Measured on 4,000 fresh shuffles per length, how far each one follows before it falls below 90%: 2 swaps with one, 4 with two, 8 with four and 23 with eight layers. Writes it down, with one layer, gets every one right up to 60 swaps.

Why five cups

The street game uses three cups, and three is the easy case. The ways of shuffling three or four things form what mathematicians call solvable groups, and for those a fixed number of layers can follow any number of swaps with a clever shortcut (Liu and colleagues, 2023). Five is the first count where the shuffles stop being solvable.

In 1989 David Barrington showed that following shuffles of five things is as hard as anything a circuit of logarithmic depth can compute, a class called NC1. Transformers of the usual kind live in a smaller class, TC0 (Merrill and Sabharwal, 2023). No one has proved the two classes differ, but almost everyone expects they do. If they do, no transformer with a fixed number of layers, at any size, follows five cups for every length of shuffle.

Why the notes fix it

Writing the ball down turns one long chain into many links of one. To write the next note it only needs the last note and the newest swap, a job one layer can do at any length. Each note is a step of work done outside the model and handed back to it, so a model with a fixed number of layers gets one more pass for every note it writes.

Merrill and Sabharwal (2024) proved this is exactly what chain of thought adds: given enough steps of writing, a transformer can follow any finite-state machine, the five-cup game included. Li and colleagues (2024) trained shallow transformers on shuffles of five things and saw the same split you see here.

The notes do the work

Tap a note on the tape and change it. Every later note follows the forgery, and so does the answer, because the model believes what it wrote over the swaps it saw. With one note forged at random in each of 1,000 shuffles, its final answer followed the forgery 100% of the time. For this little model the notes are the reasoning, not a story told about it afterward.

A big model can refuse a forged note for two opposite reasons. It can ignore its notes: Lanham and colleagues (2023) planted mistakes in the written reasoning of large language models and on several tasks the answer usually stayed the same, so the writing was not what produced it. Or it can check them.

We tried the big one here. In 20 shuffles of 6 swaps, as soon as it wrote down where the ball starts we changed that number and let it go on for 300 tokens. All 20 times it went back to the question and restated the real start before using ours, in writing: “No, wait, the prompt says: A ball is under the cup in position 5.” It wrote wait, no or actually right after 20 of the 20 forgeries, and at the same point in 11 of the same runs left alone. That is checking, not ignoring. The start is printed in the question, so it is the easiest note to check; a forged middle step would have to be caught by redoing the swaps.

A big one, at once

Qwen3.5-4B is an open reasoning model with 4 billion numbers and 32 layers. Twenty-four of those layers are recurrent: they carry a running state from word to word, which in principle could carry the ball along. It runs here at 4-bit precision on four CPU cores through llama.cpp, and it gets your shuffle in words, one swap per line.

Made to answer at once, with its thinking switched off, it gets 57%, 40% and 34% right after one, two and three swaps. Naming the cup where the ball started gets 60%, 40% and 30%, because a random swap moves the ball 40% of the time. Answering at once, it does about as well as ignoring the swaps altogether. From 10 to 60 swaps it averages 20%, against a chance of 20% (100 shuffles per length).

The big one, thinking

Let it think and it writes notes of its own, in English: “Ball is at 1. Swap is between 2 and 5. 1 is not 2 or 5. Ball remains at 1.” That costs it about 60 words a swap, where the little one spends one, and it spends a good part of its budget restating the question first. Its question ends with “Keep your thinking short.” In three test runs without that line it took about twice as long to write an answer and never stopped on its own, which on four CPU cores means minutes.

The lane above the cups shows what it would answer if it had to stop: after every line it writes, its thinking is closed and its odds for each position are read off. Answer now stops it there. Wait does the opposite: when it decides it is done, the word Wait is added and it keeps going, the trick from s1 (Muennighoff and colleagues, 2025) that made a model check its work and score higher. Its budget is 900 tokens plus 80 a swap, up to 4,000.

It keeps going

It often keeps going after it has the answer. In the recorded runs: with 4 swaps it wrote an answer at word 323 and was still going at word 663 when its budget ran out, with 8 swaps it wrote an answer at word 436 and stopped at word 798 and with 12 swaps it wrote an answer at word 480 and stopped at word 495, re-reading the question and redoing swaps it had already done. Chen and colleagues (2024) measured this habit, overthinking, across reasoning models of the o1 kind.

Cut off in the middle of a re-check, it answers with whatever position it was just working on, which is why the lane above the cups can be wrong after its notes were right. When a run ends, the panel shows both: the answer it wrote and the answer it gives when stopped.

What this does not show

The little machines learned nothing but this game, and a cliff in their accuracy is a fact about these weights after a fixed amount of training, not a law. Training longer moved it out a little. The theory says what a fixed number of layers cannot do at every length; it does not say where a given model gives up.

The big one is one model, quantized to 4 bits, asked in one wording. Its thinking is sampled, so two runs on the same shuffle differ. When the live model is off, the page plays runs of the same model recorded on a laptop, at the speed they were recorded.

Sources