Experiments 30
Guessing machinesGhosts in the Static
Fifteen open language models were shown the same 600 fair coin sequences and asked whether each looked rigged. Each dot is one sequence: across, what a model says; up, what its own predictions imply. Step through training and watch the two come apart. Every number comes from a sealed, pre-registered run.
PyTorch · Transformers · SplitMix64 · SVGView source
Qwen2.5 3B baserigged or fair
What it says follows what it expects.
Its answer barely depends on the coin (usable in 0 of 3 wordings), so this pattern is a faint echo of how it reads the flips, not a verdict.
+0.64agreement, 95% interval +0.59 to +0.69
- 4 or fewer switches
- 5 or 6
- 7 or 8
- 9 or 10
- 11 or more
Coin 232 of 6001 switch
- H
- H
- H
- H
- H
- H
- H
- H
- H
- H
- H
- H
- T
- T
- T
- T
Log-odds of “rigged” against a typical coin, by switches in 16 flips.
Same flips framed as coin flips or as a system log.
Notes
What the square shows
Each dot is one of the 600 fair coin sequences from the study, 16 flips each, regenerated here from the same seed the models saw. Its color is how often the coin switches between heads and tails.
Across is what the model says. It is asked whether the flips came from a fair coin or a rigged one, in three wordings and with the answer order swapped, and the score is how much it leans toward rigged. Up is what it expects: the same flips are placed after “Here are 16 flips of a fair coin” and after “…of a rigged coin”, and the dot rises when the model finds them likelier from a rigged coin.
If the answer came from the model’s own predictions, the dots would hug the dashed diagonal. The solid line is the fitted slope, the agreement. The coin panel ranks the picked coin against the other 599 on both readings.
What the sealed run found
7 of 8 registered predictions held. In all five base and post-trained pairs, post-training weakened the link between saying and expecting: Qwen went from +0.64 to +0.21, and Llama Instruct now says the opposite of what it expects. With the zorp answers factored out, the drop holds in 4 of 5 pairs; OLMo 2 1B, whose answers never depend on the coin, drops out.
Most of the extra switching that tuned models expect is not about coins. OLMo 2 1B expects a switch 0.506 of the time as a base model and 0.607 after tuning, and 0.489 to 0.578 when the same letters are framed as a system log.
One prediction failed: OLMo 3 never picks up that habit. Its reasoning model expects a switch only 44% of the time, so it expects streaks, yet it calls streaky coins rigged (r = −0.55).
Why there is a zorp button
This experiment began as a search for pareidolia. In a sealed pilot, Qwen2.5 3B Instruct looked at 1,000 grids of random characters and was asked whether each hid a face, or a “zorp”. Both maps of what it saw were reproducible (split-half 0.63 for the face, 0.59 for the zorp, against a chance level near 0.19), and they matched each other at 0.98.
A model’s exact answer is a fixed function of what it is shown, so any question sorts the inputs into a tidy pattern. Here every question has a twin with made-up labels, and an answer only counts as a judgment when it moves far more with the coin than the twin does.
How it was sealed
The predictions, thresholds, model revisions and the hash of every analysis file went into a protocol that was itself hashed (3447c2d0127c) at November 19, 2026 at 4:43 PM UTC, before the 600 coins of seed 200 existed. Calibration on other coins had already changed three predictions, and the protocol says which and why.
Three passes are close: one threshold met exactly, one cleared by 0.015, and one pair (OLMo 3 7B Instruct) cleared zero by 0.001. Checks added after the run, such as factoring out the zorp answers, are labeled as such.
Limits
These are small open models, from one to seven billion parameters, on one task with three wordings. The intervals cover which coins were drawn, not other wordings or retrained models. No human ratings were collected; the comparison with people rests on published studies.
Sources
- Falk and Konold, Making sense of randomness, 1997
- Gosselin and Schyns, Superstitious perceptions, 2003
- Griffiths and Tenenbaum, From algorithmic to subjective randomness, 2003
- Hu and Levy, Prompting is not a substitute for probability measurements, 2023
- Van Koevering and Kleinberg, Heads, tails, and AI fails, 2024
- Team Olmo, Olmo 3, 2025
Weights: Qwen2.5 3B, Llama 3.2 3B, OLMo 2 1B and OLMo 3 7B from Hugging Face at pinned revisions, scored in float32.