The setup
A new paper from researchers at Columbia and the Santa Fe Institute puts transformers inside controlled environments where the ground-truth Bayesian posteriors are available in closed form, and asks a sharper question than usual: can a neural network do Bayesian model selection, not just Bayesian filtering within a fixed class?
The framework, called Bayesian wind tunnels, builds small environments with known posteriors over hypothesis classes. The most interesting one uses fixed-point-free involutions, the family of functions where f(f(x)) equals x. The defining property is purely relational, which strips away the statistical shortcuts that an integer-token benchmark would let a model take.
A 2.8M-parameter transformer, run across three seeds, hits 0.01-bit entropy agreement with the Bayesian optimum on the involution task, and that result holds whether the tokens are integers or opaque symbols whose meanings change every episode. The model is not memorizing. It is genuinely computing the posterior.
The same architecture also works on non-nested class comparisons, like involutions versus 3-cycles, where neither hypothesis class is a subset of the other. On those tasks, the model gets class-posterior MAE under 0.001, which is the harder test, because subset bias and simplicity priors no longer help.
The sharp boundary
Things fall apart at a specific edge. When the discriminative statistic requires arithmetic, modular addition or multiplication in the form f(x) equals cx mod p, the architecture succeeds with integer tokens and fails completely with opaque symbols. That boundary persists under 112x scaling, from 2.8M parameters all the way up to 316M, which is the paper's strongest result against the "just scale it up" response.
A stationarity control shows why. Opaque tokens with a fixed relabeling still succeed at 0.009-bit MAE, which means stable semantics, not integer identity, are what enable the model to compile the right circuit. The paper's header subtask diagnostics localize the failure to the composition of header inversion with arithmetic, rather than to header parsing itself.
What frontier models do
Probing frontier LLMs on the same tasks shows qualitative Bayesian behavior but a large calibration gap, around 55x. The paper is careful to note that the probing is lossy, so the gap is directional rather than exact. Still, the gap is large enough to matter: frontier models act like they understand the structure, but their internal posteriors are not as sharp as the structure would warrant.
For the broader research community, the paper is a useful template. Most claims about whether transformers do Bayesian inference are made on benchmarks where the Bayesian answer is not actually known. By building environments with closed-form posteriors, the authors give the field a way to test those claims with the kind of numerical precision that is otherwise impossible.