The Experiment: Puzzles With a Difficulty Dial

Reasoning models produce long chains of intermediate steps before answering, and score far better than their predecessors on mathematics and coding benchmarks. Whether that constitutes reasoning, or very good pattern matching over problems resembling training data, is the question everyone wanted settled.

Standard benchmarks cannot settle it, for a reason that has nothing to do with model quality. Their problems were published, so a model may have seen them. A high score is consistent with reasoning and with recall, and the benchmark cannot distinguish which.

There is a second problem. Benchmark problems vary in difficulty in ways that are hard to characterise. One question is harder than another, and saying why, or by how much, is not usually possible. So you can observe that a model does worse on a harder set without knowing what dimension of harder is responsible.

A team at Apple set out to fix both, and the design they arrived at is genuinely good. It is worth understanding on its own terms before the criticism, because the criticism is about execution rather than about the idea.

The solution was to abandon benchmark questions and use algorithmic puzzles instead, chosen for one property: difficulty can be increased continuously while the underlying logic stays exactly the same.

Tower of Hanoi is the clearest example. Move a stack of discs between three pegs, one at a time, never placing a larger disc on a smaller one. The rules are identical whether there are three discs or fifteen. What changes is the number of moves required, which doubles with each disc added.

That gives a clean difficulty dial. Nothing conceptual changes as the dial turns, so a model that understands the puzzle at five discs understands it at ten, and any drop in performance is attributable to scale rather than to new concepts.

The team used four such puzzles, spanning compositional depth, planning, and constraint satisfaction, so the result would not depend on one puzzle's quirks.

And because these are classical puzzles with known algorithms, each came with a simulator, which allowed automatic checking of both the final answer and every intermediate move. That is more than most evaluations offer.

It is worth being explicit about the design's virtues, because the eventual criticism can make the whole exercise sound careless, and it was not.

Contamination is largely handled. The puzzles are famous, but the specific instances at specific sizes are generated rather than retrieved from a published set, and there is no corpus of solved fifteen-disc Hanoi traces to memorise.

Difficulty is a controlled variable rather than a judgement. You can state exactly how much harder one instance is than another, which almost no benchmark supports.

Verification is complete. The simulator checks every move, not just the final answer, so a model that reaches the right end state by an invalid route is caught.

And the confound is controlled. Because the logic is fixed across sizes, a performance drop cannot be explained by the model not understanding the task.

This is careful experimental design, and it is the standard the field should be held to. The evaluation is far better than the benchmark scores it was meant to replace. Which is what makes what happened next instructive.

The headline result is what made the paper famous, and its shape is what made it striking.

Models performed well on small instances. As complexity increased, accuracy did not decline gradually the way a system running out of capacity might. It held up, and then collapsed to essentially zero past a threshold.

A gradual decline would suggest a system doing something real and doing it less reliably as the problem grows. A cliff suggests something else: a regime where the approach works and a regime where it does not, with a boundary between them.

The paper's interpretation was that models are pattern matching within a range covered by training and failing outside it, rather than executing a procedure that would degrade smoothly.

A second observation reinforced it. The models' reasoning traces got longer as problems got harder, up to a point, and then got shorter again as they approached the collapse threshold, even with budget remaining. A system trying harder on harder problems should not reduce its effort just before failing.

That combination, a cliff plus reduced effort at the cliff, is a suggestive pattern.

The most quoted result is a follow-up designed to rule out the obvious objection.

The obvious objection is that the model does not know how to solve the puzzle. Tower of Hanoi has a short recursive algorithm, and a model that never learned it would fail regardless of any reasoning ability.

So the team put the algorithm in the prompt. The model was told exactly how to solve the puzzle, and asked to apply it.

Performance did not improve. The collapse occurred at nearly the same problem size as before.

That is a striking result, and it is why the paper landed as hard as it did. If a system is given a correct procedure and still cannot execute it, the failure is not a knowledge gap. It looks like an inability to carry out a specified sequence of steps reliably, which is close to the definition of the thing reasoning is supposed to be.

The conclusion drawn widely was that these systems do not reason, they retrieve patterns, and the appearance of reasoning is an artefact of problems close to training data.

That conclusion is where the rebuttal comes in.

The inference from the data to the conclusion has several steps, and laying them out is what makes the rebuttal possible to follow.

The observation is that accuracy is high below a complexity threshold and near zero above it, with no gradual middle.

The first inference is that a smoothly degrading system would not produce a cliff, so something qualitative changes at the boundary.

The second is that the two regimes correspond to problems within and outside the coverage of training data.

The third is that a system pattern matching within its coverage would show exactly this shape, and a system executing a general procedure would not.

The conclusion is that the models are not executing procedures.

Each step is reasonable, and the whole chain rests on one assumption stated nowhere in it: that a score of zero means the model failed to reason. Scores come from a scoring procedure, and a zero means the scoring procedure returned zero. Those are the same thing only if the scoring procedure measures what it claims.

That assumption is where the rebuttal attacks, and it does not need to dispute a single number.

The paper was published by a company with a visible commercial position in the debate, at a moment when reasoning models were the industry's main claim, and the title contained the word illusion.

It was read accordingly. One audience treated it as confirmation that the reasoning capabilities being marketed were overstated. Another treated it as a competitor's attempt to diminish a technology it was behind on. Both readings circulated widely and neither examined the methodology.

That reception is itself worth noticing, because it is the normal fate of a result that arrives with a strong headline into a contested question. The evidence gets used rather than evaluated.

The useful posture in that situation is neither acceptance nor dismissal. It is to ask what the experiment actually measured, which requires reading the setup rather than the abstract.

Within weeks, several people did that, and one wrote it up. The response was titled The Illusion of the Illusion of Thinking, and it argues that the findings primarily reflect experimental design limitations rather than fundamental reasoning failures.

What it found is the subject of the next lesson, and none of it required disputing the data.

Before reading the rebuttal, it is worth fixing the question the experiment was supposed to answer, because the criticism turns on a gap between that question and what was measured.

The intended question: can these models execute a procedure reliably as the problem grows, or does their apparent competence come from familiarity with problems like the ones they were trained on?

The measurement: when asked to solve a puzzle instance, does the model's output, evaluated by a simulator, constitute a correct solution?

Those are close and not identical, and the gap between them is where every measurement problem lives. A model might reason correctly and produce output the evaluator scores as wrong. A model might be prevented from finishing for reasons unrelated to reasoning. The problem itself might have no solution.

In each case the measurement returns zero, and the intended question has not been answered.

So the productive question when reading any evaluation result is not whether the number is right. It is: what else, besides the thing being claimed, would produce this number? That question is the whole of the next lesson.