The Rebuttal: Three Ways to Score Zero Without Failing

The response, titled The Illusion of the Illusion of Thinking, takes an unusual position. It accepts the measurements and argues they do not support the conclusion.

The claim is that the findings primarily reflect experimental design limitations rather than fundamental reasoning failures. Not that the models are better than reported, and not that the analysis was sloppy, but that the specific setup produced zeros for reasons unrelated to whether the model could reason.

This is the strongest form of methodological criticism, and also the hardest to dismiss. There is no dispute about what happened, so there is nothing to re-run to settle it. The argument is entirely about what a zero in this setup means.

Three distinct problems are identified, and they are worth separating because they are different kinds of error and each teaches something different about designing evaluations.

The first is a confound between the ability being tested and a mechanical constraint. The second is a scoring failure. The third is an unsolvable-instance problem, which is the most serious of the three.

The first objection is arithmetic, and once seen it is hard to unsee.

Solving Tower of Hanoi with n discs takes 2 to the power of n, minus one moves. Seven discs is 127 moves. Ten discs is 1,023. Fifteen discs is 32,767.

The evaluation required models to output the full move list. So at fifteen discs, a correct answer means printing over thirty-two thousand moves, each needing several tokens.

Every model has a maximum output length. Past a certain problem size, a complete correct answer does not fit in it. The model cannot produce a passing response regardless of how well it reasons, because the response is longer than it is permitted to be.

That alone would explain a cliff. Accuracy holds while solutions fit, and drops to zero at the size where they stop fitting, which is exactly the observed shape.

The rebuttal notes something that makes this more than a hypothesis. Models explicitly acknowledged the constraint in their outputs, saying in effect that the pattern continues but listing every move would be impractical.

The transcripts contained the explanation, and the scoring counted them as failures.

The output-limit objection does something a good alternative explanation should do: it accounts for a second observation the original paper offered as separate evidence.

Recall that reasoning traces got longer as problems got harder and then shorter again near the collapse threshold, which was presented as a system reducing effort just before failing, and read as evidence that it had given up rather than run out of capacity.

Under the output-limit account, that is the expected behaviour. A model that recognises the full enumeration will not fit does not attempt it. It reasons about the problem, concludes that printing the answer is impractical, and says so. That produces a shorter response than a model in the mid-range that is working through a long but feasible solution.

So the shrinking trace is not a system giving up on the reasoning. It is a system correctly assessing the output requirement and declining to attempt an impossible transcription.

An alternative explanation that accounts for two independent observations with one mechanism is considerably stronger than one that requires a separate story for each. That is a general standard worth applying.

The second objection is about the scoring, and it is the one with the widest application beyond this dispute.

The automated evaluator checked whether the output constituted a valid complete solution. It did not distinguish between a model that reasoned incorrectly and a model that reasoned correctly and was constrained from expressing the answer.

Both produce a response that is not a valid complete solution. Both score zero. The evaluator collapses two categorically different outcomes into one number, and the number is then interpreted as evidence about the first.

This is a general failure and not a slip specific to this study. Almost every automated evaluation has a single failure bucket into which everything that is not a pass falls: wrong answers, refusals, truncations, format errors, timeouts, and correct answers the parser could not read.

When the pass rate is high, that lumping is harmless. When the interesting finding is a drop to zero, it is fatal, because the entire result now depends on which kind of failure occurred, and the instrument did not record it.

The rule that follows is that a failure taxonomy is not optional in any evaluation whose conclusion rests on failures.

The third objection is the most serious, because it is not an interpretation dispute at all.

One of the puzzle families was River Crossing, the classic problem of ferrying items across a river subject to constraints about what may be left together. The rebuttal reports that some instances included in the evaluation were mathematically impossible: no solution exists, for any solver.

Models were scored as failures for not solving them.

There is no reading under which that measures reasoning ability. A model that correctly determined the instance was unsolvable and said so received the same zero as one that produced nonsense. Indeed, correctly recognising unsolvability is a more sophisticated response than attempting a solution, and it was scored identically to failure.

This is a straightforward error in instance generation, and the kind that is easy to make: a generator producing parameter combinations without checking each is satisfiable. It is caught by verifying that every instance in an evaluation has a solution before scoring anyone against it.

Whatever the other objections are worth, this one removes a portion of the evidence entirely.

Objections are cheap. What makes this rebuttal persuasive is that it proposes a test distinguishing the two explanations, and runs it.

The two hypotheses make different predictions. If models cannot reason about large instances, changing how the answer is expressed should not help, because the reasoning is what is missing. If models can reason and cannot transcribe the answer, then asking for the answer in a compact form should work.

So the rebuttal changed the request. Instead of enumerating every move, models were asked to produce a generating function: a program that outputs the solution when run.

That is a short answer for any problem size. The recursive Hanoi algorithm is a few lines whether it moves five discs or fifty, so the output limit stops binding entirely.

The preliminary result across multiple models was high accuracy on instances previously reported as complete failures.

That is the crux. The same models, the same instances, the same underlying reasoning requirement, and the only change was the format of the answer. If reasoning had been absent, changing the output format could not have recovered it.

The structure of the argument is worth holding separately from its subject, because it is a template for resolving disputes about evaluation results.

Both hypotheses predict the same original observation: a cliff in accuracy at a problem size. That is why the original data cannot distinguish them, and why arguing about the data is unproductive.

They diverge on a prediction the original experiment did not test. Under the reasoning-failure hypothesis, a compact output format changes nothing. Under the transcription-limit hypothesis, it restores performance.

The experiment is cheap, the prediction is sharp, and the outcome favours one side.

Notice what makes this good practice. The rebuttal did not accumulate objections and declare the result unsafe. It converted a disagreement about interpretation into a question with an empirical answer, and then answered it.

That is the move worth learning. When two explanations fit the same data, the productive step is not more argument about the data. It is finding the prediction on which they differ.

The rebuttal does not establish that reasoning models reason in whatever sense one might want, and it is worth being careful about what survives on each side.

What the rebuttal establishes is that this experiment does not show what it was taken to show. The cliff has an explanation not involving reasoning failure, that explanation accounts for a second observation as well, part of the evidence rested on unsolvable instances, and a discriminating test came out against the original interpretation.

What survives from the original work is more than the reception suggests. The design goal was right and the puzzle-based approach is a genuine improvement on contaminated benchmarks. The finding that models cannot reliably produce very long exact output sequences is real, and matters for anyone building on them, even if it is a different finding than the one claimed.

And the deeper question is untouched by both papers. Whether these systems reason, in any sense stronger than producing correct outputs on problems they were not trained on, remains open. This exchange was about one experiment.

That is the honest summary, and it is less satisfying than either headline.