What This Teaches About Measuring Anything

The first rule from this exchange applies to every evaluation, and it is about which direction a result points.

A pass is fairly informative. A model produced a correct answer, and although it may have got there by memorisation, the output was right.

A failure is barely informative on its own, because everything that is not a pass lands in the same bucket. The model reasoned wrongly. It reasoned correctly and was cut off. It refused. It answered in a format the parser rejected. It timed out. It correctly identified the problem as unsolvable. The task was impossible.

All of those produce a zero. The evaluation records one number for seven situations.

That asymmetry decides how much scrutiny a result needs. An evaluation whose headline is a high pass rate can survive a crude failure bucket. An evaluation whose headline is a collapse to zero cannot, because the entire claim rests on which of those seven situations occurred.

So the rule is simple. If your conclusion is about failures, you need a failure taxonomy. If you do not have one, you do not have the result.

A failure taxonomy is not elaborate. It is a handful of categories recorded alongside each zero, and the categories are mostly the same across domains.

Refused or declined. The model said it would not or could not, which is a different thing from getting it wrong.

Truncated. The response hit a length limit. Detectable from the finish reason and from whether the output ends mid-structure.

Malformed. The answer may be correct and the parser could not read it. Detectable by having a human read a sample of these.

Wrong. The model produced a complete, parseable answer that was incorrect. This is the only category that supports a claim about capability.

Invalid task. The instance had no correct answer, or the correct answer was not what the key said.

Recording these costs almost nothing at evaluation time and is impossible to reconstruct afterwards from stored scores. And the distribution across categories is often more informative than the pass rate. A model failing entirely through truncation and a model failing entirely through wrong answers have the same score and completely different problems.

The unsolvable-instance failure is the most embarrassing and the easiest to prevent, and it happens because generated evaluations are generated by programs.

When instances are written by hand, someone solved each one while writing it, so unsolvable items are rare. When instances are generated by sampling parameters, nothing guarantees each combination admits a solution, and the more constraints a puzzle has the more likely some combinations are contradictory.

The prevention is a solver. Before any model sees the evaluation, run every instance through a reference solver and discard those with no solution. If no solver exists, the instances need human verification, and if neither is feasible, the evaluation cannot support conclusions about failures.

The same check catches a related problem: instances where the answer key is wrong. A reference solver that disagrees with the key on an instance has found a bug in one of them, and either way you want to know before scoring.

This generalises past puzzles. Any evaluation with generated items, templated questions, or automatically extracted answer keys needs the same pass, and most do not get one.

The deepest lesson is that how you require an answer to be expressed is part of what you are measuring, and it is rarely treated that way.

The original experiment asked for an enumerated move list. That choice made the answer length grow exponentially with problem size while the reasoning difficulty grew far more slowly. Past a point, the format demanded more output capacity than the model had, and the measurement stopped being about reasoning.

The rebuttal asked for a generating function. Same reasoning, constant-size answer, and the ceiling disappeared.

Neither format is correct in the abstract. They measure different things. Enumeration measures reasoning plus the ability to produce long exact sequences. A generating function measures reasoning plus the ability to express a procedure compactly. Which you want depends on the claim.

The error is not choosing a format. It is choosing one without noticing that it introduced a variable, and then attributing the resulting pattern entirely to the thing you meant to study.

The check is to ask whether the output requirement scales with problem difficulty. If it grows faster than the reasoning does, you are measuring two things at once.

One detail in this story deserves separating out, because it is the cheapest safeguard available and it was skipped.

The models said what was happening. Their outputs explicitly acknowledged that the full move list would be impractical to write out. The explanation for the collapse was sitting in the transcripts, in plain language, and the scoring pipeline recorded a zero and moved on.

That is what an automated evaluation does. It applies a scoring function and produces a number, and every piece of information in the response that the scoring function does not extract is discarded.

The safeguard is to read a sample by hand. Not all of them, and not systematically, but take twenty failures and read what the model actually said. It takes an hour and it catches the class of problem no amount of statistical care will find, because the problem is not in the statistics.

This holds for evaluations generally. Aggregate metrics are how you measure; reading outputs is how you find out whether you are measuring the right thing. Skipping the second because you have the first is how a result like this happens.

Collecting the rules into an order gives a review procedure for any evaluation whose conclusion depends on things going wrong.

First, are all instances valid? Every item verified solvable, every answer key checked against a reference. If not, some portion of the failures is noise.

Second, is there a failure taxonomy? Refused, truncated, malformed, wrong, invalid task, recorded per item. If every failure is one bucket, the conclusion is underdetermined.

Third, does the output requirement scale with difficulty? If the answer gets longer faster than the problem gets harder, format is confounded with capability.

Fourth, has anyone read the failures? Twenty transcripts, by a human, looking for anything the scorer would not have captured.

Fifth, is there an alternative explanation, and does a cheap test separate it from yours?

Any evaluation passing all five supports a claim about failures. One failing the first two does not, regardless of how many items it ran or how careful the statistics are.

The last question is the one that turns criticism into progress, and it is the subject of the final step.

The most valuable thing in this exchange is not any of the objections. It is what the rebuttal did with them.

It would have been easy to list the problems and conclude the result was unreliable. That is the usual shape of methodological criticism, and it produces a stalemate: one side has a finding, the other has doubts, and nothing settles it.

Instead the rebuttal identified where the two explanations made different predictions, designed a cheap experiment on that difference, and ran it. The generating-function test cost very little and produced an answer.

That is the move worth taking from this. When two explanations fit the same data, arguing about the data is unproductive, because the data is consistent with both. The question is what else they disagree about.

And this generalises far outside AI. Any time a metric moved and there are two stories about why, the useful step is to find something the two stories predict differently and go and look. It is almost always cheaper than the argument it replaces.

The habit is worth more than the conclusion.

The last thing to take from this is about consuming results rather than producing them, because most people encounter this work through a headline.

A strong claim arriving into a contested question gets used before it gets checked. Both sides of this one had a reading ready before anyone examined the methodology, and both readings spread widely.

The protective habits are not sophisticated.

Read what was measured, not what was concluded. The gap between them is where the interesting problems live, and it is usually visible in the setup section.

When a result is surprising, look for the boring explanation first. Systems have limits, evaluators have bugs, and formats have constraints, and any of those is likelier than a deep finding about cognition.

Check whether a response exists. This paper had one within weeks, and it was substantive.

And separate the claim from the evidence. Reasoning models may well have serious limitations. This experiment did not establish it, and those two statements are entirely compatible.

That last distinction is the one worth carrying, because it is the one headlines are least able to make.