What Constraints Cannot Fix

Work on format restrictions, including the study published as Let Me Speak Freely, examined what happens to model performance when output format is constrained, and the result is more specific than the headline suggests.

Stricter format constraints degraded performance on reasoning-heavy tasks. Mathematics, symbolic reasoning and multi-step analysis suffered, with reported degradations in the region of ten to fifteen percent relative to free-form generation followed by conversion.

On classification-style tasks, the effect ran the other way. Slot filling and intent detection benefited, because the constraint removed the model's opportunity to hedge, explain, or answer in a form the parser did not expect.

Both halves matter, and quoting only the first is how this becomes an argument against structured output in general.

The honest summary is that constraining output helps when the difficulty is in producing a clean answer, and hurts when the difficulty is in working out what the answer is.

The rest of this lesson is about why, because the mechanism tells you how to keep the benefit without paying the cost.

The dominant mechanism is not subtle, and once seen it explains most of the observed damage.

A decoder-only model produces one token at a time, each conditioned on everything before it. It has no scratchpad and no ability to revise. The tokens it has already emitted are the only working memory it has.

That is what chain-of-thought exploits. Writing out intermediate steps is not a presentational choice; it is the computation. The model derives the answer by producing the derivation, and each intermediate token gives the next step something to condition on.

Now consider a schema whose first field is the answer. The grammar requires a value immediately. The model must emit the conclusion before it has generated any of the reasoning that would have produced it.

The constraint has not made the model worse at mathematics. It has removed the mechanism by which the model does mathematics, and demanded the result anyway.

Everything in the practical half of this lesson follows from that observation.

The fix follows directly, and it is the single highest-value adjustment available.

Give the schema a free-text field before the answer field, and let the model use it.

A schema whose properties are, in order, reasoning as an unconstrained string, then answer as a constrained value, preserves the derivation. The model generates its working in the reasoning field, where the grammar imposes almost nothing, then emits the answer conditioned on that working. You keep the guarantee on the field you will parse, and the model keeps its scratchpad.

This only works because field order is fixed. As the automaton lesson noted, constrained generation emits properties in schema order to keep the state count manageable. That implementation detail becomes a design tool: schema order is generation order, and generation order is thinking order.

Reversing the two fields produces a schema that validates identically and performs considerably worse, and nothing in the JSON Schema will indicate the difference.

The general principle: order fields so that anything the model needs in order to determine a value appears before that value.

A quieter mechanism operates alongside the ordering effect, and it explains degradation even in schemas whose ordering is correct.

Masking renormalises. When the model assigns high probability to a continuation the grammar forbids, that mass is redistributed onto tokens the model considered unlikely. Sampling then proceeds from a distribution the model never produced.

Do that at many steps in sequence and the generated text drifts into a region of token space the model rarely visited during training. Each individual forced choice is small; conditioned on a growing prefix of forced choices, subsequent predictions get worse.

The tokenization mismatch from the automaton lesson compounds this. The token sequence may be a non-canonical segmentation of perfectly ordinary text, so even where the characters look normal, the sequence the model is conditioning on has an unfamiliar shape.

The practical signal is that damage scales with how much the grammar disagrees with the model. A schema the model would likely have followed anyway costs almost nothing. A rigid, unusual format costs a great deal, and the cost is invisible in the output, which parses perfectly.

The most damaging schema pattern is also the most natural one to write, and it manufactures confident errors.

Suppose a field is constrained to one of four category labels. The intent is reasonable: enforce a closed vocabulary so downstream code has no unexpected values.

Now give the model an input belonging to none of the four. It has no legal way to say so. Every token except those four labels is masked, so it will emit one of them, and it will emit the closest, with no signal that it was forced.

The result is worse than a parse error, because a parse error is visible and this is not. Downstream systems receive a valid label and treat it as a classification the model made.

The fix is to widen the enum with an explicit escape. Add an other or unknown value, and where it matters a confidence field ordered after the label so it can reflect the choice just made.

The general rule is that any constraint must include a legal way to express the true state of affairs, including uncertainty and inapplicability. Whatever the grammar omits cannot be said.

The contrast between a schema that constrains well and one that constrains badly is entirely about ordering and escape hatches. Both validate identically.

The damaging version asks for the answer first, from a closed enum, with a required numeric field. The model must commit before reasoning, has no way to signal that no category applies, and must produce a number whether or not it has grounds for one.

The better version puts a free-text reasoning field first, so the derivation exists in context before anything is committed. The category then follows, including an explicit unknown option. The numeric field is nullable rather than required. A confidence field comes last, so it can reflect what was actually produced.

The difference in downstream quality can be large, and a JSON Schema validator cannot see it at all. Both documents are correct.

The habit worth forming: read a schema in generation order and ask, at each field, whether the model has everything it needs to produce this value, and whether it has a legal way to say it does not know.

It is worth restating the boundary plainly, because constrained decoding is routinely deployed as a hallucination control and it is not one.

A grammar constrains the shape of the output. It has no access to the world. A schema requiring an ISO date will produce a well-formed date; whether that date is the one in the document is untouched. A schema requiring an integer identifier will produce an integer; whether that customer exists is untouched.

What constrained decoding removes is one specific failure class: output your parser cannot read. That class is worth removing. It is not the class most people are worried about.

The distinction matters for how you evaluate. A test suite that checks outputs parse and conform to the schema will pass one hundred percent of the time by construction, which makes it a test of the grammar engine rather than of the system.

Correctness needs its own evaluation, against ground truth, and that evaluation should compare constrained against unconstrained output on the same inputs, since that comparison is the only way to see what the constraint cost you.

Where reasoning genuinely matters and the ordering trick is not enough, the alternative is to separate the two jobs entirely.

Call one: ask the question with no format constraint at all. The model reasons freely, in whatever shape suits the problem, at full capability.

Call two: give a small, cheap model the free-form answer and a schema, and ask it to extract. This call is pure transcription, exactly the classification-shaped task where constraints were found to help rather than hurt, and a small model does it reliably.

The costs are real: an extra round trip, extra tokens, and a second place for information to be lost in transcription.

The benefit is that the expensive reasoning happens unconstrained, and the constraint applies only to a task that constraints suit.

The decision rule is straightforward. If constrained and unconstrained accuracy are close on your evaluation, take the single constrained call and its lower latency. If the gap is material, the second call is cheaper than the errors. Measure rather than assume, because the gap is task-dependent and reverses between task types.

Since the effect is task-dependent and can go either way, it has to be measured, and the measurement is simple enough that there is no excuse for skipping it.

Take a set of inputs with known correct answers. Run three configurations on all of them: unconstrained with a prompt asking for the format, constrained decoding with your schema, and generate-then-convert.

Score all three on task accuracy, not on parse rate. Parse rate is one hundred percent for the constrained arm by construction and zero information.

Then separate the failures. A parse failure in the unconstrained arm is a different cost from a wrong answer, since one is detectable and retryable and the other is not.

What this reveals is usually one of three pictures. Accuracy is unchanged and constrained decoding is free, which is the common case for extraction and classification. Accuracy drops and reordering the schema recovers it, which is the ordering effect. Or accuracy drops and reordering does not recover it, which is the point at which the two-call structure earns its extra round trip.

The measurement costs an afternoon and settles an argument that otherwise runs indefinitely.

Reducing the path to what survives contact with production.

Constrain the output you parse, not the thinking that produces it. A free-text field before the answer costs tokens and buys back most of the reasoning gap.

Schema order is generation order. Any value the model needs in order to decide another value must come first.

Every constraint needs an escape hatch. Enums need an unknown member, required fields should be nullable where the information may be absent, and anything the grammar omits cannot be expressed no matter how true it is.

Use the weakest mechanism that works. A closed set of five labels does not need a grammar engine.

Constraints are a parsing guarantee, not a correctness guarantee, and evaluating parse rate measures the grammar engine rather than the system.

And measure the constrained-versus-unconstrained gap on your own task, because the published result cuts both ways: reasoning-heavy work suffers, classification-shaped work improves, and which one you have is not something to assume.