Why Asking Nicely Does Not Guarantee JSON

Ask a capable model for JSON, give it a schema, add an example, and it will comply most of the time. Most of the time is the problem.

Suppose 5 percent of responses fail to parse. On a demo that is invisible. In a pipeline where the output feeds a function call, it means one request in twenty enters an error path.

The usual patch is a retry loop, and the arithmetic looks reassuring until you read the right statistic. Retrying up to three times drops the total failure rate to about one in eight thousand. Expected calls per success rise only to 1.053, so average cost barely moves.

But latency percentiles move enormously. Ninety-five percent of requests are served in one call, which means the 95th percentile request takes two calls. The p95 latency of your endpoint just doubled, and it doubled because of a formatting problem rather than anything about the task.

And retries are not free of side effects when the call has already invoked a tool.

To get a guarantee rather than a high probability, you have to intervene somewhere the model cannot overrule you. There is exactly one such place, and it is inside the decoding loop.

At each generation step the model produces a vector of logits, one real number per vocabulary entry, typically on the order of 128,000 of them. A softmax converts those to probabilities, and a sampler picks one token.

The logit vector is a plain array of floating point numbers sitting in memory between two steps you control. Nothing prevents you from editing it.

Set the logit of a token to negative infinity and its probability after softmax is exactly zero. Not small. Zero. No sampling strategy, no temperature, no random seed can select it.

That is the whole mechanism. Constrained decoding is the practice of computing, at every step, which tokens would keep the output valid, and setting every other logit to negative infinity before sampling.

The model is not persuaded to follow the format. It is made unable to leave it.

It is worth being precise about what masking does to the probabilities, because it is not neutral and the consequences appear later.

Suppose at some step the model's top choices are the token for the word Sure at probability 0.4, an opening brace at 0.3, and a newline at 0.1. The grammar permits only the opening brace.

Masking sets everything except the brace to negative infinity. After softmax, the brace has probability 1.0.

Two things happened. The output is now guaranteed valid, which is what you wanted. And the probability mass that the model assigned to other continuations, 70 percent of it here, was redistributed onto the survivors rather than discarded.

That redistribution is renormalisation, and it is why constrained decoding is not free. You are sampling from a conditional distribution: the model's distribution given that the output is valid. When the model strongly wanted something the grammar forbids, you are forcing it down a path it considered unlikely.

When the grammar agrees with the model, masking changes almost nothing. When they disagree, it changes a great deal, and that gap is where quality problems come from.

Structured output arrives under several names that behave differently, and mixing them up produces arguments in which both sides are right about different things.

Format restriction instructions. You describe the schema in the prompt and ask for compliance. No machinery, works with any endpoint, and offers no guarantee. This is what most systems start with.

Constrained decoding. The logit mask described above. Guarantees syntactic validity by construction, requires access to the decoding loop or a provider that exposes it, and changes the sampling distribution.

Generate-then-convert. Let the model answer in natural language, then use a second, cheaper call or a parser to convert it into the target format. Costs an extra step and preserves the model's unconstrained reasoning.

The distinction matters because published comparisons of structured output are often comparing these, not comparing schemas. A finding that JSON mode hurts reasoning is a finding about constrained decoding relative to generate-then-convert, and it does not mean structured output is bad.

Most production systems end up combining them, and the last lesson covers how.

Constrained decoding is a small insertion into a loop that already exists, and seeing where it sits removes most of the mystery.

The unconstrained loop runs: the model produces logits for the next token, a sampler selects one, the token is appended to the sequence, and the loop repeats until a stop condition.

The constrained loop adds two boxes. A validator holds the state of the format being generated, and after each token is emitted it advances that state. Before sampling, it produces a mask over the vocabulary saying which tokens are permissible from the current state. That mask is applied to the logits, and sampling proceeds normally on what remains.

Two properties follow directly. The sampler is untouched, so temperature, top-p and beam search all still work, over a restricted support. And validity is maintained inductively: if the sequence is a valid prefix and every permitted token keeps it a valid prefix, the sequence is always a valid prefix.

The engineering question, and the whole content of the next lesson, is how the mask gets computed fast enough to run 128,000 times per token without dominating the step.

The inductive argument in the previous step deserves attention, because it explains a constraint that is easy to get wrong when implementing this yourself.

The question the validator must answer is not whether the string so far is valid. It is whether the string so far can still be extended into something valid.

Consider generating JSON. After the characters open-brace quote n a m, the text is not valid JSON and never will be on its own. But it is a valid prefix, since it can be completed to a full object. The validator must permit it.

Conversely, after open-brace quote name quote colon, a closing brace would produce something that can never become valid, so it must be forbidden even though braces are obviously part of JSON.

So the machinery has to reason about completability, not correctness. That is what makes it more than a parser, and it is why the natural formalisms here are automata, which have a notion of a state from which an accepting state is still reachable.

An implementation that merely runs a parser after each token and asks is this valid yet will reject every legitimate prefix and generate nothing at all.

Guaranteed parseability is the headline, and several other consequences follow that are easy to miss.

Closed vocabularies become enforceable. Constraining a field to one of four category names means the model cannot return a fifth, so the downstream switch statement has no default branch to worry about.

Output length becomes bounded. A grammar that permits at most three array elements makes an unbounded list impossible, which removes a class of runaway generation.

Token cost falls slightly. The model does not spend tokens on preamble, apology, or a markdown code fence, because those tokens are not reachable from the start state.

And stopping becomes deterministic. When the automaton reaches its accepting state with no valid continuation, generation ends because the format is complete, not because a stop sequence happened to appear or a token limit was hit.

That last point matters more than it sounds. Truncation at a token limit mid-object is one of the more annoying failure modes of prompt-based structured output, and a grammar removes it by construction rather than by tuning the limit.

Masking requires access to the logits, which determines where the technique is available.

Self-hosted inference engines expose it directly. The major serving stacks ship structured output backends, and the mechanism runs inside the same process as the model, so the mask can be applied with no round trip.

Hosted APIs expose a restricted version. Several providers offer a structured output or JSON mode that accepts a schema and enforces it server-side using exactly this machinery. You get the guarantee without the control: typically a JSON Schema subset rather than arbitrary grammars.

A weaker relative is a logit bias parameter, which shifts named tokens by a fixed amount. That is useful for nudging a small closed set and cannot express a grammar, because the required mask changes at every step and a static bias does not.

The practical consequence is that if your format is expressible as JSON Schema, a hosted API will usually do it. If you need a domain-specific grammar, a SQL dialect, a constrained programming language, a bespoke wire format, you need control of the decoding loop.

Before the mechanism, it is worth seeing why this is a research area rather than a weekend project. The naive implementation is unusably slow.

At each generation step you need to know which of the vocabulary's tokens are permissible. The obvious method is to try each one: append it to the current output, check whether the result is still a valid prefix, keep it if so.

With a vocabulary of 128,000 and a 500-token response, that is 64 million validity checks per response, each involving string manipulation and a parse. This runs on the CPU while an expensive accelerator waits.

That overhead would dwarf the model forward pass and make constrained decoding strictly worse than retry loops.

The entire content of the next two lessons is how that cost gets removed: precomputing the answer per automaton state rather than recomputing per step, splitting tokens into those whose validity depends on context and those whose does not, and overlapping the remaining work with the model's own computation. Modern implementations reduce the per-step cost to near zero.

One boundary should be set now, because it is the source of most disappointment with this technique.

Constrained decoding guarantees that the output matches the grammar. It guarantees nothing about whether the output is correct.

A schema requiring an object with a name string and an age integer will always produce an object with a name string and an age integer. The name may be wrong. The age may be invented. The model may have had no information and filled the fields anyway, because the grammar demanded a value and provided no way to decline.

That last case is worth dwelling on. Constraints do not only fail to prevent fabrication; they can actively induce it. A grammar that requires a field the model cannot fill leaves it no legal way out, and it will emit something.

So the technique moves failures from a place where they are loud, a parse error, to a place where they are quiet, a well-formed object containing a plausible fiction. That is usually the right trade, and it is a trade rather than a free win.