Structured Outputs and Constrained Generation

Passing one layer does not mean passing the next. The gap opens at the very first one, and the JSON standard documents it.

RFC 8259, the specification of JSON, published in December 2017, says the names in an object SHOULD be unique. Then it says what follows when they are not: “An object whose names are all unique is interoperable in the sense that all software implementations receiving that object will agree on the name-value mappings. When the names within an object are not unique, the behavior of software that receives such an object is unpredictable.”

Unpredictable how? The same section lists the divergence in the field. Many implementations report only the last name/value pair. Others report an error, or fail to parse the object at all. Some report all of the duplicates, the repeats included. An object that carries the same name twice is fully valid JSON, accepted by the parser, and worth three different meanings to three different receivers.

That is the syntactic layer failing on its own terms. Nobody has yet asked about schema conformance, evidence or authorization.

A schema-aware generator can restrict token choices so the output stays compatible with a grammar or a type definition. That reduces formatting errors and simplifies parsing downstream. It cannot tell whether a value is present in the source, current, or authorized. The model can produce a perfectly shaped falsehood.

That sentence has been measured, by Geng and colleagues in 2023. Constituency parsing, Penn Treebank, eight shots: an input-dependent grammar raised LLaMA-33B's parse-tree validity from 64.2% to 100.0%. Every single output was a well-formed tree. Bracketing F1 rose only from 42.9 to 54.6, against the supervised parsers the paper lists at 92.1–95.7 (Kitaev and Klein 95.6, Zhang et al. 95.7). Total structural success. Roughly half the linguistic content wrong.

Their own summary says as much: “In conclusion, GCD substantially improves the performance of LLMs on constituency parsing, but performance still falls short of the F1-scores achieved by supervised methods (95% and above).”

The vendors say the same thing about their own products. OpenAI's Structured Outputs guide says of its own coverage that “While Structured Outputs supports much of JSON Schema, some features are unavailable either for performance or technical reasons”, and it warns, where it handles edge cases: “The model will always try to adhere to the provided schema, which can result in hallucinations if the input is completely unrelated to the schema.” The constraint is a pressure to fill the fields. It is not a source of the values that fill them.

Fields should capture what the workflow needs to know, including absence and evidence status. So should the order they are declared in, which is a semantic decision rather than a matter of taste.

Format restriction has been measured on GSM8K. Adding an explicit schema to a JSON prompt cut claude-3-haiku from 86.99% to 23.44% accuracy, and gpt-3.5-turbo from 74.70% to 49.25%. For gpt-4o-mini-2024-07-18, natural language scored 94.57%, against 86.95% in JSON-mode and 91.71% under JSON-Schema.

On the Last Letter Concatenation task, Tam and colleagues traced the shortfall to key order. 100% of GPT-3.5-Turbo JSON-mode responses placed the "answer" key before the "reason" key. Zero-shot chain-of-thought had quietly become zero-shot direct answering. Their finding: “However in reasoning related task, JSON-mode failed to adhere to the order of reasoning first followed by answer causing a large drop in final performance.”

OpenAI's guide confirms the mechanism is structural: “outputs will be produced in the same order as the ordering of keys in the schema”. Whoever writes the schema decides where the reasoning happens. A sixty-point accuracy drop can be entirely a consequence of which field was listed first.

Treat model output as a candidate object. Treat the generator as an engine with declared limits, not as a guarantee.

JSONSchemaBench, released in January 2025, put numbers on those limits: 10,000 real-world JSON schemas plus the official JSON Schema Test Suite, run against six constrained-decoding engines. On the GitHub-Easy split, OpenAI's structured-output API declared support for only 30% of the schemas and Gemini for 8%, against 90% for Guidance. On GitHub-Medium OpenAI fell to 13%. Among the schemas they did accept, compliance was 0.97 and 0.88. Geng and colleagues read that as a deliberate trade: “While closed-source implementations have low empirical coverage, they have very high compliance rates, indicating that their providers have taken a more conservative strategy, implementing only a subset of JSON Schema features that they can reliably support.”

So the "Generate under constraints" gate may simply decline most of the schemas you actually hold. Acceptance is not conformance either. On the test suite, XGrammar was under-constrained — it accepted instances it should have rejected — in 38 feature categories, against 1 for Guidance.

"Parse and validate" is not redundant with the generator. It is the only step that has read your schema, rather than the subset of it the engine implements.

Choose a source-dependent task and make missing evidence explicit. Design the field order deliberately: reasoning before conclusion. Then test against the two failure modes this lesson measured. A document that parses perfectly and means different things to different parsers. An output that is 100.0% well-formed while its content scores 54.6.

Schemas make model output easier to parse, test, log, and route. Designed carefully, they also show which fields may be absent and what evidence stands behind each one. What they do not do is settle anything about the world. The constraint that produced 100.0% valid parse trees produced 54.6 bracketing F1 in the same run, and OpenAI's own guide warns that forcing adherence to a schema can itself induce fabrication.

The next lesson uses structured arguments for tool calls. Once generated data can cause an external action, authorization and idempotency become central.