Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation

AI summary

Overview

Research area: Natural Language Processing — structured data serialization for Large Language Models, comparing output formats and decoding strategies.

Technical level: Intermediate. The paper assumes familiarity with JSON, Pydantic-style validation, constrained/guided decoding (e.g., xgrammar), in-context learning, and token accounting, but explains each track clearly enough for a practitioner.

Scope: A benchmark of three generation tracks (plain JSON, JSON with constrained-decoding structured output, and TOON with one-shot in-context learning) across 21 models, 4 test cases, 10 runs per model, measuring one-shot accuracy, final accuracy after repair cycles, and token budget.

What This Paper Is About

Token-Oriented Object Notation (TOON) was introduced as a serialization format intended to replace JSON when passing structured data to LLMs, using significantly fewer tokens. Its comprehension accuracy in LLMs has been reported, but it had not been tested on the generation side: no benchmark compared TOON generation against JSON generation. This paper builds such a benchmark, including a plain JSON track, a structured-output (constrained decoding) JSON track, and a TOON track using one-shot in-context learning, to see whether TOON's shorter syntax actually pays off once the instructional prompt overhead and repair loops are counted.

Key Contributions

1. A reproducible benchmark suite for structured LLM generation that compares three output tracks — JSON (J), JSON with structured output / constrained decoding (JSO), and TOON (T) — with source code, dataset generators, prompt templates, and raw evaluation logs released at https://github.com/vetertann/TOON-generation-benchmark.
2. A four-case test set spanning structural complexity: users (simple tabular), order (nested structure with array), company (department and employee hierarchy), and invoice (items and totals), with gold standards generated from Pydantic models and serialized to .gold.json and .gold.toon via the @toon-format/cli.
3. An evaluation protocol covering prompting, TOON CLI decoding, Pydantic validation, canonicalization, comparison against gold JSON, and up to three repair cycles, with one-shot accuracy, final accuracy, and per-format token totals recorded at case-level granularity.
4. A per-model and per-case result set over 21 models served through the Nebius API, plus an aligned/non-aligned domain analysis and token-efficiency metric (final accuracy per 1000 tokens) that together frame a scaling hypothesis for TOON.

Main Findings

• Plain JSON wins on accuracy overall: Plain JSON generation showed the best one-shot and final accuracy, even compared with constrained decoding structured output. For example, NousResearch/Hermes-4-405B reached 92.5% one-shot and 92.5% final in JSON (3252 tokens) versus 35.0% one-shot and 100% final in JSO (4759 tokens).
• Constrained decoding acts as a guardrail for weaker models: Qwen/Qwen2.5-Coder-7B-fast scored 0.0% one-shot and 0.0% final in plain JSON while consuming 37,705 tokens; under JSO it reached 75.0% one-shot and 75.0% final with 4,440 tokens.
• Constrained decoding can hurt strong models: For larger reasoning models, grammar constraints degraded initial performance. Hermes-4-405B dropped from 92.5% one-shot in JSON to 35.0% in JSO, although it recovered to 100% final after repair cycles with lower total token consumption than plain JSON in that track (4759 vs 3252 — i.e., more in JSO, per the reported figures).
• JSO generally has the lowest token usage: The paper reports this as the only significant advantage of constrained decoding, traded for slightly decreased accuracy overall and significant degradation for some models.
• Constrained decoding beats TOON on tokens for simple structures: In the users case, JSO used 556 tokens versus TOON's 840 tokens, and in aggregate the paper notes the "lowest token usage" of constrained decoding outperformed even TOON for simple structures. This hints that enforcing TOON through frameworks such as xgrammar may not yield the desired results.
• TOON is competitive only on aligned data: On the TOON-aligned group (users, order, invoice), TOON reached 54.9% one-shot and 73.8% final accuracy, versus JSON at 88.9%/88.9% and JSO at 86.4%/92.8%. In the users case TOON achieved 90.5% one-shot accuracy while using 22% fewer tokens than plain JSON, and in order it returned 78.6% one-shot accuracy (JSO: 83.3%).
• TOON collapses on deep hierarchies: In the company case TOON's one-shot accuracy was 0.0%. Repair cycles brought final accuracy to 48.6%, the highest of the three tracks (JSON 43.8%, JSO 48.1%), with the lowest total token consumption (2567 vs 2592 for JSO and 3575 for JSON) — attributed to the syntactic overhead of deeply nested JSON paying off immediately.
• TOON carries a "prompt tax": Because TOON is a novel format, it requires a heavy instructional prompt. For Qwen/Qwen3-235B-A22B-Inst, this overhead made TOON consume 4715 tokens versus 2772 for plain JSON, despite the lighter output syntax.
• Repair loops are expensive for TOON: In the invoice case the TOON track failed initially (0.0% one-shot accuracy), and feeding the error history back into a large system prompt doubled token usage compared to JSON (3626 vs 1723). TOON generation is efficient when it works and expensive when it fails.
• Case-level accuracy and cost profile: Across all models and runs, users scored 94.8%/94.8% for J (1078 tokens), 92.9%/100% for JSO (556), 90.5%/90.5% for T (840); order scored 81.9%/81.9% for J (1746), 78.6%/83.3% for JSO (1255), 74.3%/78.6% for T (1585); company scored 18.6%/43.8% for J (3575), 21.9%/48.1% for JSO (2592), 0.0%/48.6% for T (2567); invoice scored 90.0%/90.0% for J (1723), 87.6%/95.2% for JSO (1349), 0.0%/52.4% for T (3626).
• Scaling hypothesis: TOON's true efficiency potential likely follows a non-linear curve, shining only beyond a point where cumulative syntax savings on large datasets amortize the initial prompt overhead — though this may introduce new risks of indentation drift over long context windows.

Methodology in Plain English

The author created four structured-data scenarios of increasing structural difficulty (users, order, company, invoice) and generated gold-standard answers by defining Pydantic models, then serializing them to canonical JSON and to TOON using the official TOON CLI.

Each scenario was then generated three ways. In the JSON track, a model was asked for JSON and the result was validated with Pydantic. In the JSON-SO track, the same request was made with response_format="json_object", meaning the inference engine constrains decoding — it compiles a grammar (such as JSON Schema or regex) into a state machine like xgrammar and masks logits so only grammatically legal next tokens can be produced. In the TOON track, the model was given a TOON instruction prompt with rules (2-space indentation, fieldName: value scalars, arrayName[N]: arrays of objects, arrayName[N]{field1, field2}: tabular arrays, [N] must equal the actual item count, output only a ```toon code block) plus one universal reference example — deliberately not per-case examples, to keep the comparison fair with JSON generation that needs no schema example.

After generation, TOON output was decoded back to JSON via the CLI (CLI errors triggered a repair cycle), all formats were validated with Pydantic, canonicalized, and compared against the gold JSON. If the data did not match, a repair cycle ran for up to three attempts, re-injecting the previous output and error text into the prompt.

Twenty-one models served through the Nebius API were each run 10 times per test case at temperature 0 to check output stability and consistency across API calls (not sampling variance); other parameters were left at provider defaults. One-shot accuracy is the fraction of the four cases solved on the first attempt; final accuracy is the fraction solved after up to three repair cycles. Token usage was recorded per case and format, and per-format totals are the absolute sum of all prompt and completion tokens across the four cases in a run. The paper states that no separate one-shot token budget is computed: when a case succeeds first try, its one-shot and final token costs coincide. For the efficiency analysis, the author averages final accuracy per case within a scenario group and per-case token cost, then defines efficiency as final accuracy per 1000 tokens, reporting separate plots for aligned cases (users + order) and non-aligned cases (invoice + company).

Why This Matters

Impact on research. The paper reframes the TOON-versus-JSON debate from syntax size to end-to-end generation economics. It shows that measuring only token length of a serialization format is insufficient: prompt overhead, repair-loop costs, and structural topology determine whether a compact format actually saves money. It also supplies a concrete counterexample to the assumption that constrained decoding is always a strict win, complementing prior observations that grammar constraints can interfere with a model's natural probability distribution.

Real-world applications:

• High-volume, cost-sensitive ETL pipelines and batch processing on tabular or transactional data (orders, invoices), where TOON may reduce token spend if the schema stays inside its "aligned" design space.
• SQL query result dumps and log streams or audit trails with uniform tabular rows, which the paper explicitly lists as TOON-aligned workloads.
• Simple structured extraction tasks, where JSON-SO with constrained decoding was the absolute token efficiency winner (556 tokens versus TOON's 840 in the users case).
• Deeply nested configuration, DOM-like structures, or complex state trees, where the paper states TOON is not production-ready and recommends conventional serialization or JSO if token limits are tight.

Industry relevance. Teams choosing an output format for LLM pipelines now have per-model and per-case evidence that plain JSON is the accuracy default, JSO is a token-saving guardrail with model-dependent accuracy risk, and TOON is a domain-specific optimization whose fixed prompt cost must be amortized by output size. The recommendation to benchmark significantly larger dataset sizes before adopting TOON is directly actionable for cost-driven deployments.

Future Directions

• Run large-scale tests on much larger datasets to find the point at which TOON's cumulative syntax savings offset its prompt overhead, as the paper's scaling hypothesis predicts a non-linear efficiency curve.
• Investigate indentation drift over long context windows, since TOON's whitespace reliance may make it more vulnerable to errors in long generations.
• Test whether customizing the one-shot TOON example per case to better match the schema reduces structural mistakes, and whether data-structure size scaling produces non-linear error growth.
• Extend the pipeline to a TOON-aware constrained decoding setup — providing the TOON vocabulary to an xgrammar-like state machine — to test inference enforcement, which this benchmark deliberately left as a starting point for another experiment.

Target Audience

Practitioners and researchers working on structured output from LLMs, prompt and serialization format design, and constrained decoding; engineers building data-extraction, ETL, or transactional-document pipelines who need to pick between JSON, structured output, and alternative formats; and evaluation researchers interested in how token-budget accounting, prompt overhead, and repair loops should be measured alongside accuracy. The paper is also useful for anyone considering TOON in production, because it clearly delineates where the format is aligned with the data and where it is not.

Authors’ abstract

Recently presented Token-Oriented Object Notation (TOON) aims to replace JSON as a serialization format for passing structured data to LLMs with significantly reduced token usage. While showing solid accuracy in LLM comprehension, there is a lack of tests against JSON generation. Though never present in training data, TOON syntax is simple enough to suggest one-shot in-context learning could support accurate generation. The inevitable prompt overhead can be an acceptable trade-off for shorter completions. To test this, we conducted a benchmark creating several test cases with regard to structural complexity, a validation pipeline, and comparing plain JSON generation vs structured output (via constrained decoding) JSON generation vs TOON one-shot in-context learning generation. JSON structured output was included to establish a minimum token budget baseline and to set a starting point for future experiments testing TOON constrained decoding inference enforcement. Key findings: TOON shows promising accuracy/token consumption ratio for in-domain generation tasks, though this advantage is often reduced by the "prompt tax" of instructional overhead in shorter contexts. Plain JSON generation shows the best one-shot and final accuracy, even compared with constrained decoding structured output, where the only significant advantage is the lowest token usage as a trade-off for slightly decreased accuracy overall and significant degradation for some models. Notably, for simple structures, this "lowest token usage" of constrained decoding outperformed even TOON, hinting that TOON enforcing via frameworks such as xgrammar may not yield the desired results. Furthermore, the results suggest a scaling hypothesis: TOON's true efficiency potential likely follows a non-linear curve, shining only beyond a specific point where cumulative syntax savings amortize the initial prompt overhead.