Thinking Before Constraining: A Unified Decoding Framework for Large Language Models

AI summary

Overview

Research area: Natural Language Processing / structured text generation from large language models, specifically constrained decoding and the tension between free-form reasoning and guaranteed output formats.

Technical level: Intermediate. The paper assumes familiarity with autoregressive decoding, logit masking, and regular-expression- or grammar-guided generation, though the core idea is explainable without deep math.

Scope: The paper proposes and evaluates In-Writing, a single-call decoding framework that lets an LLM reason freely and only enforces a structured output schema after a trigger token is emitted, across classification and reasoning benchmarks and 18 open-source models.

What This Paper Is About

Large language models generate fluent text, but they do not guarantee adherence to a predefined output structure, which limits their use in schema-based extraction and structured question answering. Existing fixes sit at two extremes: hard constrained decoding guarantees format but can hurt reasoning by restricting the search space too early, while natural generation (or a two-stage NL-to-Format pipeline) preserves reasoning but offers no structural guarantees and adds cost. The paper's goal is to combine free-form reasoning with guaranteed structured output inside a single generative call, by decoupling when the model reasons from when the model formats.

Key Contributions

1. A framework, In-Writing, that unifies natural and structured generation within one inference call, with reported minimal token overhead. Reasoning proceeds unconstrained until a trigger token (e.g., <eos> or {) is generated, at which point regex- or grammar-based constrained decoding takes over.
2. Evidence that constrained decoding acts as a more effective parser and corrector of the model's own final answer than using a separate model to extract it, including cases where the constraint incidentally fixes small output errors.
3. An investigation of trigger-token strategies to mitigate premature triggering, a failure mode in which constrained decoding interrupts ongoing reasoning. In-Writing-Base uses trigger_token_ids = { <eos>, { }; In-Writing* uses <eos> as the sole trigger token.
4. An empirical demonstration that forcing a model to reason entirely inside a constrained grammar space is less effective than In-Writing, based on a comparison against CRANE.

Main Findings

• Overall accuracy gains: Evaluations across classification and reasoning datasets show accuracy gains of up to 27% over natural generation (attributed to extraction by parsing rather than by an LLM-based parser).
• Improvement over NL-to-Format: In an overlap analysis with identical prompts and identical reasoning traces, In-Writing* achieved up to an 11.8% improvement over NL-to-Format, correctly parsing many cases where NL-to-Format failed, while the reverse was much less common.
• Premature triggering is real and costly: In-Writing-Base results show premature triggering truncates reasoning, with over 30% degradation compared to In-Writing* on GSM8K, especially on mathematically intensive tasks; the effect on classification tasks was minimal.
• A single <eos> trigger mitigates the failure mode: Setting <eos> as the unique trigger token (In-Writing*) generally outperformed NL-to-Format and vanilla constrained decoding across model scales.
• 100% parse rate: In-Writing achieved a 100% parse rate with low inference overhead, because only the formatting component is constrained while the reasoning segment remains free and can be bounded by a maximum token limit.
• Token cost: In-Writing* adds 5–20 newly generated tokens; NL-to-Format adds 2–5 tokens, but relies on an additional LLM whose input includes the parser prompt plus the first model's output, giving substantially higher input token usage and computational overhead.
• Vanilla constrained decoding is unstable: It exhibited high variance across tasks and generally underperformed other methods, consistent with prior work; JSON-style constraints sometimes improved scores but the gain was largely attributable to increased token generation.
• NL-to-Format depends on the parser: Its performance varied strongly with the second-stage parser's prompt or model and could fail outright (e.g., on DDXPlus).
• CRANE comparison: On GSM-Symbolic, constraining reasoning to << >> fields limited symbolic reasoning because restricted operator sets prevented generating needed functions such as round() and encouraged incorrect use of int(). Under nearly identical prompts, In-Writing* consistently outperformed CoT due to improved parsing, yielding improvements of up to 32% compared to CRANE.
• Reported inconsistency in model-size range: The introduction describes models spanning 1.5B–14B parameters, while Section 5.3 states sizes range from 1.5B to 32B parameters. The paper does not reconcile these two statements.
• Prompt-inconsistency findings in prior work: The authors report discovering errors in the prompts of the prior constrained-decoding study, including Shuffled Objects task descriptions that list seven answer choices but instruct the model to choose from four, format variations that vary across tasks, and few-shot examples not aligned with the target dataset; they therefore restricted Shuffled Objects evaluation to zero-shot.
• No prompt optimization for In-Writing: The evaluation was deliberately run with suboptimal prompts giving no guidance on the In-Writing output format; increasing few-shot examples improved the baseline but degraded both NL-to-Format and In-Writing, an effect attributed to increased adherence to the "answer is:" prefix.

Methodology in Plain English

The researchers start from a simple observation about how constrained decoding works: at every decoding step, a finite-state machine built from a regex or grammar masks out tokens that would violate the required format. If that mask is active from the very first token, it can remove reasoning paths that are logically valid but momentarily off-format. Their formal argument shows this as conditioning the reasoning distribution on the format constraint, so invalid-looking traces are given zero probability.

In-Writing instead lets the model generate normally until it emits a designated trigger token. Only after that does the mask turn on, so the format constraint applies to the final answer field but never to the reasoning. The authors state this makes the reasoning distribution independent of the format constraint, while still guaranteeing a structured tail.

They implement this with Litelines, an open-source library built on Outlines that adds allow_preamble and trigger_token_ids. Output schemas are defined with Pydantic and rendered as JSON containing a think_step_by_step key and a final_answer key.

Evaluation covers seven datasets in the NL-to-Format comparison (GSM8K, Last Letter Concatenation, Shuffled Objects, DDXPlus, MultiFin, Sports, and Task 280) using the same prompts, zero-shot setting, and preprocessing as prior work, and GSM-Symbolic in the CRANE comparison using an eight-shot setting. Baseline models were LLaMA3-8B-Instruct and Gemma2-9B-Instruct, extended with Qwen3 (1.7B, 4B, 8B), Qwen3.5 (2B, 4B, 9B), and SmolLM3-3B; the CRANE comparison used Qwen2.5 (1.5B, Coder-7B, Math-7B, Coder-14B), LLaMA3.1-8B-Instruct, and DeepSeek-R1 Distill variants (7B/14B Qwen, 8B LLaMA). For NL-to-Format, the second-stage parser was the open-source Qwen3-32B, replacing claude-3-haiku-20240307 used in prior work. Metrics were accuracy and parse rate, with exact string match against ground truth and answers reported as mean accuracy ± standard deviation over 9 variations. All experiments ran on an NVIDIA A40 GPU with seed=1, temperature=0, and do_sample=False. In the CRANE comparison, model outputs on GSM-Symbolic were manually evaluated against ground-truth formulae because of operator misuse or omission in the gold annotations, treating /, //, and frac as equivalent and int() and round() as equivalent.

Why This Matters

Impact on research. The paper reframes constrained decoding from a generation strategy into a post-reasoning parser and corrector, and offers a formal argument for why applying format constraints during reasoning is harmful. It also documents prompt and annotation inconsistencies in the prior constrained-decoding literature it builds on, and introduces an overlap analysis methodology for comparing two methods that share identical reasoning traces.

Real-world applications:

• Schema-based information extraction where output must conform to a defined JSON, XML, or YAML schema.
• Structured question answering and multiple-choice tasks that require a machine-parsable final answer field.
• Industrial pipelines needing strictly typed model outputs, which the authors note natural generation does not reliably follow, especially for multi-field extraction.
• Symbolic or mathematical reasoning tasks where restricting the operator set (as grammar-constrained reasoning does) blocks necessary functions.

Industry relevance. The method requires no external verifier and no second model, keeps latency overhead minimal (5–20 tokens), and guarantees 100% parse rate with open-source models, which matters for deployed systems where downstream code must reliably consume model output.

Future Directions

• Developing prompt optimization strategies specifically for In-Writing, since the authors deliberately evaluated under a suboptimal prompting setup with no format guidance.
• Extending the approach beyond JSON schemas to other formats such as XML or YAML, and to multi-field or complex extraction with heterogeneous constraints.
• Investigating trigger-token selection more systematically, building on the finding that a single <eos> trigger mitigates premature triggering, to handle remaining failure cases where NL-to-Format succeeds and In-Writing does not.
• Investigating the reported discrepancy in the model-size range (1.5B–14B in the introduction versus 1.5B–32B in Section 5.3), and resolving the gold-answer operator and annotation ambiguities identified in GSM-Symbolic.

Target Audience

Researchers and engineers working on constrained decoding, grammar-guided generation, and structured output from LLMs; practitioners building production systems that must consume model outputs programmatically, such as information-extraction and tool-integration pipelines; and readers interested in the trade-off between reasoning fidelity and output format guarantees. It is most useful to those already comfortable with autoregressive decoding and logit masking, though the central idea is accessible to a general NLP audience.

Authors’ abstract

Natural generation allows Large Language Models (LLMs) to produce free-form responses with rich reasoning, yet the lack of structure makes outputs difficult to verify. Conversely, constrained decoding ensures standardized formats but can inadvertently restrict reasoning capabilities by imposing constraints too early in the generation process. We propose a hybrid approach, namely In-Writing, that combines free-form reasoning and structured generation in a single call. The model first performs unconstrained reasoning and only applies structured decoding after a trigger token is generated, explicitly decoupling reasoning from formatting. We establish that our trigger-token strategies are able to virtually eradicate premature triggering, a failure mode in which constrained decoding interrupts on-going reasoning. Evaluations across diverse datasets covering classification and reasoning tasks demonstrate that our approach outperforms the state-of-the-art by achieving accuracy gains of up to 27% over natural generation. Our code are available at: https://github.com/Nokia-Bell-Labs/InWriting.