Plan-and-Write: Structure-Guided Length Control for LLMs without Model Retraining

AI summary

Overview

Research area: Natural Language Processing — prompt engineering for controlling the output length of large language models (LLMs).

Technical level: Intermediate. The method itself is prompt-only and easy to follow, but the evaluation uses quantitative metrics (Mean Absolute Percentage Deviation, LLM-as-a-Judge quality scoring) and cross-model comparisons.

Scope: The paper proposes "Plan-and-Write," a prompt-engineering method that makes LLMs hit exact word-count targets without retraining or extra inference tooling, and evaluates it across several off-the-shelf models on document summarization (with additional story-generation experiments).

What This Paper Is About

LLMs often ignore explicit length instructions — the paper cites prior work reporting that modern LLMs violate explicit length constraints in over 50% of cases — and current fixes such as Regularized DPO, Length-Instruction Fine-Tuning (LIFT), or inference-time tools require expensive retraining or extra computation. The goal is a deployment-ready alternative: a prompt structure that guides a model to plan, count, and then rewrite its output at an exact target word count, using only the model's existing instruction-following ability.

Key Contributions

1. A two-phase prompt methodology ("Plan-and-Write"): generation is decomposed into a planning phase where the model numbers each word as it drafts, and a verification phase where the same content is rewritten into a coherent paragraph at the exact target length.
2. A model-agnostic, no-retraining design: the approach operates entirely in prompt space, requires no parameter updates and no additional inference passes, and works with black-box LLMs.
3. A systematic evaluation across six models (plus Qwen 2.5 7B as an additional open-weight test) and eight target lengths (20, 50, 100, 200, 500, 1000, 2000, 5000 words), comparing four prompt variants on document summarization, with 960 individual generations (6 models × 8 target lengths × 5 attempts × 4 prompt variants).
4. A quality and cost analysis: LLM-as-a-Judge scoring on correctness, faithfulness, completeness, and relevance, plus a token/latency cost-benefit comparison for the open-weight model.

Main Findings

• Improved length fidelity for most models on summarization: Thinking V1 or Thinking V2 produced the lowest MAPD for four of six models. The largest reported gain is Claude 3.7 Sonnet, where Thinking V1 reached MAPD 0.088 versus the best vanilla result of 0.141 — a 37.6% improvement. Claude 3 Haiku's best was Thinking V2 at 0.120 versus vanilla V2 at 0.130.
• Not all models benefit: Llama 3.1 70B already showed strong length control, achieving the lowest overall MAPD of 0.027 with Vanilla V2. Mistral Large performed best with Vanilla V1 (0.328); its thinking variants were 0.349 (Thinking V1) and 0.402 (Thinking V2).
• Consistency improves: the standard deviation of MAPD was typically lower for the thinking approaches than for vanilla prompts, especially for more advanced models, indicating more predictable length control.
• Longer targets are easier than short ones: most models adhered better to longer target lengths (500+ words), though the paper states that beyond 500 words length fidelity tends to decrease across all prompting strategies, and the shorter-target advantage over vanilla was less pronounced with Plan-and-Write.
• Quality is maintained or improved: under LLM-as-a-Judge, Thinking V1 scored highest on faithfulness (0.96), completeness (0.85), and relevance (0.87), while Vanilla V1 scored marginally higher on correctness (0.91 vs. 0.90). Thinking V2 scored 0.87 correctness, 0.94 faithfulness, 0.84 completeness, and 0.73 relevance; Vanilla V2 scored 0.91, 0.93, 0.73, and 0.69 respectively.
• Open-weight result is essentially a tie: for Qwen 2.5 7B (deployed on AWS SageMaker), Thinking V2 achieved the lowest MAPD (0.280 ± 0.636), only marginally better than Vanilla V2 (0.281 ± 0.169).
• Costs: on Qwen 2.5 7B on an AWS ml.g5.12xlarge instance, thinking prompts used 8,046 average tokens versus 7,914 for vanilla (1.02×), but latency rose from 1,000.1 ms to 1,573.8 ms (1.57×). For that model the extra cost did not translate into better length fidelity.
• Task dependence: on creative story generation, Plan-and-Write did not provide consistent benefits — vanilla prompting achieved better length fidelity for five of six models (the exception being Llama 3.1 70B, where Thinking MAPD was 0.117 versus vanilla 0.133).
• Counting discrepancy in the paper's own framing: the abstract describes evaluation across six LLMs, while the introduction and conclusion say seven; the primary experiment table lists six models, with Qwen 2.5 7B added later as a seventh open-weight test. The conclusion states five of seven models benefit, while the results discussion states four out of six.

Methodology in Plain English

The researchers kept the model frozen and changed only the prompt. A baseline "vanilla" prompt simply asks the model to summarize a document in exactly N words. The Plan-and-Write prompts instead ask for a two-step process: first write out the content with each word numbered in sequence inside a <thinking> block, then rewrite that same content as a normal paragraph inside a <final_answer> block, still with exactly N words. Only the text between the tags is counted. A second variant ("Thinking V2") frames the task as a scientific methodology with an information hierarchy, controlled expansion, and meticulous counting.

Four prompt variants were compared: Vanilla V1, Vanilla V2 (a rephrasing), Thinking V1, and Thinking V2. Each model/length combination was run five independent times per variant, with a delay between attempts to manage API rate limits. The task was summarizing Amazon's 2023 Shareholder Letter, supplied as a PDF. Word counts were computed with NLTK's word_tokenize, excluding punctuation. The main metric is Mean Absolute Percentage Deviation (MAPD), the absolute difference between generated and target words divided by the target words, so lower is better. Quality was scored by an LLM-as-a-Judge on correctness, faithfulness, completeness, and relevance (0–1 scale). The paper states that significance testing was conducted but does not report the test results in the provided content.

Why This Matters

Impact on research: the paper argues that strong length control can be activated through prompt structure alone rather than through training or external tooling, and it identifies a task-dependence pattern — structure-guided prompting helps information condensation (summarization) more than creative generation (stories).

Real-world applications identified by the paper:

• Voice interfaces that need concise spoken answers.
• Mobile applications constrained by screen space.
• Research summaries that need comprehensive detail.
• Documentation systems that must produce varying levels of information density.

Industry relevance: the target setting is production deployment with black-box LLMs, where model parameters are inaccessible and retraining is impractical or cost-prohibitive. The method is described as immediately deployable. The paper also reports the trade-off directly: about 2% more tokens but roughly 57% more latency, and for at least one small open-weight model that overhead did not buy better length fidelity — a consideration for anyone weighing predictable response lengths against serving cost.

Future Directions

1. Hybrid approaches combining prompt engineering with lightweight inference-time modifications.
2. Testing whether structure-guided prompting generalizes to constraints other than length.
3. Adaptive prompting strategies that adjust based on model capabilities and the specific length target.
4. Understanding and addressing the drop in length fidelity beyond 500-word targets, and the failure of the method on creative story generation, both of which point to unresolved limits of prompting-based control.

Target Audience

Practitioners and engineers deploying LLMs in production who need predictable output lengths but cannot retrain or fine-tune models — particularly those working with black-box APIs or cost-constrained serving. It is also relevant to prompt-engineering and evaluation researchers interested in where prompt structure substitutes for training, and to researchers studying controllable text generation, who can use its MAPD comparisons and its negative results (story generation, Qwen 2.5 7B, Mistral Large) as a baseline.

Authors’ abstract

Length control in Large Language Models (LLMs) is a crucial but under-addressed challenge, with applications ranging from voice interfaces requiring concise responses to research summaries needing comprehensive outputs. Current approaches to length control, including Regularized DPO, Length-Instruction Fine Tuning, and tool-augmented methods, typically require expensive model retraining or complex inference-time tooling. This paper presents a prompt engineering methodology that enables precise length control without model retraining. Our structure-guided approach implements deliberate planning and word counting mechanisms within the prompt, encouraging the model to carefully track and adhere to specified length constraints. Comprehensive evaluations across six state-of-the-art LLMs demonstrate that our method significantly improves length fidelity for several models compared to standard prompting when applied to document summarization tasks, particularly for shorter-to-medium length constraints. The proposed technique shows varying benefits across different model architectures, with some models demonstrating up to 37.6% improvement in length adherence. Quality evaluations further reveal that our approach maintains or enhances overall output quality compared to standard prompting techniques. Our approach provides an immediately deployable solution for applications requiring precise length control, particularly valuable for production environments where model retraining is impractical or cost-prohibitive.