Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generation

AI summary

Overview

Research area: Natural Language Processing — LLM personalization, reasoning-augmented generation, and long-form text generation.

Technical level: Intermediate. The core idea is intuitive (two models working side by side), but the training and inference parallelism arguments require comfort with latent reasoning, hidden states, and autoregressive decoding.

Scope: This paper introduces FlyThinker, a framework that runs a separate latent reasoning model in parallel with a generation model so that personalized long-form text can be produced with token-level, continuously updated reasoning rather than a single upfront reasoning step.

What This Paper Is About

Preference-aligned LLMs are typically tuned toward population-level preferences, which overlooks what any individual user actually wants. Personalization methods based on prompt customization or fine-tuning struggle because user interests are usually implicit — visible only in past behavior — and existing "think-then-generate" approaches do one static reasoning pass before writing, which is poorly suited to long outputs whose content evolves as it is written. FlyThinker's goal is to interleave reasoning with generation at the token level ("think-while-generating") while keeping both training and inference efficient.

Key Contributions

1. A new paradigm. The paper puts forward the "think-while-generating" paradigm for personalized long-text generation, in which a reasoning step is produced after each generated token to guide the next segment of response.
2. The FlyThinker framework. A design that pairs a separate Reasoner (producing latent token-level reasoning from the query and previously generated response) with a Generator that fuses that reasoning into token embeddings, allowing reasoning and generation to run in parallel.
3. Parallel training and inference. Because each reasoning step depends only on the query and previously generated response tokens — not on prior reasoning tokens — all reasoning tokens for a training example can be computed in a single forward pass. At inference, the two models operate in a staggered manner so reasoning adds negligible latency.
4. Empirical validation. Extensive experiments on three LongLaMP tasks using Qwen2.5-3B-Instruct (with additional results on Qwen2.5-7B-Instruct and Gemma-7B-it in appendices L and M) showing gains in personalization quality and efficiency over tuning-free and tuning-based baselines.

Main Findings

• Consistent gains over strong baselines: On Product Review, FlyThinker reaches ROUGE-1 of 0.3663 (+3.1% over SFT) and BLEU of 4.3620 (+11.5%). Similar trends hold on Abstract Generation and Topic Writing, where the paper reports BLEU improvements of about 10% over SFT and higher ROUGE-L scores.
• Robustness across domains: Gains hold regardless of task type or text length. On Abstract Generation, FlyThinker reaches BLEU 6.3383, versus CoT at 5.8538 and Coconut at 5.2378.
• Better than latent-reasoning baselines: Coconut, which uses a think-then-generate latent reasoning strategy, is consistently outperformed by FlyThinker on ROUGE and BLEU across all tasks.
• Training efficiency: FlyThinker trains much faster than reasoning-based baselines (CoT and Coconut, whose autoregressive reasoning is sequential) while remaining close to plain SFT. When the Reasoner is scaled down, cost approaches SFT.
• Inference efficiency: FlyThinker nearly matches SFT latency because the Reasoner prepares reasoning for the next step while the Generator predicts the current token. The paper notes that computational cost is not reduced in the same way and memory usage increases.
• Mitigating context drift: Segmenting outputs into four contiguous position-based chunks shows all baselines degrade in later segments (the "context drift" problem), while FlyThinker outperforms all baselines substantially in the [100, 200] and [200, 300] token ranges on METEOR.
• Reasoner scale matters non-monotonically: Reducing the Reasoner from 3B to 1.5B yields nearly identical ROUGE-1/L, BLEU, and METEOR. Dropping to 0.5B causes clear drops in ROUGE-L and BLEU.
• The weighting parameter λ is stable in a range: Sweeping λ ∈ {0, 0.2, 0.5, 1, 1.5, 2, 5}, moderate values (the paper cites 0.5–2) give the best overall performance. Very small λ underuses reasoning; very large λ destabilizes it. Within [0.2, 2.0], all results remain above the SFT baseline.
• Where to inject reasoning: Ablations comparing input-only, output-only, and global (both) augmentation find global enhancement most consistent, with input-only producing diverse paraphrases but weaker alignment (lower ROUGE/BLEU) and output-only underperforming due to weak contextual grounding.

Methodology in Plain English

FlyThinker splits the job between two LLMs. The Reasoner looks at the user's query and history plus the response text produced so far, and emits one latent reasoning token per step — a hidden-state vector taken from the final layer, not readable text. The Generator is a second LLM that predicts the next response token, but before doing so it adds the Reasoner's latent vector (scaled by λ) to the token embedding at each position, so reasoning directly shifts the generation.

The critical design choice is that the Reasoner never looks at its own earlier reasoning outputs — only at the query, history, and already-generated response. That severs the chain of sequential dependencies. During training, the full ground-truth response can therefore be fed in, letting every latent reasoning token be produced in one forward pass, followed by one parallel pass of the Generator to predict all tokens. Both models are trained jointly end-to-end with the standard next-token prediction loss, with no extra supervision or auxiliary objectives for the reasoning.

At inference the two models run in a staggered parallel pattern: while the Generator samples the current token, the Reasoner computes the reasoning that will guide the next one. Experiments use the user-based split of the LongLaMP benchmark across Product Review, Abstract Generation, and Topic Writing. Baselines fall into tuning-free methods (Non-pers, RAG, CoS) and tuning-based methods (SFT, LLM-TRSR, NextQuill, CoT, Coconut), evaluated with ROUGE-1, ROUGE-L, BLEU, and METEOR (with BERTScore in Appendix E). All runs used four NVIDIA A100 GPUs, HuggingFace Transformers, Python 3.10, and the Verl training framework.

Why This Matters

Impact on research. The paper reframes personalization reasoning as a continuous, token-level process rather than a one-shot preamble, and shows that this can be done without paying the sequential-decoding tax that makes CoT-style and latent-reasoning methods slow. It provides a concrete architectural recipe — depend on responses, not on prior reasoning — for making reasoning parallelizable in training.

Real-world applications:

• Personalized long-form review writing, where a system drafts in the style, tone, and opinions a specific user has shown in past reviews.
• Customized report or abstract generation, where the output must stay aligned with an individual's established framing over many paragraphs.
• Creative and topic writing assistants that need to keep a user's voice consistent as a piece grows longer.
• Any deployment where reasoning-enhanced personalization must run under latency constraints, since the method is designed to add negligible latency over a non-reasoning model.

Industry relevance. The efficiency story is the practical hook: personalized reasoning quality is usually traded against serving cost, and FlyThinker claims near-SFT latency with reasoning on every token. The headline caveat is that memory usage increases, which matters for deployment economics.

Future Directions

• Reducing memory overhead. The paper explicitly notes that inference cost is not reduced the same way latency is, and that memory usage increases — an obvious target for follow-up work.
• Understanding the Reasoner capacity floor. The ablation shows 1.5B preserves quality while 0.5B degrades it, but the paper does not identify what capability the smaller model lacks or whether the threshold shifts with task, domain, or response length.
• Extending beyond the evaluated settings. Results with Qwen2.5-7B-Instruct and Gemma-7B-it are placed in appendices L and M rather than the main text; broader backbone coverage, additional personalization benchmarks beyond LongLaMP's three tasks, and very long sequences remain open.
• Integrating with existing reasoning and alignment methods. Whether FlyThinker's parallel latent reasoning composes with explicit CoT, reinforcement-learning-based alignment, or retrieval-augmented personalization is not reported in this paper.

Target Audience

Researchers and engineers working on LLM personalization, reasoning-augmented generation, and efficient inference. It is most useful for readers already familiar with autoregressive decoding and latent reasoning who want a concrete architecture for making reasoning concurrent with generation, and for practitioners weighing personalization quality against serving latency and memory in long-form generation systems.

Authors’ abstract

Preference alignment has enabled large language models (LLMs) to better reflect human expectations, but current methods mostly optimize for population-level preferences, overlooking individual users. Personalization is essential, yet early approaches-such as prompt customization or fine-tuning-struggle to reason over implicit preferences, limiting real-world effectiveness. Recent "think-then-generate" methods address this by reasoning before response generation. However, they face challenges in long-form generation: their static one-shot reasoning must capture all relevant information for the full response generation, making learning difficult and limiting adaptability to evolving content. To address this issue, we propose FlyThinker, an efficient "think-while-generating" framework for personalized long-form generation. FlyThinker employs a separate reasoning model that generates latent token-level reasoning in parallel, which is fused into the generation model to dynamically guide response generation. This design enables reasoning and generation to run concurrently, ensuring inference efficiency. In addition, the reasoning model is designed to depend only on previous responses rather than its own prior outputs, which preserves training parallelism across different positions-allowing all reasoning tokens for training data to be produced in a single forward pass like standard LLM training, ensuring training efficiency. Extensive experiments on real-world benchmarks demonstrate that FlyThinker achieves better personalized generation while keeping training and inference efficiency. Our code is available at https://github.com/wcb0219-sketch/FlyThinker.git.