onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

AI summary

Overview

Research area: Natural Language Processing — data annotation tooling and human-in-the-loop data collection for LLM alignment and agent training.

Technical level: Intermediate. The paper is primarily a systems and tooling paper, but interpreting it requires familiarity with on-policy data, SFT, preference data, logprobs, and process reward models.

Scope: The paper introduces onPanda, an interactive annotation tool whose core interaction is token-level correction of model responses and agent trajectories, and reports a controlled annotation study, a user study, production deployment statistics, and the release of the Panda-CVL dataset plus a token-level correction benchmark.

What This Paper Is About

Building high-quality alignment data is expensive, and the existing options trade off against each other: manually writing or post-editing responses gives good quality but is slow and produces off-policy data, while preference ranking is cheaper but gives only coarse response-level supervision and cannot fix responses the model never samples. onPanda's goal is to let a human steer a model's output by correcting single tokens — locating the first inappropriate token, either clicking a substitute from the model's own top-k candidates or typing a free-form replacement, and then letting the model continue generating from the corrected prefix — so that annotation is fast, the resulting data stays close to the rollout model's distribution, and each correction is recorded as fine-grained supervision.

Key Contributions

1. The onPanda annotation tool, built around a "locate–correct–continue" token-level correction loop, which produces alignment data that stays close to the rollout model's sampling distribution rather than off-policy human-written text.
2. Experimental validation of annotation efficiency and on-policy fidelity, via a controlled three-annotator study against POTATO (manual post-editing) and Argilla (preference ranking), plus a NASA-TLX user study, and demonstration of multimodal and agent-trajectory annotation support.
3. The Panda-CVL dataset, a predominantly Chinese vision-language dataset of 7,491 annotation sessions (6,839 training, 652 test) annotated with onPanda, released together with an accompanying benchmark for token-level correction.
4. A production deployment report covering vision, audio, and agentic data streams (25,596, 105,143, and 1,257 annotation sessions respectively) and roughly 388K token-level corrections.

Main Findings

• Annotation speed: In the controlled study, onPanda's median time was 330 s per prompt, 51.5% less than POTATO's 681 s and on par with Argilla's 336 s; in means, onPanda (516 s, 515.6 s in the text) was 27.5% and 24.7% lower than POTATO (711.1 s) and Argilla (684.5 s). The abstract states that a small controlled study suggests onPanda reduces median annotation time by 52% over manual post-editing.
• Output quality: onPanda attained the highest pairwise win rate (66.7%), judged by GPT-5.5 with each pair evaluated in both orders; a separate anonymized human comparison preferred onPanda over POTATO in 54.8% of pairs.
• On-policy fidelity: onPanda and Argilla reached PPL 1.181 (+0.86%) and 1.161 (−0.83%) against a re-sampling baseline of 1.171, both within 1% of baseline and within re-sampling noise (per-prompt PPL fluctuates by about ±2.8% across rollouts); POTATO yielded 1.596 (+36.31%).
• SFT coverage and preference pairs: Argilla yielded a qualified SFT response on only 11/21 (52%) prompts, while POTATO and onPanda reached 100% (21) coverage by design. onPanda derived 7.43 preference pairs per prompt, versus 6.00 for Argilla's four-way ranking and 0.95 for POTATO.
• Workload: On an adapted 0–10 NASA-TLX scale (lower is better), onPanda scored 3.1, compared with Argilla 5.4 and POTATO 6.8.
• Sparse human intervention at scale: Across all qualified production responses, 97.0% of tokens were model-generated, 2.1% were selected from candidates, and only 0.9% were manually typed.
• Token-level correction is hard for current models: On the Panda-CVL test set, the best F1 reached only 17.09% (GPT-5.5, which also had the highest GoodAcc at 53.37%). GPT-6 obtained the highest Loc.-NG (24.46%) and Corr.-NG (15.83%), and GPT-5.6-sol the best Format score (99.98%). Seven of the nine reasoning models scored above 90% Format while Corr.-NG stayed below 16% for all of them.
• Annotation agreement is above chance but imperfect: On Panda-MultiRef-21 (21 image-description prompts, four independent annotations each, 84 annotations), pairwise human location agreement was 30.95% under exact matching versus a 0.20% uniform random-location baseline; with a four-token tolerance the rates were 44.44% and approximately 1.84%. Replacement-token agreement was 69.44% conditional on identical correction positions.

Methodology in Plain English

The tool streams generation with logprobs, recording each token's sampled probability and its top-k candidates (default 20). Probabilities are color-coded below each token, so low-confidence — often error-prone — tokens are visually obvious. The annotator reads the response, finds the first inappropriate token, and either clicks an alternative from the displayed candidate list or double-clicks to type a replacement. The system then keeps everything before that point unchanged, discards everything after it, and asks the model to continue generating from the corrected prefix. This loop repeats until the response is acceptable.

Because tokenizers can split characters or emoji, the interface groups tokens into readable "chunks" along grapheme boundaries, though stored records stay token-precise. Free-form typed text initially has no probability data; a single prompt_logprobs request recomputes per-token probabilities and candidates for the whole response, which also lets onPanda score arbitrary pasted text or a different model's output. Every correction forks a new node in an annotation tree, so the initial rollout and each intermediate version are saved automatically, with an operation log recording operation type, timestamp, correction content, whether the operation is marked on-policy, and a snapshot of the sampling configuration. Each dialog carries a quality verdict (is_good) and a project-defined annotation schema; a companion Python library parses the serialized .panda.json file into SFT samples (nodes with is_good=Y) and into preference pairs from positive–negative nodes under the same prompt.

For agent and reasoning data, a "response template" mechanism converts bidirectionally between structured messages (reasoning, content, tool_calls) and the model's native token stream, so special tokens such as </think> and <|tool_call_begin|> are visible and directly correctable. External tools connect via MCP, and a harness_to_mcp adapter wraps harnesses including Claude Code, Codex, and OpenClaw; tool calls can be configured to await annotator approval, and rejected trajectories are kept as negative samples.

The controlled evaluation used 3 annotators labeling 21 image-description prompts split into 3 groups, with a Latin-square design rotating the order of the three methods; no annotator labeled the same prompt with different methods, and none were onPanda developers. Initial rollouts came from Qwen3.5-35B-A3B in instruct mode with temperature 0.7 and top-p 0.8. The benchmark asks models to perform the same correction task as annotators, using a find-and-replace output format; the 652 test conversations were expanded into 2,126 evaluation instances (652 good, 1,474 not good), and correctness was checked via tokenizer-agnostic correction triples computed with the Qwen3.6 tokenizer.

Why This Matters

Impact on research: The paper reframes annotation not just as a way to produce final answers but as a way to produce position-precise, naturally paired positive–negative supervision. Each correction records the error position, the substitute token, and the samples before and after the correction, which the authors argue can be converted into preference data for reward model or DPO training, or into process reward data for PRM training. It also opens a new evaluation target — the released benchmark shows current models score at most 17.09% F1 on the correction task, which the authors treat as a novel supervision signal for more efficient post-training.

Real-world applications:

• Vision-language data pipelines: the released Panda-CVL dataset and its production sibling show how image-description data can be annotated and quality-gated at scale.
• Audio data annotation, which accounted for the largest production stream at 105,143 annotation sessions.
• Agent trajectory annotation in realistic environments, where corrected tool calls are actually executed through MCP-connected harnesses and real feedback is fed back into the context.
• Model inspection and data quality auditing, using per-token probability color coding and entropy statistics to diagnose whether a faulty rollout comes from the model or from a pipeline error.

Industry relevance: All authors are affiliated with StepFun (with Xiamen University co-authorship), and onPanda is integrated into an in-house production annotation system across three data streams, with 32, 42, and 24 annotators and 3,557.4, 18,871.3, and 523.5 total human-hours for vision, audio, and agentic work respectively. The lightweight web-app mode requires no database and only needs an inference API supporting prefix continuation and top-k logprobs — capabilities the authors note are already available from vendors including Doubao, DeepSeek, Kimi, and StepFun, and natively supported by frameworks such as vLLM and SGLang.

Future Directions

• Post-training on token-level correction data: the authors are developing a "token-level correcting model" (a token-level counterpart of reward models) and "token-level correction optimization" (a token-level counterpart of DPO); the downstream training benefit of these signals remains unverified by training experiments in this paper.
• An annotation–training flywheel: a correcting model could pre-locate suspicious tokens and propose fixes for annotators to review, while each newly annotated batch triggers an incremental training step that updates the served rollout model, keeping data on-policy with respect to the latest weights and mitigating on-policy decay.
• Online learning from textual feedback: upcoming support for annotating earlier turns would let later-turn feedback (tool errors, user complaints) supervise earlier generation, turning arbitrary textual feedback from annotators, environments, or end users into token-level corrections.
• Open limitations to address: extending the modest controlled study (3 annotators × 21 image-description prompts, a single rollout model) and moving beyond an in-house annotator pool; reducing reliance on LLM-as-a-judge for quality evaluation; building out controlled agent studies; and handling the sparse-error premise, since the efficiency and on-policy advantages diminish when corrections become dense.

Target Audience

Researchers and engineers working on LLM alignment data pipelines, RLHF/DPO/PRM data construction, and agent trajectory collection will benefit most, as will practitioners building annotation platforms or choosing between post-editing and preference-ranking workflows. The paper is also relevant to teams evaluating model-inspection tooling based on token probabilities, and to those interested in the released Panda-CVL benchmark for vision-language token-level correction, since it defines a new task and provides reference annotations. Finally, it is useful reading for anyone interested in whether fine-grained, position-precise human corrections can serve as an alternative or complement to response-level preferences.

Authors’ abstract

We present onPanda, an interactive tool for efficiently annotating LLM alignment data and agent trajectories. onPanda adopts token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate token and either picks a substitute from the model's candidate tokens or types the correct text via free-form editing. The system then truncates everything after that position and continues generation from the corrected prefix, repeating this locate-correct-continue loop until a satisfactory response is obtained. This mechanism lets annotators precisely steer model outputs at low cost: a small controlled study suggests that onPanda reduces median annotation time by 52% over manual post-editing. Since the vast majority of tokens in the final response are generated by the model itself, the resulting data largely preserves the model's sampling distribution and is well suited for constructing on-policy SFT and preference data. Furthermore, the token-level corrections recorded during annotation provide fine-grained supervision with precise positions and naturally paired positive--negative samples. onPanda also connects to external tools and harnesses, enabling interactive trajectory annotation in realistic environments. In addition, we release Panda-CVL, a dataset annotated with onPanda, together with a benchmark for token-level correction.