Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

AI summary

Overview

Research area: Computer Vision / multimodal document and media parsing (omni-modal structured output for LLM agents).

Technical level: Advanced — the paper details encoder architecture, positional-encoding design, a four-stage pre-training curriculum, RLVR, and a custom on-policy distillation procedure.

Scope (one sentence): The paper introduces Youtu-Parsing-Omni, a single 5B-parameter model with one shared perception encoder and one JSON output schema that parses documents, natural images, audio, and video into spatially and temporally grounded structured records across ten tasks.

What This Paper Is About

LLM agents increasingly need to read documents, images, audio, and video, but re-encoding raw pixels and waveforms at every step is costly and hard to audit, since a raw-input answer does not say which region or time interval supports it. Omni-modal parsing solves this by converting each input once into a structured text record of what it contains and where or when each part occurs, so any LLM (even text-only) can read, act on, and cite it.

The problem is that existing parsers cover this range only partially: document parsers are strong on standard pages but leave chemical structures and sheet music to specialist models, and parsing across images, audio, and video was only recently formulated with a dedicated benchmark. The goal is therefore a single parser with one encoder for all modalities and one output schema for all tasks.

Key Contributions

1. Youtu-Omni-Encoder — one encoder for all modalities. A single shared Transformer (initialized from a pre-trained language model, Qwen3-0.6B-Base, without its embedding layer and LM head) reads images, audio, and video with shared weights and a unified space-time positional encoding. Only lightweight input stems and merger–projector heads remain modality-specific, so nearly all perception parameters are shared and adding a modality requires a new stem and projector rather than a second tower.

2. Fusion and temporal alignment inside perception. Vision and audio tokens write into the same temporal rotary slots, and a few fusion layers let a frame attend to the sound of the same instant inside short audio-visual windows, so cross-modal correspondence is computed during perception rather than reconstructed by the language model.

3. OmniSchema — one output schema for ten tasks. A single JSON schema (modality, localized elements or segments, global_description, plus task-specific fields) covers document parsing, chemical structure recognition, sheet music recognition, chart parsing, flowchart parsing, geometric figure parsing, natural image parsing, audio parsing, natural video parsing, and text-rich video parsing.

4. OSAD — self-evolving post-training. OmniSchema-Aware On-Policy Distillation trains on the model's own rollouts against token-level targets from a frozen, reference-conditioned copy of itself, using Jensen–Shannon divergence for content tokens and forward KL for structure tokens. After each round the trained weights refresh the teacher, so parser and supervision co-evolve; the paper runs six OSAD rounds, following SFT and schema-routed RLVR.

Main Findings

• State-of-the-art document parsing: the single 5B-parameter checkpoint scores 96.96 Overall on OmniDocBench v1.6, the best result reported in the paper.
• Strongest open-weight omni-modal parser: 75.08 Avg on OmniParsingBench, ahead of Logics-Parsing-Omni (72.26), with the largest gains on audio and video.
• Best overall chart parsing: a score of 95.02, the highest overall among compared systems.
• Chemical structure recognition: exact fingerprint-match rate (Tani@1.0) of 54.2, versus 49.6 for DeepSeek-OCR 2.
• Sheet music transcription: character error rate (CER) of 22.73, on par with the specialist model LEGATO (23.30, as reported in LEGATO's original paper), while staying far ahead of general-purpose systems.
• Highest exact chemical-fingerprint agreement among the compared systems, and leading performance on most domain-specific benchmarks (Fig. 1c).
• Ablation-style evidence for the design: the authors evaluate successive OSAD checkpoints (Sec. 5.7) to show that teacher refresh improves the parser over rounds, and Stage 2 keeps both the fused and independent video routes usable at inference via a fused-route probability of 0.8.

Methodology in Plain English

Architecture. A text decoder (following the Youtu-LLM-2B pre-training recipe, dense, keeping 1D rotary positions) is fed by one encoder. In that encoder, simple stems turn image patches and log-mel audio frames into tokens; a shared stack of Transformer blocks contextualizes any packed mix of images, audio chunks, and videos in a single pass; and small merger–projector heads map the result to the decoder's width.

Position and fusion. The positional budget is split into three segments — time, height, width — so 2D layout, 1D time, and their correspondence all fit in one rotary encoding. Frames take their time index from the frame, audio from its offset, and both share a time scale tied to the audio token rate. On the "fused" route, video events are packed in temporal order into groups (one or more frames followed by the audio chunks before the next frame), and each group becomes an attention window in the fusion layers.

Training, in stages. Four pre-training stages progressively expand what is trained and what is seen: Stage 0 trains only the encoder with the decoder frozen, on small inputs, to align new stems with pre-trained weights; Stage 1 unfreezes everything and consumes 1.28T tokens with native-resolution images and four fusion layers; Stage 2 anneals the learning rate from 10⁻⁴ to 10⁻⁵ over 512B tokens with layer-wise learning-rate decay; Stage 3 extends context to 128k tokens for multi-page documents and long videos.

Post-training, in three steps. Supervised fine-tuning (16B tokens) locks in the OmniSchema output format and instruction following. Schema-routed RLVR then optimizes on the model's own parses with a reward gated by decodability and payload presence, combining schema validity with penalties for repetition and incomplete coverage plus task-specific content accuracy. Finally, OSAD supplies token-level supervision: admitted rollouts (passing JSON-validity, schema-conformance, box-syntax, and repetition checks) are scored by a student with LoRA adapters and by a frozen teacher that also sees the reference parse, with different divergences for content versus structure tokens.

Why This Matters

Impact on research. The paper argues that grounded parsing — not captioning or question answering — is the right interface between non-text data and language models, and shows that one encoder plus one schema can match or beat specialist models on symbolic domains (chemistry, sheet music) that are usually served by dedicated systems. It also offers an alternative to scalar-reward RL for heterogeneous structured outputs, replacing a hand-designed reward with token-level targets derived from the model itself.

Real-world applications:

• Document digitization and archival: converting scanned pages, tables, formulas, and charts into structured, citable records at native resolution.
• Media indexing and search: turning audio and video into time-stamped segments with transcripts, speakers, on-screen text, actions, and scene descriptions that a text-only LLM can query.
• Scientific and technical literature mining: extracting chemical structures as SMILES and sheet music as ABC notation from figures inside documents.
• Auditable agent pipelines: giving agents spatial bounding boxes and time intervals so any claim can be traced back to the region or interval that supports it.

Industry relevance. Because parsing happens once and produces a compact structured record, downstream systems avoid repeatedly re-encoding long visual and acoustic token sequences — a direct cost and latency saving. A single 5B checkpoint serving ten tasks also simplifies deployment compared with maintaining one specialist model per input type. Code and model weights are publicly released (GitHub: TencentCloudADP/youtu-parsing; Hugging Face: tencent/Youtu-Parsing-Omni).

Future Directions

• Adding modalities beyond the current three. The design claims that a new modality needs only a new stem and projector, but the shared weights must still be trained on the new input; whether this holds for further modalities is untested here.
• Extending fusion beyond short windows. Fusion currently operates within small audio-visual windows, leaving dependencies across windows to the language model; longer-range cross-modal reasoning is an open question.
• Schema evolution. OmniSchema is fixed at ten tasks; how the schema and the token-role objective adapt when tasks or fields change is not addressed.
• Independent replication of the omni-modal benchmark scores. OmniParsingBench results are scored with the authors' own toolkit rather than the official leaderboard, so third-party verification would strengthen the comparison.

Target Audience

Researchers and engineers working on document AI, multimodal LLMs, and agent perception layers; practitioners building retrieval or parsing pipelines over mixed media; and readers interested in post-training methods (RLVR, on-policy distillation, self-distillation) for structured generation. The paper assumes familiarity with Transformer architectures, rotary position embeddings, and reinforcement-learning fine-tuning, so beginners will find the model and training sections dense but the problem framing and benchmark results accessible.

Authors’ abstract

We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.