Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders

AI summary

Overview

Research area: Computer vision and multimodal learning — specifically vision-language models (VLMs) and the design of their vision encoders.

Technical level: Advanced. The paper assumes familiarity with contrastive pretraining (CLIP/SigLIP), transformer attention variants (causal vs. bidirectional, QK normalization), rotary positional embeddings, and multi-stage VLM training pipelines.

Scope: A technical report from Tencent AI Lab presenting Penguin-VL, a 2B and 8B parameter VLM family whose vision encoder (Penguin-Encoder) is initialized from a text-only LLM rather than a contrastively pretrained encoder, together with the encoder pretraining objectives, data curation pipeline, token compression scheme, and training recipe that support it.

What This Paper Is About

Most leading VLMs assume that the vision encoder must be initialized through massive contrastive image-text pretraining (CLIP, SigLIP). The authors argue this creates an objective mismatch: contrastive learning optimizes for discrimination, which enforces coarse, category-level invariances and suppresses the fine-grained visual cues that dense captioning and complex multimodal reasoning require. Their goal is to test whether a vision encoder initialized from a text-only LLM can beat that paradigm while keeping the overall model small enough (2B and 8B) to run on compute-constrained devices such as smartphones and robots.

Key Contributions

1. Penguin-Encoder. A vision encoder adapted directly from a text-only LLM architecture (Qwen3-0.6B), reusing the LLM backbone weights. The LLM's causal self-attention is converted to bidirectional full attention and the encoder is equipped with 2D rotary positional embeddings (2D-RoPE) to support variable-resolution inputs.

2. Mixed-supervision encoder pretraining. Three auxiliary reconstruction/distillation losses — amplitude, direction, and relation loss — combined with standard image-text loss, allowing joint use of large-scale labeled and unlabeled structured data (for example charts) during the early pretraining stage.

3. Unified training recipe. A full Penguin-VL pipeline integrating a low-to-high resolution curriculum, Temporal Redundancy-Aware (TRA) video token compression, and a two-stage supervised fine-tuning strategy that harmonizes image and video capabilities.

4. Strong performance at compact scale. Consistently strong results across standard image and video benchmarks while remaining parameter-efficient (2B and 8B variants).

Main Findings

• Comparable mathematical reasoning to a leading VLM. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs such as Qwen3-VL on mathematical reasoning. The specific benchmark scores are not reported in the paper content provided.

• Reported advantages in three task families. Penguin-VL surpasses these leading models on document understanding, visual knowledge, and multi-perspective video understanding.

• Representation, not scale, drives the gains. The authors attribute the improvements to better visual representation rather than to model scaling, since they are obtained with a lightweight architecture.

• Penguin-Encoder beats contrastive-pretrained encoders in ablations. The encoder comparison studies consistently show the LLM-initialized encoder outperforming contrastive-pretrained alternatives, preserving fine-grained spatial and temporal cues critical for dense perception and complex reasoning.

• Relation loss is validated separately. The paper states that the effectiveness of the relation loss (one of the three reconstruction objectives) is validated in Table 3; the numbers themselves are not in the content provided.

• Compact-model standing. Penguin-VL 2B is described as achieving strong performance across all modalities in the image (OCR, Math, Knowledge) and video (Long-form & Temporal, General Understanding) benchmark groupings, with clear advantages over existing state-of-the-art open-source models.

• Encoder parameter scale. After removing parameters unrelated to visual processing, the encoder uses approximately 400M parameters for visual modeling, described as broadly consistent with SigLIP.

Methodology in Plain English

The authors start from a text-only language model (Qwen3-0.6B) and turn it into a vision encoder. Two changes make this possible: making attention bidirectional so every visual patch can attend to every other patch (rather than only to earlier tokens), and adding 2D rotary positional embeddings so images of different sizes and aspect ratios can be processed. Inputs are handled at native resolution when the token budget allows and resized otherwise.

Because a text model has never seen images, the authors first run a low-resolution pre-training stage: roughly 100M samples capped at 2048 visual tokens (about 600x600 pixels), supervised by original, often noisy captions. To use unlabeled structured data such as charts — where good natural-language descriptions are scarce — they add a teacher-guided reconstruction signal made of three losses: an amplitude loss matching feature magnitudes, a direction loss matching feature directions via cosine similarity, and a relation loss matching inter-patch similarity structure. A second high-resolution stage drops the reconstruction branch, raises the resolution to 10240 visual tokens, and fine-tunes on high-quality re-captioned data.

Data is curated carefully. Penguin-Recap-I contains 57.2 million image-text pairs sampled from COYO-700M and DataComp-1B, filtered for quality and balanced through hierarchical k-means clustering plus greedy intra-cluster selection, then annotated across ten structured aspects (subjects, actions, spatial relationships, OCR text, mood, knowledge-intensive inferences, and others) before being synthesized into a single long caption. Penguin-Recap-V contains 3.7M video-text pairs from 29 public video datasets, deduplicated by clustering and filtered by optical-flow motion scores, with annotations at three levels: event-level atomic descriptions, chapter-level narratives, and holistic video summaries.

For the VLM itself, a lightweight two-layer MLP projector maps visual features into the LLM's hidden size without spatial restructuring. Long videos are handled by Temporal Redundancy-Aware (TRA) token compression, which splits frames into key frames (rapid temporal change) and intermediate frames (stable context) and allocates tokens through a three-stage cascade: keep native resolution if the budget allows; otherwise downscale both frame types synchronously by a shared factor with intermediate frames held at a 4x spatial down-sampling relative to key frames (T_k ≈ 16 T_i); and finally, once intermediate frames hit the minimum per-frame token count, clamp them and push the remaining compression onto key frames only.

Training proceeds in three stages. Stage 1 is encoder training (language decoder frozen, encoder and projector optimized). Stage 2 is VLM pre-training with everything trainable on roughly 121M samples, of which general caption data is 64 percent and the remainder includes document (14.45 percent), grounding (6.31 percent), text (4.45 percent), math (2.88 percent), OCR (2.42 percent), MM code (2.38 percent), code (1.54 percent), region caption (1.2 percent), interleaved (0.49 percent), and science (0.27 percent). Grounding uses 7.7 million samples with bounding boxes normalized to an integer [0, 1000] coordinate space, and 1.5 million self-constructed region caption QA pairs provide the reverse direction. Stage 3 is two-stage supervised fine-tuning: general compliance using a blend of image and video instructions first, then complex video-centric reasoning, including temporal ordering and temporal grounding tasks built from dense time-level descriptions.

Why This Matters

Impact on research. The paper challenges a largely unquestioned default — that a strong VLM requires a contrastively pretrained vision backbone. If an LLM-initialized encoder can match or exceed that setup, the field gains an alternative pretraining path, and the argument that gains come from representation quality rather than parameter count reframes where efficiency research should focus.

Real-world applications:

• Mobile and edge assistants on smartphones, where the 2B variant targets deployment under strict latency and compute constraints.
• Robotics, where a compact on-device model with reliable temporal and spatial understanding is required.
• Document and chart understanding, a domain the authors specifically engineer for with unlabeled chart data and reconstruction supervision.
• Long-form and multi-perspective video understanding, including dense captioning, temporal ordering, and temporal grounding of actions.

Industry relevance. The work comes from Tencent AI Lab and ships with released artifacts: code at a public GitHub repository and 2B and 8B checkpoints on Hugging Face. The encoder is described as a strong drop-in alternative for compute-efficient VLMs, which matters for vendors balancing capability against serving cost.

Future Directions

• Scaling the vision encoder. The authors note that the framework can be readily extended to train vision encoders with substantially larger model capacities, for example 1.5B parameters, if future scenarios require it.
• How far the LLM-initialization advantage holds. Because the encoder size is fixed independently of the VLM backbone, it remains open whether the approach's gains persist or grow as encoder capacity increases relative to the downstream LLM.
• Weaker reliance on proprietary annotation. Image captions and video annotations were synthesized with a proprietary model and a proprietary video annotation model; whether comparable quality is reachable from open models is not addressed in the content provided.
• Broadening the objective beyond discrimination. The paper frames contrastive learning's mismatch with sequence token prediction as a general issue; whether reconstruction-based auxiliary losses transfer to other multimodal settings such as audio is not explored here.

Target Audience

Researchers and engineers working on multimodal foundation models, particularly those focused on parameter efficiency, vision encoder pretraining, or video token compression. It is also relevant to practitioners deploying VLMs on edge hardware who need to understand the trade-offs between encoder choice, resolution budgets, and token compression. Readers without background in transformer architectures and contrastive learning will find the methodology sections dense, though the core argument about contrastive objectives suppressing fine-grained detail is stated accessibly in the introduction.

Authors’ abstract

Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: https://github.com/tencent-ailab/Penguin-VL