SenseNova-U1.5: Towards Native Unified Visual Intelligence

AI summary

Overview

Research area: Multimodal foundation models — specifically native unified models that perform visual understanding, reasoning, and image generation inside a single end-to-end system.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, diffusion/flow-matching objectives, reinforcement learning from preference feedback (GRPO-family methods), and on-policy distillation.

Scope: This paper describes SenseNova-U1.5, an 8B-parameter Mixture-of-Transformers model that generates and understands images directly in pixel space without a vision encoder or VAE, and introduces a spatially coupled image decoder plus a "specialize-then-unify" post-training pipeline to raise fidelity, text rendering, and editing quality up to 4K resolution.

What This Paper Is About

Most image generation systems see the world through one representation and create through another: a pretrained vision encoder converts images into semantic features for understanding, while a VAE compresses them into latents for generation. These two spaces do not align, so perception and creation must be stitched together rather than learned jointly. SenseNova-U1.5 continues a line of work (NEO-unify, SenseNova-U1) that removes both components, and its goal is to fix the main weakness of that approach — a decoder that painted each visual token independently, causing visible seams and texture breaks, especially at high resolution.

Key Contributions

1. Spatially joint pixel reconstruction. The per-token MLP decoder is replaced with a lightweight spatial decoder that restores 2D token topology and reconstructs pixels through Pixel Shuffle upsampling (factors 2, 2, 8) interleaved with 3×3 convolutions, so neighbouring regions exchange information before pixels are finalized.

2. Resolution-aware noise conditioning extended to 4096×4096. A resolution-dependent noise-scale embedding is normalized against a 4K reference and added to the diffusion timestep embedding, allowing the denoiser to adapt consistently across resolutions and aspect ratios.

3. Specialize-then-unify post-training. Four reinforcement-learning experts (aesthetics, bilingual OCR/text rendering, infographic generation, image editing) are trained separately with task-specific data, rewards, samplers, and regularization, then consolidated into one policy via multi-expert on-policy velocity-field distillation along the student's own generation trajectories.

4. A scaled, curated training recipe with a unified objective. Autoregressive language modelling, pixel-space flow matching (x-prediction), and an LPIPS perceptual loss are combined into one loss, with a five-stage curriculum spanning generation pretraining, unified mid-training, unified supervised fine-tuning, capability-specific RL, and distillation. Training code for SFT, RL, and OPD is slated for open-source release.

Main Findings

• High-resolution generation improves markedly. Joint spatial reconstruction reduces seams, grid artifacts, and texture discontinuities that appear when patches are decoded in isolation, and the model natively supports output up to 4K.

• Bilingual text rendering and infographics strengthen. A graded OCR reward (a multiset intersection-over-union over recognized versus intended text) trains text fidelity, and a separate infographic expert improves small-text rendering, dense layouts, and information hierarchy — without introducing any infographic-specific reward model.

• Editing preserves what should not change. The editing expert uses a bottleneck-style reward that takes the minimum across instruction fulfillment, edit execution, overall visual quality, text-editing quality, and unedited-region preservation, so strength in one dimension cannot mask a critical failure in another. This improves subject identity, geometry, and unmodified-region preservation.

• Capabilities consolidate into one model. Multi-expert on-policy distillation transfers the four specialists into a single unified policy, exploiting complementary strengths that joint reward optimization tends to entangle.

• Understanding transfers to visual creation. Despite limited exposure to structured generation formats during training, the model generalizes to long, complex, structured visual instructions — evidence that planning and structural knowledge acquired through multimodal understanding carry over to image synthesis.

• Compact representation stays efficient. Each 32×32-pixel region is compressed into a single visual token, keeping sequences tractable while still supporting fine-grained control and multi-reference composition.

• Architectural unification without full parameter sharing. Understanding and generation share self-attention as a communication interface but retain separate attention projections, normalization, and feedforward modules, routed by token type.

Methodology in Plain English

The model processes raw images and text in one sequence. Convolutional layers downsample images by 16× and then 2×, so every 32×32 pixel block becomes one token; two-dimensional sinusoidal embeddings keep spatial coordinates, and special tokens mark image boundaries. Text uses an ordinary language tokenizer. Both go into a single Mixture-of-Transformers backbone, where shared self-attention lets the two streams interact while stream-specific weights keep their computations distinct. Text tokens attend causally; tokens inside clean image blocks attend bidirectionally; generation tokens attend bidirectionally within their block and to preceding clean context, with the reverse path masked so clean representations never see noisy generation states.

Generation learns directly in RGB space through flow matching. Training forms a noisy sample by interpolating between the clean image and Gaussian noise scaled by a resolution-dependent magnitude, then trains the model to predict the clean endpoint, from which a velocity is derived. An LPIPS perceptual loss supplements the RGB objective.

Training proceeds in five stages. First, the generation branch is trained from scratch against a frozen understanding branch over three phases (180K, 100K, and 185K steps), escalating from 256² up to 4096². Second, 80K steps of unified mid-training mix 30% understanding data, 40% text-to-image, 20% editing, and 10% interleaved; the understanding loss is down-weighted to 0.1 so the harder generative objective dominates. Third, 10.5K steps of supervised fine-tuning on high-quality instruction data consolidates instruction following.

Fourth comes specialized reinforcement learning. Each expert uses its own sampler, learning rate, KL coefficient, and reward. The aesthetic expert interleaves preference and typography data routed to separate rewards (HPSv3++ for perceptual quality, PaddleOCR-based scoring for text); the OCR expert uses Precise sampling with GRPO-Guard for stability; the editing expert avoids stochastic SDE rollouts to prevent residual noise and trains with a five-dimension reward plus progressive sliding-window optimization; the infographic expert goes through its own mid-training, a DPO stage, and a final alternating OCR/aesthetic stage.

Fifth, on-policy distillation unifies the experts. Each sample is hard-routed to the frozen expert matching its capability. The student generates its own trajectory, and both student and expert evaluate velocity at the same state, timestep, and condition, with gradients stopped through the trajectory. The query timestep distribution shifts from Beta(2,5) to Beta(5,2) over training, moving emphasis from global structure at high noise to fine detail at low noise. Training runs 800 optimizer steps with 25,600 samples per domain while the understanding branch, the last three generation layers, and the output head stay frozen.

Why This Matters

Impact on research. The paper argues that a single compact visual representation can serve as the shared substrate for seeing, reasoning, and creating, eliminating parallel visual pathways and repeated conversion between encoder features and VAE latents. If that holds, it reframes multimodal unification as more than architectural tidiness — a capability acquired through understanding can reinforce generation, and vice versa. The specialize-then-unify recipe also offers a reusable template for combining competing objectives that resist joint optimization.

Real-world applications:

• Design and marketing tooling — generating infographics, posters, and multilingual marketing visuals with accurate embedded text.
• Photo and content editing — instruction-driven edits that change a specified region while provably preserving subject identity and everything else.
• Product and e-commerce imagery — composing products into new scenes from multiple reference images at high resolution.
• Document and data visualization — turning structured information into readable charts, diagrams, and teaching material.
• Accessibility and media workflows — interleaved text-image generation where the model alternates between describing and producing visual content.

Industry relevance. The encoder-free, VAE-free design removes two large pretrained components and the engineering overhead of keeping their representations aligned, which simplifies deployment and reduces the number of stages in a serving pipeline. Compact 32×32 tokenization keeps inference cost manageable at high resolution. Most importantly, the authors commit to open-sourcing training code for SFT, RL, and on-policy distillation, making the post-training recipe reproducible rather than a black box — valuable for teams that need to adapt unified models to domain-specific visual tasks.

Future Directions

• Extension beyond still images. The architecture uses native RoPE for unified spatiotemporal encoding and the training corpus already includes video content, but the paper scopes its results to images; temporal coherence at 4K is an open problem.

• Pushing past 4K and toward better high-resolution behavior. Resolution-aware conditioning is calibrated up to 4096×4096, and the paper notes that patch-wise factorization problems get worse as resolution grows, so further scaling will stress the spatial decoder.

• Reward design and RL stability. The pipeline depends on heterogeneous, partly external reward models (HPSv3++, PaddleOCR-based scoring, a VLM-based editing judge), and the authors freeze layers, use GRPO-Guard, and avoid SDE sampling in editing specifically to control instability. More principled reward modeling and unified sampling strategies remain open.

• Where native unification still trails modular systems. The paper positions native modelling as promising rather than settled, leaving open how far the gap to encoder-plus-VAE pipelines can be closed and how to benchmark unified models fairly across understanding, generation, editing, and interleaving at once.

Target Audience

Researchers working on unified multimodal models, diffusion and flow-matching generation, and RL post-training for generative systems will find the architectural and training contributions most directly useful. Practitioners building production image generation, editing, or design assistance pipelines benefit from the encoder-free/VAE-free design rationale and the concrete data and reward recipes. Readers interested in whether understanding capabilities transfer to visual creation will find the generalization result the most provocative part of the paper.

Authors’ abstract

We launch SenseNova-U1.5, an 8B-MoT native unified multimodal model that understands, reasons about, and generates visual content within an encoder-free and VAE-free architecture. We strengthen its visual interface through spatially coherent patch reconstruction and scale its training with carefully curated generation and editing data, improved task formulation, structural prompt enhancement, and native resolutions of up to 4K. For post-training, we optimize specialized experts for visual aesthetics, bilingual text rendering, infographic generation, and image editing, and consolidate their capabilities through multi-expert on-policy distillation. Across extensive evaluations, SenseNova-U1.5 largely advances image fidelity, text rendering, complex composition, multi-reference editing, and interleaved generation, while improving instruction following and preserving subject identity, geometry, and unmodified regions. Despite limited exposure to structured formats in its generation data, SenseNova-U1.5 generalizes effectively to long, complex, and structured visual instructions, further proving that multimodal understanding can transfer to visual planning and creation. Together, these findings position native unified modelling as a promising path towards systems that perceive, reason and create within a fully end-to-end framework. We will open-source training code, including supervised fine-tuning, reinforcement learning, and on-policy distillation.