JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces

AI summary

Overview

Research area: Multimodal generative modeling — building a single model that both understands and generates images and text.

Technical level: Advanced. The paper assumes familiarity with diffusion and flow matching models, transformer architectures, classifier-free guidance, and standard vision-language benchmarks such as GenEval, DPG-Bench, VQAv2, and CIDEr.

Scope: The paper proposes JUICE, a single flow matching Transformer trained from scratch that generates both images and text as continuous states inside frozen pretrained encoder spaces, and it reports controlled pretraining studies plus 2.1B-parameter results on generation, question answering, and joint planning tasks.

What This Paper Is About

Most "unified" multimodal models are not really unified: a pretrained language model writes text, a separate diffusion head draws images, and any output containing both is produced one modality after the other, so the first part cannot react to the second. The authors ask whether both modalities can instead be produced by one shared process, learned jointly from scratch, by modeling the joint distribution directly in the continuous representation spaces of frozen pretrained encoders. The goal is a model where the task — image generation, captioning, visual question answering, or joint image-and-text generation — is determined simply by which modalities are given clean and which are generated from noise.

Key Contributions

1. Joint modeling in pretrained representation spaces. JUICE models the joint distribution of images and text as continuous states of frozen pretrained encoders, using one flow matching Transformer (MM-RiT) trained from scratch. Giving each modality its own noise level turns every task into a path through a 2D time square, with no fixed generation order.

2. Systematic design studies. Section 4 reports controlled pretraining runs at a 668M backbone scale, varying one design choice at a time: how text is generated (continuous states versus tokens), which denoising tasks are trained, and which encoder spaces are read and written.

3. A scaling study from 179M to 2.1B parameters. The paper tracks how encoder preferences change with model size, finding that rankings can flip — DINO leads image generation at 179M while SigLIP2 leads at 2.1B.

4. Scaling and joint reasoning. Two 2.1B-parameter models, pairing Flan-T5 with SigLIP2 or with DINO, use one set of weights for compositional image generation, generation requiring world knowledge, and visual question answering, and coupled sampling lets a drawn route and its written moves agree without a fixed generation order.

Main Findings

• Continuous text does not damage image generation. At a shared guidance scale, JUICE scores 48.41 GenEval and 78.85 DPG-Bench at 668M, versus 47.86 and 77.50 for a model trained only for text-to-image generation. All three token-text baselines fall below that reference: Transfusion-style reaches 44.23 GenEval (46.07 when given the same frozen T5 condition), D-DiT-style 34.50 (37.58 with T5), and MMaDA-style 41.61 (43.35 with T5).

• But continuous text costs exact wording. JUICE reaches 80.20 CLIPScore, essentially matching autoregressive text at 80.37, while its CIDEr of 88.80 is just over half the autoregressive figure of 160.52 and below masked text diffusion at 115.22. The gap is attributed mainly to the generated states rather than the parallel caption decoder, which recovers 99.2% of tokens from clean T5 states.

• Train where you sample. Training only on the edges of the time square (Multi-task, T2I plus I2T) reaches 36.30 GenEval; training only in the interior (Joint only) reaches 31.33; the default Mixing recipe, at 55.6% joint and 22.2% each conditional, reaches 48.41. The ordering holds when guidance is tuned per recipe: Mixing 48.41, Multi-task 44.29, Joint only 31.33.

• Joint denoising gives more than extra supervision. Controls that match the number of supervised targets or block cross-modal attention both fall short of Mixing, by 2.51 and 2.92 GenEval points respectively. Multi-task trained for 77.8k updates to match target count closes the CIDEr gap (89.31 versus 88.80) but not the GenEval gap (43.16 versus 48.41).

• Semantic spaces beat reconstruction-oriented ones. SigLIP2 and DINO reach 48.41 and 49.07 GenEval, against 35.42 for FLUX VAE latents and 24.48 for raw pixels — roughly the reverse of reconstruction quality, since the FLUX VAE has the best rFID among the learned spaces (0.288 versus 0.435 for SigLIP2).

• Language-aligned image encoders help captioning. SigLIP2 leads DINO by 20.0 CIDEr and 8.5 CLIPScore on captioning, even though the two are close on image generation at this scale.

• Flan-T5 is the best text space. T5 leads on all four metrics; Qwen3-Embedding comes close on CIDEr (88.50 versus 88.80) but trails by 5.3 GenEval points, and SigLIP2 text features give the lowest GenEval at 33.08. Alignment between text and image spaces does not make a better generation target here.

• Encoder rankings can flip with scale. Every metric improves from 179M to 2.1B parameters, but DINO leads image generation at 179M while SigLIP2 leads at 2.1B, and SigLIP2 writes better captions at every size.

• The final 2.1B models are competitive without language-model initialization. At 224×224, the two models reach 0.87 and 0.88 GenEval; the SigLIP2 variant reaches 85.22 DPG-Bench, 0.45 WISE, and 73.1 VQAv2.

• Coupled sampling resolves ambiguity between modalities. On mazes with several valid solutions, joint success is 54.6% with independent sampling, 95.2% with the image generated first, 85.3% with text first, and 93.7% with coupled sampling — which requires no ordering decision. On visual spatial planning, JUICE solves 99.8% of maps at training sizes, and 94% and 69% at two larger sizes unseen in training.

Methodology in Plain English

The authors start from the observation that images and text can both be turned into continuous vectors by frozen pretrained encoders — SigLIP2 or DINO for images, Flan-T5 for text — and that these spaces already contain knowledge learned at scale. Rather than attaching a diffusion head to a language model, they train one transformer, MM-RiT, to reverse noise in both spaces at once using flow matching.

The key trick is to give each modality its own noise level. A value of 1 means clean, anything below 1 means still being generated. This makes each task a path through a 2D grid: image generation runs along the edge where text is clean, captioning runs along the edge where the image is clean, question answering keeps the image and question clean while generating the answer tokens, and joint generation cuts diagonally through the middle where both are noisy simultaneously.

During training, each example is randomly assigned to one of three denoising tasks — joint, text-to-image, or image-to-text — mixed at fixed ratios, with the loss computed only on corrupted target positions. Separate noise scales and schedules per modality account for their different dimensionality and statistics, and the image schedule is shifted toward noisier states. Inside the transformer, each block keeps separate Q/K/V projections and feedforward networks per modality, concatenating the sequences for one bidirectional joint attention operation. Lightweight fixed decoders turn generated states back into pixels and tokens, with text decoded in parallel rather than autoregressively.

At inference, guidance is applied in velocity space using an uninformative reference input — the caption "a photo." for text conditions, or a fully noised image for image conditions — rather than a trained null token. The authors then run controlled pretraining experiments at 668M parameters, changing one variable at a time, before training the two 2.1B models.

Why This Matters

Impact on research. The paper provides a controlled, apples-to-apples comparison of design choices that unified multimodal papers usually leave unexamined: whether text should be continuous or discrete, how training tasks should be mixed, and which encoder spaces to use. Its finding that rankings flip between 179M and 2.1B parameters is a caution for anyone drawing conclusions from small-scale ablations, and its "train where you sample" result gives a simple principle for anyone building joint generative models.

Real-world applications:

• Robotics and navigation instructions — a model that draws a route and writes the matching step-by-step directions at the same time, without one modality being fixed before the other.
• Diagram, map, and slide generation — producing an image and its accompanying text as a mutually consistent pair rather than one conditioned on the other.
• Visual question answering and captioning — assistive tools for describing images or answering questions about them with a single model.
• Layout and document authoring — generating figures alongside explanatory prose where the two must agree.

Industry relevance. Because text is decoded in parallel rather than autoregressively, text generation is much faster than in language-model-based unified systems, and because no language model initialization or finetuning of encoders is required, the recipe avoids depending on a large pretrained LLM. The 2.1B results suggest a competitive unified model can be trained from scratch at moderate scale.

Future Directions

• Closing the caption-wording gap. Continuous text matches autoregressive CLIPScore but loses roughly half its CIDEr; whether better continuous text decoding or a different text space can recover exact wording is left open.

• Extending the scaling study. The paper tracks 179M to 2.1B parameters and reports that encoder rankings flip; whether the same flips occur beyond 2.1B, and whether the task-mixture conclusions survive, is not established.

• Generalizing joint reasoning beyond planning. Coupled sampling is demonstrated on mazes and visual spatial planning; whether it helps on other tasks where multiple valid image-text pairs exist is untested.

• Resolving remaining design questions. The provided excerpt covers the framework, task-mixture, and representation studies in detail but is truncated during the scaling section, so the full scaling results and the detailed 2.1B training configurations are not reported in the content available here.

Target Audience

Researchers and engineers working on multimodal generative models, diffusion and flow matching, or unified architectures will get the most from this paper, particularly those deciding how to represent text in a generative system or how to mix training tasks for joint modeling. The controlled ablations and the explicit reporting of negative results (such as the captioning CIDEr gap) also make it useful for graduate students studying empirical methodology in multimodal machine learning. Readers without a background in diffusion models and transformer architectures will find the method section demanding, though the framing of tasks as paths through a 2D time square is accessible.

Authors’ abstract

Most unified multimodal models treat images and text asymmetrically: a pretrained language model generates text, an attached diffusion head generates images, and outputs that contain both are produced in a fixed order, so the first output cannot adapt to the second. We ask whether both modalities can instead be generated by one process that is trained on them jointly from scratch. Pretrained encoder spaces hold both modalities in one continuous form and carry knowledge learned at scale, and recent work finds them effective as generative targets for vision and language. We therefore model the joint distribution directly in these spaces. We introduce JUICE (Joint Unified multimodal modeling In Continuous Encoder spaces), a single flow matching Transformer over frozen image and text representations, in which each modality has its own noise level. Which modalities are clean and which are generated then decides the task: image generation, captioning, visual question answering, or joint generation. Controlled pretraining runs test each design choice. At equal guidance, generating text as continuous states leaves image generation intact (48.41 GenEval, against 47.86 for a model trained only for text-to-image generation), while captions lose some exact wording. Training on joint and conditional denoising together beats either alone, and semantic encoder spaces beat VAE latents and raw pixels as targets. At 2.1B parameters, with no language model initialization and at 224x224, JUICE reaches up to 0.88 on GenEval and 73.1 on VQAv2. By sampling both modalities from the joint distribution (coupled sampling), it can also draw a route on a map while writing out the matching moves. On mazes with several valid solutions, joint success rises from 54.6% with independent sampling to 93.7%, and no modality has to go first. On visual spatial planning, JUICE solves 99.8% of maps at its training sizes.