MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

AI summary

Overview

Research area: Multimodal machine learning / computer vision — specifically any-to-any generative modeling across many modality types within a single decoder-only Transformer.

Technical level: Advanced. The paper assumes familiarity with autoregressive next-token prediction, flow matching, latent diffusion, mixture-of-transformers architectures, and standard vision benchmarks (depth estimation, surface normals, grounding, VQA, text-to-image).

Scope in one sentence: Modus is a single decoder-only model that extends a pretrained image–text foundation model (BAGEL-7B) to 15 modalities spanning text, RGB, geometry, structure, semantics, and learned feature representations, supporting arbitrary modality-to-modality generation without modality-specific heads, losses, or task pipelines.

What This Paper Is About

Existing any-to-any multimodal models are usually trained from scratch with encoder–decoder or diffusion backbones, which means they cannot inherit the priors of large pretrained decoder-only models and therefore pay higher training costs with limited scalability. The authors ask whether any-to-any generation across many diverse modalities can instead be realized inside a single fully decoder-only model that treats every modality symmetrically as both a possible input and a possible output. Their goal is to extend a pretrained decoder-only foundation model to many modalities with a simple training recipe while remaining competitive with task-specific specialists.

Key Contributions

1. Decoder-only any-to-any modeling. The authors show that any-to-any generation across diverse modalities can be achieved in a single, fully decoder-only model with no modality-specific heads, losses, or task pipelines — unlike prior any-to-any systems built on encoder–decoder or diffusion backbones trained from scratch.

2. A recipe for extending a pretrained model. Rather than training from scratch, they introduce uniform timestep sampling plus a staged curriculum that extends a pretrained decoder-only foundation model (BAGEL-7B) to any-to-any generation, inheriting its priors and scalability and yielding strong out-of-the-box performance at a fraction of the compute.

3. Diverse modalities at scale, openly released. Modus covers 15 modalities spanning geometry, structure, semantics, and learned representations, trained on the 29M-sample Modus-Dataset. The dataset and two model checkpoints are released openly.

4. New emergent capabilities. The unified design enables chained generation, cross-modal self-verification, and visual representation composition, which the paper studies experimentally.

Main Findings

• Competitive zero-shot performance from one model. In the system-level comparison (Table 1), Modus obtains MMMU 51.1, GenEval 0.81, DIODE 0.285, NYUv2 surface normal 19.92, RefCOCO val 54.5, and ImageNet 77.9 / 92.5. Single-task specialists in the same table include DeepSeek-VL2 (MMMU 51.1), FLUX.1-dev (GenEval 0.82), DepthAnything2 (DIODE 0.249), Marigold (NYUv2 16.40), GroundingDINO (RefCOCO val 50.4), and DINOv2 (ImageNet 82.1 / 93.9). Decoder-only comparators include Bagel (MMMU 53.2, GenEval 0.86), Janus-Pro (MMMU 41.0, GenEval 0.80), GPT-4o (MMMU 69.1, GenEval 0.84), and Kosmos-2 (RefCOCO val 52.3). Modus covers all these tasks within a single model.

• Timestep sampling controls modality confusion. Logit-normal timestep sampling — standard in diffusion generation models — frequently causes the model to generate the wrong target modality (for example, a surface normal map when depth is requested). Under identical 6B-token training budgets (Table 5), uniform sampling gives GenEval 0.81, NYUv2 depth 8.4, and NYUv2 surface normal 21.10, versus logit-normal's 0.77, 21.9, and 50.54, and mode sampling's 0.80, 8.6, and 22.47. The explanation given is that early timesteps determine the target modality distribution while later timesteps refine visual quality, and logit-normal sampling undersamples early steps.

• The three-stage curriculum progressively builds capability. In Table 6 (eight-modality ablation setting), Modus-Stage1 (30B tokens) reaches MMMU 51.4, GenEval 0.81, retrieval 78.8 / 92.8, with no NYUv2 or chained generation; Modus-Stage2 (20B tokens) adds NYUv2 6.5 and retrieval 77.9 / 92.5; Modus-Stage3 (15B tokens) reaches MMMU 50.3, GenEval 0.81, NYUv2 6.6, retrieval 77.2 / 92.2, and enables chained generation. Bagel is listed for reference at approximately 5T tokens.

• Chaining helps only when the intermediate supplies spatially aligned structure. On NYUv2 surface normals (Table 2), independent RGB-to-surface-normal gives 20.02; routing through depth gives 20.06; through DINOv2 global features gives 20.71; through canny edges improves to 19.87. The authors conclude that complementary pixel-aligned cues help, while redundant geometric cues (depth) or non-aligned semantic cues (DINOv2) do not.

• Cross-modal self-verification improves generation without an external verifier. Selecting the best of 4 sampled images per prompt using the model's own grounding confidence or VQA answer likelihood raises GenEval from 0.81 to 0.84 (Table 3); grounding alone gives 0.82.

• Both ViT and VAE conditioning are needed for faithful 2D outputs. In Table 4, ViT-only conditioning gives NYUv2 depth 15.1 and surface normal 35.30; VAE-only gives 6.9 and 19.96; combining ViT and VAE gives 6.5 and 19.92. ViT features preserve coarse semantic identity but distort fine-grained geometry; VAE features maintain local geometric consistency but lack semantic robustness. The paper notes the same failure mode has been reported for GPT-4o, though Modus's internals are not comparable.

• Training cost. Training takes approximately 5,664 GH200 GPU-hours across three stages (35 h, 31 h, and 22.5 h on 64 GPUs).

• Chaining has a modest latency cost. Chaining runs one generation per intermediate modality, adding decoding steps in exchange for more consistent outputs; the paper reports independent-versus-chained efficiency in Appendix Section C.7 and states the added latency stays modest.

Methodology in Plain English

Modus starts from BAGEL-7B, a pretrained decoder-only image–text model with a Mixture-of-Transformers design, and adds a second "expert" inside the same decoder.

Modalities are split into two groups. 1D sequential modalities — text, captions, bounding boxes for detection and grounding, and learned feature representations such as DINOv2, CLIP, and ImageBind — are handled by a 1D Expert that predicts the next discrete token with a cross-entropy loss. Bounding box coordinates are normalized to the range [0, 999] and written as x1 y1 x2 y2, using 1000 discrete tokens per coordinate; representation modalities use an MLP-based tokenizer with a codebook of size 8192. 2D spatial modalities — RGB, depth, surface normals, segmentation, and canny edges — are handled by a 2D Expert trained with flow matching in the latent space of a pretrained FLUX VAE, while a pretrained SigLIP-2 ViT encoder extracts semantic features from the same input. All modality tokens share one autoregressive sequence: attention is causal between modalities, causal within 1D modalities, and bidirectional within a 2D modality, so any token can condition all subsequent ones.

Training samples are aligned tuples: one image plus annotations for all its modalities. For each sample, one modality is the generation target and between 1 and 3 of the remaining modalities are randomly sampled as conditioning inputs, specified through instruction-following prompts. This is what produces arbitrary (input, target) pairs, including ones rarely covered by existing corpora such as depth to canny edge or canny edge to surface normal.

Two stabilization choices matter. First, uniform timestep sampling replaces the usual logit-normal schedule to prevent the model from drifting between modalities. Second, training proceeds in three stages: Stage 1 on 1D modalities (captions, grounding boxes, DINOv2 global features), Stage 2 adding 2D modalities (depth, surface normals, segmentation, canny edges), and Stage 3 increasing the number of conditioning modalities per sample to strengthen long-context and multi-condition behavior. At inference, conditioning modalities are encoded first to fill the KV cache, discrete tokens are predicted autoregressively, 2D latents follow a fixed set of flow-matching denoising steps with classifier-free guidance, and generated outputs can be fed back as conditioning for further steps.

The data comes from Modus-Dataset, a 29M-sample corpus built by extending BLIP-3o's image–caption corpus with aligned annotations from off-the-shelf experts: DepthAnything for depth, Marigold for surface normals, Grounded-SAM and SAM for segmentation, the Canny operator and SAM for edges, GLaMM for grounding boxes, ViTDet with an EVA-02 backbone for detection, and DINOv2, CLIP, and ImageBind for representational features.

Why This Matters

The paper argues that the architectural choice of training any-to-any models from scratch has been a limiting factor: encoder–decoder any-to-any architectures tend to do well on reconstruction but less well on open-ended generation, diffusion models excel at image generation but struggle with text understanding and compositional generation, and prior decoder-only models mostly stick to image and text. Modus shows that a pretrained decoder-only prior can be extended to many modalities with a relatively cheap recipe, and that a single model can be competitive with dedicated specialists while supporting capabilities those specialists do not have — chained generation, cross-modal self-verification, and non-standard mappings such as canny to depth.

Real-world applications implied by the supported modalities and capabilities:

• 3D content and scene pipelines, where RGB to depth, surface normals, and edge maps within one model can feed downstream reconstruction and asset creation.
• Robotics and embodied perception, where a single network can convert between geometry, semantics, and language instructions rather than stitching together separate task-specific models.
• Visual inspection and editing, using edge or layout intermediates to keep structure consistent across modalities, and using the model's own grounding or VQA output to score and select candidate renders.
• Scientific data integration, since the paper frames any-to-any modeling as relevant to domains such as genomics, ecology, and astronomy where diverse data sources must be related compositionally and reused.

Industry relevance: the release of the 29M-sample dataset, two checkpoints (Modus-15modality-14B-A7B and Modus-15modality-77B-A13B), and code lowers the barrier for teams that cannot train any-to-any models from scratch. The compute footprint reported — approximately 5,664 GH200 GPU-hours — is substantially smaller than pretraining a foundation model, which makes this style of adaptation practical for groups with modest clusters. The authors also include an image–text capability preservation analysis (Appendix Section C.3), which speaks to the practical concern of not degrading a pretrained model when extending it.

Future Directions

• Which intermediate modality to route through. The chaining study covers only surface normal estimation on NYUv2 with three candidate intermediates (canny, depth, DINOv2). Whether the same pattern — spatially aligned complementary cues help, redundant and non-aligned semantic cues do not — holds for other targets and longer chains is open.

• Scaling to more modality families and instantiations. The paper describes modalities as an open, extensible collection where each family can host multiple instantiations and new modalities can be added without architectural changes, but it does not characterize how performance scales as more modalities are added or how much modality confusion returns.

• Stronger self-verification. Cross-modal verification currently uses best-of-4 selection with grounding or VQA scores, improving GenEval from 0.81 to 0.84. Whether richer verification signals, larger candidate pools, or verification on non-image modalities can yield larger gains is unexplored.

• Quantitative evaluation beyond RGB- and text-centric benchmarks. The authors note that most existing benchmarks are RGB- or text-centric, and they report quantitative results for non-standard any-to-any and multi-condition mappings (canny to depth, surface normal to depth, RGB+canny to depth) only in Appendix Section C.4, leaving broad comparison for such mappings an open measurement problem.

Target Audience

This paper is most useful to multimodal and generative modeling researchers who already understand autoregressive transformers, flow matching, and latent diffusion; to engineers building unified perception-and-generation systems who need a concrete recipe (staged curriculum, uniform timestep sampling, unified tokenization) for extending a pretrained decoder-only model; and to practitioners in vision tasks such as depth, surface normals, segmentation, grounding, and retrieval who want one checkpoint instead of a stack of specialists. Readers looking for an introductory treatment of multimodal learning will find the paper dense, since it assumes fluency with Mixture-of-Transformers, classifier-free guidance, KV caching, and standard benchmark conventions.

Authors’ abstract

Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.