NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference

AI summary

Overview

Research area: Information retrieval (cs.IR), specifically multimodal and multilingual representation learning and visual document retrieval.

Technical level: Advanced. The paper assumes familiarity with Transformer encoder architectures, masked-language/diffusion objectives, contrastive retrieval, and late-interaction (ColBERT-style) scoring.

Scope: The paper introduces NeoMME, a family of 260M- and 800M-parameter single-tower bidirectional encoders that process text tokens and raw 32×32 image patches in one shared Transformer trained from scratch, and demonstrates them as visual document retrievers (NeoMME-Retriever) on the ViDoRe benchmark family.

What This Paper Is About

Most multimodal document retrievers are built by repurposing vision–language models: a separately pretrained vision encoder (such as SigLIP2) is bolted onto a language model, and the whole stack is then used as an encoder for a task that never generates text. That approach carries the parameter count and compute cost of a generative VLM. The paper's goal is to build a native bidirectional multimodal encoder from random initialization — no pretrained vision tower, no causal decoder — that handles multilingual text and raw image patches in a single shared Transformer, and to show it is competitive and more efficient for visual document retrieval.

Key Contributions

1. NeoMME, a multimodal-native multilingual foundation encoder. A tokenizer plus 260M and 800M bidirectional single-tower Transformer backbones trained from scratch on raw image patches, using image-conditioned masked discrete-diffusion pretraining over multilingual text, code, mathematics, natural images, and document images, with long-context dynamic-resolution encoding. It is released in the Hugging Face Transformers library and under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.

2. NeoMME-Retriever, an efficient visual document retrieval embedder. A single backbone forward pass produces both a dense pooled representation and a late-interaction multi-vector representation, making it compatible with Sentence Transformers for dense and multi-vector retrieval.

3. Late-Interaction Kernels (LIK). A suite of fused MaxSim kernels reported to reduce peak VRAM and runtime for late-interaction inference and training.

4. Efficiency and compression study. The paper characterizes input resolution, representation storage and compression, indexing throughput, and query-encoding latency.

Main Findings

• ViDoRe v3 results: NeoMME-Retriever 260M reaches 0.523 nDCG@10 and NeoMME-Retriever 800M reaches 0.556. The 260M model is reported to outperform all evaluated models strictly below 800M parameters.

• Throughput: At a matched 2048 × 2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages at 51.3 pages per second, about 1.97× the throughput of ColModernVBERT (described in the abstract as about 2×).

• Embedding compression: Hierarchical token pooling at factor 8 combined with asymmetric quantization (int8 queries, binary documents) compresses a NeoMME-260M embedding from roughly 1.5 MB per ViDoRe v3 document to 6 kB — a 255× reduction — while retaining more than 95% of baseline nDCG@10.

• Tokenizer efficiency: Aggregated over 14 target languages from FLORES-200 devtest, the NeoMME tokenizer emits 44.4% fewer tokens than ModernBERT, 39.4% fewer than LFM2.5-Encoder-230M, 6.3% fewer than mmBERT-base, and 16.9% fewer than EuroBERT-210m. The paper also notes weaker coverage outside the original target set when analyzed across all 204 languages in FLORES-200.

• Image evidence is used, not ignored: Over the final 50 probe events, both models showed positive mean image gain at every tested corruption rate (30%, 60%, 90% masking). Gain rises as text context is removed, reaching 38.4 percentage points for NeoMME-260M and 40.5 percentage points for NeoMME-800M at 90% masking.

• Training dynamics: The NeoMME-260M pretraining loss drops into a lower regime around step 36,000; NeoMME-800M shows a visible change around step 25,000. A final loss decline coincides with the scheduled warmup–stable–decay learning-rate anneal to 1% of peak.

• Generation probe: The step-500,000 pretrained NeoMME-260M was tested on unprompted image-conditioned text generation without captioning or OCR fine-tuning, using uniform and blockwise LLaDA-style decoding with cosine remaining-mask schedules and MaskGIT confidence ranking. The paper states these selected outputs show image use but do not measure average generation quality.

• Pareto-optimality claim: NeoMME-Retriever is reported as Pareto-optimal on ViDoRe v1, v2, and v3.

• Context length: Both models support 16,384 tokens, stated as enough to encode up to two standard 3,840 × 2,160 4K UHD images after 32-pixel patching.

Methodology in Plain English

One tower, not two. Text tokens enter through an ALBERT-style factorized embedding (a 256-dimensional lookup projected to model width). Images are converted to RGB and cut into non-overlapping 32×32 patches; each patch's 3,072 values are normalized and pushed through a 2-layer MLP trained from scratch. After these modality-specific input layers, text and patches share every Transformer layer. Structural tokens mark input, grid, and patch-row boundaries, and segment offsets stop attention from crossing packed inputs.

Positions and resolution. RoPE is extended to two axes so image patches get row and column coordinates while text keeps the usual diagonal (i, i) ordering. Images keep their aspect ratio, and the pipeline randomly samples a longest-side cap between 1,024 and 2,048 pixels per example during pretraining and fine-tuning.

Attention layout. Most layers use symmetric sliding-window attention; every sixth layer and the final layer use global attention. Among sliding-window layers, the half-window alternates between 256 and 1,024 tokens. Both layer types use grouped-query attention. Queries and keys are independently RMS-normalized before RoPE. Blocks use RMS pre-normalization and a squared-ReLU MLP, with branch scaling of (2L)^(-1/2).

The pretraining objective. The model is trained as a discrete masked-diffusion denoiser over text. Each document segment draws a corruption rate: text-only segments from U(0,1), multimodal segments from U(0.30, 1). Selected text positions are always replaced by the mask token (unlike BERT's 80-10-10 mixture), and loss is computed only at those positions, with inverse-rate weighting capped so rare low-corruption samples don't dominate. Image patches stay visible and condition the prediction. No masked image modeling or pixel-prediction head is used.

Scale and systems. The text stream draws from 14 datasets in 58 configurations; 55% of packed input tokens come from text-only data and 45% from multimodal data, with a planned total of about 524B packed input tokens. Runs used AWS p5.48xlarge instances with EFA networking, managed with SkyPilot: 2 nodes/16 H100s for the 260M model and 4 nodes/32 H100s for the 800M model, each processing 1,048,576 packed input tokens per global step for 500,000 steps. Optimizers were split by parameter type: a custom MasterAdamW for embedding tables and one-dimensional parameters, and NorMuon for matrix-valued parameters.

Downstream. NeoMME is fine-tuned with jointly trained dense and late-interaction heads (late-interaction representation width r = 128) to form NeoMME-Retriever.

Why This Matters

Impact on research. The paper tests the proposition that a single bidirectional Transformer, trained from random initialization on raw patches with an image-conditioned masked-diffusion objective, can serve as a multimodal retrieval backbone without inheriting a pretrained vision tower or a causal decoder. If it holds at larger scale, it reframes visual document retrieval as an encoder problem rather than a VLM-repurposing problem, and it unifies the model lifecycle — the same backbone can be pretrained, fine-tuned, parallelized, and served across modalities rather than requiring separate towers with asymmetric architectures and forward passes. The paper also positions itself against the Ettin finding that native masked encoders remain stronger than causal decoders and decoder-to-encoder adaptations on classification and retrieval.

Real-world applications:

• Visual retrieval-augmented generation (Visual RAG): retrieving top-k page images to feed a VLM so it can answer from layout, tables, and figures that text extraction may lose.
• Enterprise and scientific document search over PDF corpora, where OCR-based pipelines lose layout information.
• Large-scale multilingual document indexing, given the 131,072-entry tokenizer and 16,384-token context.
• Cost-sensitive deployment where the 255× embedding compression and reported throughput matter for indexing large corpora.

Industry relevance. The efficiency framing is the practical pitch: a 260M model reported to beat all evaluated sub-800M models on ViDoRe v3, roughly 2× the throughput of ColModernVBERT at matched input size, and embedding storage cut from about 1.5 MB to 6 kB per document with over 95% of retrieval quality retained. The release is permissively licensed (Apache 2.0) and integrated into Hugging Face Transformers, which lowers the barrier to adoption.

Future Directions

• Scaling beyond 800M. The 800M configuration scales mainly through width, keeping 20 layers and 4 global-attention layers. Whether the single-tower recipe keeps its advantage at larger sizes is left open.
• Generation quality. The image-conditioned generation probe was qualitative and explicit that it does not measure average generation quality. Whether the pretrained backbone is a competitive generative model, rather than only an encoder, is unmeasured.
• Tokenizer coverage. The tokenizer is efficient on the 14 target FLORES-200 languages but the paper reports weaker coverage outside that set across all 204 FLORES-200 languages, leaving multilingual breadth as an open question.
• Retrieval evaluation detail. The provided text truncates during the retrieval section's problem formulation, so the full ViDoRe v1/v2/v3 comparison tables, text-retrieval results, and latency breakdowns referenced by the paper are not reproduced here.

Target Audience

Researchers and engineers working on multimodal representation learning, visual document retrieval, and Visual RAG, as well as practitioners who need to index large multilingual PDF or image corpora under realistic compute and storage budgets. It is also relevant to readers interested in masked-diffusion objectives applied to non-generative representation tasks and in tower-free multimodal architecture design. Readers without a background in Transformer architecture, retrieval objectives, or late-interaction scoring will find the methodology sections dense.

Authors’ abstract

Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.