MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding

AI summary

Overview

Research area: Computer vision / multimodal representation learning, applied to e-commerce product understanding.

Technical level: Advanced. The paper assumes familiarity with contrastive learning (InfoNCE-style objectives), Mixture-of-Experts routing, Multimodal Large Language Model (MLLM) backbones, and standard retrieval metrics such as Recall@k.

One-sentence scope: The paper proposes MOON2.0, a dynamic modality-balanced multimodal representation learning framework built on an MLLM, and releases MBE2.0, a co-augmented e-commerce benchmark, reporting state-of-the-art zero-shot results on MBE2.0, M5Product, and Fashion200K.

What This Paper Is About

E-commerce systems need a single, task-agnostic representation of a product that works across retrieval, recommendation, classification, and attribute prediction. Existing methods build this from product images and titles, but they suffer from three problems: training on a fixed mixture of image-only, text-only, and multimodal queries causes "modality imbalance" that hurts downstream tasks unevenly; supervision only models relationships between different products and ignores the image-text alignment available inside a single product; and raw e-commerce data contains noisy, redundant text and cluttered or viewpoint-poor images. MOON2.0 aims to fix all three at once — in the architecture, the training objective, and the data pipeline.

Key Contributions

1. MOON2.0 framework. The authors describe it as the first dynamic modality-balanced multimodal representation learning framework for e-commerce product understanding. It is built on an MLLM and supports Multimodal Joint Learning, jointly optimizing contrastive objectives over image-only, text-only, and multimodal queries against multimodal targets in a single end-to-end training stage, to mitigate modality imbalance.

2. Four technical components. A Modality-driven Mixture-of-Experts (MoE) that routes experts by input modality composition; Dual-level Alignment combining inter-product contrastive alignment with intra-product image-text alignment; MLLM-based Image-text Co-augmentation for textual enrichment and multi-granularity visual expansion; and Dynamic Sample Filtering to suppress noisy triplets during contrastive training.

3. The MBE2.0 benchmark. A co-augmented multimodal representation benchmark for e-commerce with 6.4 million real-world e-commerce samples, containing a co-augmented multimodal training set and a test set supporting product retrieval, classification, and attribute prediction. Released at https://huggingface.co/datasets/ZHNie/MBE2.0.

4. Empirical validation. State-of-the-art zero-shot results on MBE2.0 and public datasets (M5Product, Fashion200K), plus attention-based heatmap visualization offered as qualitative evidence of improved image-text alignment.

Main Findings

• MBE2.0 retrieval (Table 1). MOON2.0 reaches R@1 / R@5 / R@10 of 27.34 / 56.80 / 63.09 for text-to-multimodal, 41.07 / 87.31 / 91.08 for image-to-multimodal, and 43.34 / 89.31 / 94.21 for multimodal-to-multimodal retrieval. The strongest baseline on these axes, MOON, reports 16.94 / 36.54 / 43.24, 32.18 / 68.78 / 78.11, and 35.18 / 68.06 / 80.78 respectively.

• Largest gains on non-traditional cross-modal tasks. MOON2.0 reports 25.05 / 61.99 / 73.12 for text-to-image and 20.47 / 52.59 / 64.91 for image-to-text retrieval on MBE2.0, versus 17.79 / 35.56 / 44.02 and 12.56 / 26.69 / 36.65 for MOON. The authors attribute this to Dual-level Alignment leveraging both inter-product and intra-product image-text semantics.

• Classification and attribute prediction on MBE2.0. MOON2.0 reports 68.08 accuracy, 63.88 precision, 73.60 recall, 65.68 F1 for product classification, and 84.29 accuracy, 82.07 precision, 82.99 recall, 79.39 F1 for attribute prediction. For classification this exceeds BGE-VL-Large (59.36), GME (64.92), and MOON (59.70); for attribute prediction it exceeds GME (70.76) and MM-Embed (63.98).

• M5Product (Table 2). MOON2.0 reports 15.27 / 25.69 / 38.45 for text-to-multimodal, 11.28 / 24.43 / 32.37 for image-to-multimodal, and 15.21 / 27.37 / 39.27 for multimodal-to-multimodal retrieval, with 95.50 classification accuracy. MOON reports 10.14 / 20.33 / 33.31, 9.11 / 21.59 / 26.53, 14.27 / 25.05 / 32.13, and 73.12 accuracy on the same benchmark.

• Fashion200K (Table 3). MOON2.0 reports 13.05 / 25.25 / 31.39 for text-to-image and 13.10 / 23.16 / 27.09 for image-to-text retrieval, with 66.44 accuracy, 68.90 precision, 69.55 recall, and 64.21 F1 for classification. GME reports the best R@10 on text-to-image (26.06) and R@1 on image-to-text (10.31) among baselines shown, while MOON reports 10.82 / 22.89 / 27.73 and 11.71 / 20.02 / 25.09.

• Ablation: Dual-level Alignment matters most (Table 4, R@10 / Acc). Removing it drops text-to-multimodal R@10 from 63.09 to 37.99, image-to-multimodal from 91.08 to 65.72, multimodal-to-multimodal from 94.21 to 67.45, text-to-image from 73.12 to 31.45, image-to-text from 64.91 to 23.35, classification accuracy from 68.08 to 57.12, and attribute accuracy from 84.29 to 67.24.

• Ablation: MoE removal is the second-largest drop. Without the Modality-driven MoE, text-to-multimodal R@10 falls to 51.29, image-to-multimodal to 74.59, multimodal-to-multimodal to 78.45, text-to-image to 62.16, image-to-text to 56.21, classification accuracy to 62.55, and attribute accuracy to 75.62.

• Ablation: augmentation and filtering give more modest declines. Removing co-augmentation reduces text-to-multimodal R@10 to 59.69 and classification accuracy to 66.21; removing Dynamic Sample Filtering reduces image-to-multimodal R@10 to 83.40 and multimodal-to-multimodal R@10 to 80.00. Attribute accuracy remains at 84.04 without filtering, close to the full model's 84.29.

• Heatmap evidence. Visualization is reported to shift attention from non-critical regions such as "high quality" and "women" toward key attributes like "knitted cardigan", "polo-neck", and "Teddybear", and toward the brand name "Coshehkg".

Methodology in Plain English

Base setup. The model starts from an in-house generative MLLM for e-commerce. A training example is a triplet: a query, a positive item, and a negative item. Each element is instantiated in three input forms — multimodal (image plus text), image-only (paired with an instructional prompt), and text-only. Titles and enriched titles are tokenized, images and their augmentations pass through a vision encoder and projector into visual tokens, and the LLM processes them together. Hidden states from the last layer are mean-pooled into a single representation vector.

Handling imbalanced modalities. Rather than training with a fixed ratio of image-only, text-only, and multimodal queries — MOON uses a 12:3:2 mix — the authors replace the LLM's feed-forward layers with a Mixture-of-Experts whose gating network produces token-level expert weights. On top of that, they add a learnable "Dual-alignment Matrix" that expresses each expert's preference for each alignment objective (for example, matching an image-only query to a multimodal positive). An objective-specific weight aggregates token routing and expert preferences, and an entropy-based sparsity loss pushes each expert toward specializing in a narrow set of alignment objectives. A standard load-balancing term is also kept.

Two levels of alignment. Inter-product alignment is a contrastive objective pulling a query representation (text-only, image-only, or multimodal) toward the multimodal representation of its positive item and away from negatives. Intra-product alignment adds a finer objective that pulls an image and its paired text together within the same product and pushes apart text drawn from the unrelated item of the triplet. The total loss sums the two alignment terms plus weighted MoE auxiliary and sparsity terms.

Improving the data. Enriched titles are generated by an MLLM that combines the original title, the product description, extracted entity candidates, and the product image. Visual expansion is two-stage: first the MLLM edits the image into a standardized main-subject image, then it synthesizes several semantically consistent variants that vary background, viewpoint, and detail using prompts. CLIP is used to score image-title consistency and filter low-quality samples. During training, Dynamic Sample Filtering computes a reliability score per triplet from the margin between the positive and negative similarities, with the reliability threshold fixed at 0.6 and a decaying margin; triplets scoring below threshold are down-weighted in the loss.

Benchmark construction. MBE2.0 is built from Taobao user logs covering January 1, 2023 to June 30, 2025. Purchases following a query are positives; skipped exposures with low relevance scores are negatives. Image-query and text-query logs are joined on the positive item so that a text query and its image query form a multimodal query. Data is anonymized, keeping only visual and textual product content. The result contains 5,751,594 training samples and 636,241 test samples. Co-augmentation is applied only to training data; the test set keeps original product content with category labels and attribute annotations, so evaluation is a fair zero-shot assessment.

Training cost. Single-stage supervised finetuning with a learning rate of 1×10⁻⁵ and a cosine scheduler, approximately 18 hours on 64 NVIDIA A100 GPUs with a batch size of 4 per GPU. All experiments are zero-shot. Baselines include SigLIP2, BGE-VL-Large, FashionCLIP, InternVL3-2B, Qwen2.5-VL-3B, GME, MM-Embed, CASLIE-S, and MOON; dual-flow baselines use ℓ2-normalized embeddings from separate encoders, and generative MLLM baselines use mean-pooling over last-layer hidden states with a unified prompt.

Why This Matters

The paper argues that modality imbalance, weak intra-product supervision, and noisy data are the specific bottlenecks holding back e-commerce product representations — and that fixing them improves zero-shot generalization across retrieval, classification, and attribute prediction simultaneously. That matters because task-agnostic representations are what let a platform avoid training a separate model per downstream task.

Real-world applications (drawn from tasks evaluated in the paper):

• Product retrieval, including image-to-multimodal, text-to-multimodal, multimodal-to-multimodal, and the cross-modal text-to-image and image-to-text directions.
• Product recommendation, which the introduction identifies as a core product understanding task.
• Product classification, evaluated on MBE2.0, M5Product, and Fashion200K.
• Attribute prediction, evaluated on MBE2.0.

Industry relevance. The work comes from Alibaba Group and is built on data from Taobao, so the setup — logs, SKU and creative images paired with one shared title, noisy descriptions — reflects an operational e-commerce setting rather than a curated academic one. Releasing MBE2.0 as a public benchmark gives other groups a way to compare e-commerce representations on the same footing.

Future Directions

• Generalizing beyond e-commerce. The conclusion states that the architecture and training paradigm suggest promising directions for broader applications, and that these could advance general-purpose multimodal representation learning while maintaining efficiency and robustness.

• Extending the modality-balanced training to other modality mixes. Multimodal Joint Learning is demonstrated over image-only, text-only, and multimodal queries; whether the Modality-driven MoE and dual-alignment matrix transfer to other compositions is not evaluated in the provided content.

• Sustaining expert specialization at larger scale. The sparsity loss forces experts toward peaked preference distributions over alignment objectives; how that specialization behaves as the number of experts or objectives grows is not reported.

• Scalable data quality control. Co-augmentation uses an MLLM plus a CLIP consistency filter, and Dynamic Sample Filtering down-weights unreliable triplets with a fixed threshold of 0.6. Whether these remain sufficient as raw catalogue noise and scale grow is left open.

Target Audience

Researchers and engineers working on multimodal representation learning and retrieval, particularly those building task-agnostic embeddings for large product catalogues. It is also relevant to practitioners who need a single embedding model for retrieval, recommendation, classification, and attribute prediction, and to anyone looking for a public e-commerce benchmark (MBE2.0) with 5,751,594 training samples and 636,241 test samples. Readers without background in contrastive learning or Mixture-of-Experts routing will find the loss formulations and routing equations in Section 3 demanding.

Authors’ abstract

Recent Multimodal Large Language Models (MLLMs) have significantly advanced e-commerce product understanding. However, they still face three challenges: (i) the modality imbalance induced by modality mixed training; (ii) underutilization of the intrinsic alignment relationships among visual and textual information within a product; and (iii) limited handling of noise in e-commerce multimodal data. To address these, we propose MOON2.0, a dynamic modality-balanced MultimOdal representation learning framework for e-commerce prOduct uNderstanding. It comprises: (1) a Modality-driven Mixture-of-Experts (MoE) that adaptively processes input samples by their modality composition, enabling Multimodal Joint Learning to mitigate the modality imbalance; (2) a Dual-level Alignment method to better leverage semantic alignment properties inside individual products; and (3) an MLLM-based Image-text Co-augmentation strategy that integrates textual enrichment with visual expansion, coupled with Dynamic Sample Filtering to improve training data quality. We further release MBE2.0, a co-augmented Multimodal representation Benchmark for E-commerce representation learning and evaluation at https://huggingface.co/datasets/ZHNie/MBE2.0. Experiments show that MOON2.0 delivers state-of-the-art zero-shot performance on MBE2.0 and multiple public datasets. Furthermore, attention-based heatmap visualization provides qualitative evidence of improved multimodal alignment of MOON2.0.