UG-UMRE: Uncertainty-Guided Modality Augmentation and Distributional Calibration for Unified Multimodal Relation Extraction

AI summary

Overview

• Research area: Multimodal relation extraction (MRE) — specifically Unified Multimodal Relation Extraction (UMRE), which combines computer vision and natural language processing to identify relations between textual entities and visual objects. The paper is listed under Computer Vision (arXiv:2608.04949v1, cs.CV) and was accepted to the 34th ACM International Conference on Multimedia (MM '26), Rio de Janeiro, November 10–14, 2026.
• Technical level: Advanced. The method relies on the Variational Information Bottleneck, Gaussian posterior modeling, bidirectional KL divergence, contrastive learning with InfoNCE, hierarchical cross-modal attention, and Mixture-of-Experts routing. Readers need prior familiarity with variational inference and multimodal transformers to follow the equations, though the paper's core intuition (denoise each modality, then calibrate them, then fuse) is graspable without them.
• One-sentence scope: The paper proposes UG-UMRE, an uncertainty-guided network that models unimodal features as Gaussian distributions to filter aleatoric noise (UDUA) and globally calibrates cross-modal distributions before fusion (JAUA), evaluated on the UMRE, MORE, and MNRE benchmarks.

What This Paper Is About

Existing multimodal relation extraction systems feed raw, uncalibrated text and image features directly into cross-modal fusion networks, so noise inside each modality propagates and misaligns the two feature spaces. UG-UMRE addresses both problems at once by treating each feature as a probability distribution whose variance explicitly quantifies how uncertain (noisy or ambiguous) that sample is, then using that uncertainty to denoise each modality separately before synchronizing the text and visual distributions. The goal is a plug-in module set that improves any backbone multimodal relation extractor without changing its inputs, outputs, or core parameters.

Key Contributions

1. The UG-UMRE network itself. The authors state it is the first work to integrate uncertainty modeling into the UMRE task, framing modal noise and distributional heterogeneity as a probabilistic problem and adopting a "denoising first, calibrating second, fusing last" pipeline.
2. UDUA (Uncertainty-Driven Unimodal Augmentation). A module built on the Variational Information Bottleneck that represents each sample as a multivariate Gaussian — the mean as stable core semantics, the variance as aleatoric uncertainty — and uses reparameterization-based, uncertainty-aware self-supervised contrastive learning to pull semantic variants toward the stable mean while filtering noise.
3. JAUA (Joint Aleatoric Uncertainty Alignment). A global, distribution-level pre-calibration module that applies a symmetric (bidirectional) KL divergence between the text and visual Gaussian posteriors, jointly penalizing disagreements in both semantic centers and uncertainty scales, to build a shared latent space before fine-grained interaction.
4. Demonstrated plug-and-play behavior and efficiency. The modules are added to two different backbones (REMOTE and FocalMRE) with different input configurations and produce consistent gains, and the paper quantifies the added FLOPs, parameters, memory, and latency.

Main Findings

• State-of-the-art on all three benchmarks. Integrated into the previous SOTA model REMOTE, UG-UMRE reaches F1 scores of 69.98% on UMRE, 66.76% on MORE, and 89.59% on MNRE, improving over original REMOTE by 2.34%, 3.95%, and 2.25% F1 respectively (REMOTE baselines: 67.64%, 62.81%, 87.70%).
• General multimodal LLMs perform poorly on UMRE. Qwen2-VL-7B reaches only 8.95% F1 on UMRE, Qwen2.5-VL-7B 12.19%, and Llama-3.2-11B-Vision 8.28%, versus 69.98% for UG-UMRE — the authors attribute this to the difficulty of fine-grained extraction requiring precise semantic boundaries.
• Consistent gains across backbones and input configurations. On FocalMRE with its native inputs (no added depth or caption features), adding UDUA+JAUA raises F1 from 63.57% to 65.49% on UMRE, from 61.60% to 64.22% on MORE, and from 87.42% to 88.31% on MNRE — with no change to the backbone's input/output interfaces or core parameters.
• Both modules contribute, and they synergize. In ablations on UMRE, removing UDUA (text) drops F1 from 69.98% to 68.77%; removing UDUA (image) drops it to 68.35%; removing uncertainty-aware SCL drops it to 69.33%. The visual branch degrades more than the text branch (1.63% versus 1.21% F1 on UMRE), which the authors attribute to visual objects having less explicit semantic boundaries in complex scenes.
• Strong robustness under injected noise. On a constructed "Aug-Noise" challenge set, at 100% noise REMOTE collapses to 35.01% F1 while REMOTE+UDUA holds at 52.33% (a +17.32 point gain). Gains at 100% noise are +14.71 for image-only noise and +7.02 for text-only noise.
• The distributional gap is closed measurably. A t-SNE visualization over 150 text–image pairs from the UMRE test set (500 iterations, perplexity 10) shows text and visual features initially separated, only loosely overlapping with the interaction module alone, and tightly interwoven once JAUA is added.
• Improvements hold across subtask types. On UMRE subtasks, REMOTE+UG-UMRE reaches 79.28% accuracy / 68.00% F1 for entity–entity, 73.60% / 45.24% for object–object, and 82.31% / 76.10% for entity–object — all above both FocalMRE and REMOTE.
• Long-tail relations benefit most, with exceptions. Per-relation analysis on UMRE shows large gains on sparse relations such as /per/per/opponent (+19.36 F1) and /per/misc/race (REMOTE 0.00% to 66.67%), but declines on some relations, including /per/loc/place_of_birth (41.11% to 37.97%) and /misc/misc/part_of (33.33% to 31.03%). Relations with 5 or fewer test samples remain at or near 0.00% for both models.
• Modest computational overhead. Adding UG-UMRE to REMOTE costs 2.36M parameters (0.49%), 3.7G FLOPs, 100 MiB of peak memory, and 4.13 ms per sample of inference time, while adding 2.34 F1 points. The same integration adds 1.92 F1 points over FocalMRE (63.57% to 65.49%).
• Sensitivity analysis identifies specific optimum settings. F1 is best at β₁ = β₂ = 1, τ = 0.175, and λ₁ = 1e-3, λ₂ = 1e-5, λ₃ = 1e-3 on all three datasets.

Methodology in Plain English

The authors treat the multimodal relation extraction pipeline as three sequential stages rather than one tangled fusion step.

Stage 1 — denoise each modality (UDUA). Text and images are first encoded separately (BERT for text; ViT plus Depth-Anything for visual objects; Qwen2.5-VL-7B is used to generate captions that are concatenated with the text). A Multilevel Optimal Transport step aggregates hierarchical features. Instead of representing each sample as a single point in feature space, the model converts it into a Gaussian distribution: a mean (the stable semantic core) and a variance (how uncertain or ambiguous that sample is). This is driven by the Variational Information Bottleneck, which keeps information useful for predicting the relation label while discarding redundant detail, with KL divergence against a standard Gaussian prior enforcing the compression. Because the variance is learned, semantically clear samples get small variances and ambiguous ones get large variances. To use that uncertainty constructively, the model samples a "positive" variant of each sample within its own variance-bounded neighborhood and trains a contrastive (InfoNCE-style) objective that pulls those sampled variants back toward the mean and pushes them away from other samples in the batch — regularizing consistency over plausible perturbations rather than a single fixed embedding.

Stage 2 — calibrate the two modalities (JAUA). Even after denoising, text and image features come from different encoders and live in different distributions. JAUA computes a symmetric KL divergence between the text Gaussian and the visual Gaussian, penalizing both center mismatch and uncertainty-scale mismatch. This forces the two modalities into a common statistical space. The calibrated latent variables are used for alignment, while the original multi-level feature sequences are kept intact for later interaction so that modality-specific local detail is not overwritten.

Stage 3 — fuse and predict. A Hierarchical Cross-Modal Interaction method uses high-level features of one modality as queries attending to each level of the other, producing visual-guided text features and text-guided visual features. A Mixture-of-Experts module then dynamically weights these alongside the text and visual latent representations to produce the final fused feature. For entity–entity relations the text feature marked by the ⟨s⟩ delimiter is used; for visual-object targets, the caption feature marked by ⟨o⟩ combined with the visual feature is used. An MLP with cross-entropy loss produces the final relation prediction, and the whole system is trained jointly with the task loss plus weighted IB, SCL, and JAUA losses.

Experimental setup: a single NVIDIA RTX 4090 GPU, BERT-base and ViT-B/32 backbones, feature dimensions of 768 for text and 4096 for visual objects, AdamW optimizer, 30 epochs, batch size 16, learning rate 1e-5. Datasets: MNRE (9,201 images, 9,201 sentences, 15,485 triplets, 23 relations), MORE (3,559 images, 3,559 sentences, 13,520 visual objects, 20,264 triplets, 21 relations), and UMRE (12,737 images, 12,737 sentences, 20,978 visual objects, 55,021 triplets, 28 relations). Metrics are Precision, Recall, and F1. Text-to-image generative augmentation methods (TMR, ES-MRE, HGMAF, CAMIM) are deliberately excluded from comparison because they alter layout and positional semantics, which the authors argue is incompatible with unified relation extraction.

Why This Matters

Impact on research. The paper reframes a long-standing problem in multimodal learning — that heterogeneous encoders produce misaligned distributions — as something fixable at the distributional level before interaction rather than at the fusion level after it. It also argues against using uncertainty purely for sample weighting or noise estimation, as many prior works do, and instead uses uncertainty to drive targeted denoising and cross-modal calibration. The plug-and-play result is practically significant: the modules improve two different backbones with two different feature recipes without touching either backbone's interface, suggesting the technique generalizes beyond the specific REMOTE implementation.

Real-world applications.

• Multimodal knowledge graph construction, where relation triplets extracted from text-image pairs populate structured knowledge bases.
• Social media and news analysis, where images carry irrelevant background clutter and text uses ambiguous slang or abbreviations — exactly the aleatoric noise the paper targets.
• Biomedical interaction modeling, which the paper cites as a downstream use of MRE.
• Visual question answering and cross-modal retrieval, both listed as downstream applications in the introduction.

Industry relevance. The reported overhead is small: 0.49% additional parameters, 3.7G FLOPs, 100 MiB peak memory, and 4.13 ms per sample on a single RTX 4090. The noise-robustness results are arguably the most industrially relevant finding — under 100% injected noise the baseline loses roughly a third of its F1 (67.64 to 35.01) while the augmented model retains 52.33%, functioning as what the authors call a "safety valve" for deployment on messy user-generated data. The caveat is that these measurements are on one dataset with batch size 16 on a single GPU; broader latency and scaling behavior is not reported.

Future Directions

• Dynamic uncertainty thresholds for sparse relations. The paper's own conclusion notes that current uncertainty estimates rely on holistic paired inputs and fixed loss weights, which may be less reliable for sparse relations; the authors propose dynamic thresholds for few-shot and long-tail relations as future work.
• Open-domain robustness under unknown or severe conditions. The authors call for modeling under unknown noise, severe modality mismatch, and missing modalities — scenarios not tested in the current Aug-Noise experiments, which used controlled token masking and Gaussian feature perturbation.
• Closing the remaining long-tail gap. Several UMRE relations with 5 or fewer test samples still score 0.00% F1 for both REMOTE and UG-UMRE, and a handful of relation types regressed relative to the baseline, leaving clear room for improvement.
• Beyond holistic, paired-input modeling. Since UDUA currently operates on globally pooled features rather than token- or object-level noise, extending uncertainty quantification to finer granularity while keeping the alignment tractable is an open design question the paper implicitly raises.

Target Audience

This paper is most useful to researchers and graduate students working on multimodal learning, multimodal relation extraction, and knowledge graph construction, particularly those interested in uncertainty modeling, variational information bottleneck methods, or contrastive learning applied to noisy real-world data. It is also relevant to practitioners who need to improve an existing multimodal extraction pipeline without retraining a backbone or changing its interfaces, since the modules are presented as lightweight and non-intrusive. Readers without a background in variational inference will need to work through the equations carefully, but the paper's staged framing — denoise, calibrate, fuse — is accessible enough to guide implementation decisions.

Authors’ abstract

Unified Multimodal Relation Extraction (UMRE) aims to identify intra-modal and cross-modal relations between textual entities and visual objects. However, existing UMRE studies still encounter two critical issues: ignoring inherent aleatoric uncertainty causes noise propagation, and deep-seated heterogeneity between distinct modal distributions hinders alignment. To address these issues, we propose the Uncertainty-Guided UMRE Network (UG-UMRE). Specifically, we design an Uncertainty-Driven Unimodal Augmentation (UDUA) module, which models features as Gaussian distributions based on the Variational Information Bottleneck. By incorporating an uncertainty-aware self-supervised contrastive learning mechanism, UDUA effectively filters out noise while maintaining semantic consistency. Furthermore, we introduce the Joint Aleatoric Uncertainty Alignment (JAUA) module as a global semantic pre-calibration mechanism. JAUA leverages probabilistic distribution consistency to construct a shared latent space, eliminating the distributional gap by synchronizing cross-modal statistical properties, thereby laying a robust foundation for fine-grained interaction. Experiments on three benchmark datasets (UMRE, MORE, and MNRE) demonstrate that UG-UMRE achieves state-of-the-art performance. Further analysis validates the pluggable and effective performance of the proposed UDUA and JAUA modules.