HMVLA: Hyperbolic Multimodal Fusion for Vision-Language-Action Models

AI summary

Overview

Research area: Robotics / Embodied AI — specifically Vision-Language-Action (VLA) models that connect multimodal perception to robotic control.

Technical level: Advanced. The paper builds on hyperbolic geometry (the Lorentz model of hyperbolic space), entailment-cone constraints, and Mixture-of-Experts routing.

Scope: The paper proposes HMVLA, a VLA framework that embeds image and text features in hyperbolic space and fuses them with a sparsely gated Mixture-of-Experts, evaluated on the LIBERO benchmark and on a re-constructed generalization dataset.

What This Paper Is About

Most VLA systems take a pre-trained Vision-Language Model and fine-tune it on robot data, feeding visual and semantic features straight into a policy network. The authors argue this disrupts the hierarchical structure of vision and language — for example, a model trained on "grasp the cup" against a white table with a blue mug may latch onto spurious correlations like "white equals background" and "blue equals cup" rather than grounding the action and the object. HMVLA's goal is to preserve those hierarchical semantic-visual relationships by doing the alignment in hyperbolic space rather than Euclidean space, assisted by a Mixture-of-Experts module across modalities.

Key Contributions

1. Hyperbolic geometry brought into the VLA domain. The framework uses the Lorentz model of hyperbolic space, leveraging its exponential expansion property to better preserve hierarchical relationships during multimodal fusion.
2. A Mixture-of-Experts module for adaptive routing. Information is routed across experts to strengthen alignment between visual and linguistic modalities for action prediction. The abstract describes it as "sparsely gated"; the method section describes a soft MoE that replaces the Q-Former's feed-forward layers.
3. A hyperbolic entailment loss combined with contrastive loss. An entailment-cone constraint penalizes image embeddings that fall outside the cone defined by their paired text embedding, adding structural supervision on top of the standard CLIP-style contrastive objective.
4. A reconstructed generalization dataset. The authors rebuild the four LIBERO datasets (Spatial, Object, Goal, LONG) into a new "Gen" dataset to systematically test cross-domain adaptability and generalization.

Main Findings

• Best average accuracy on LIBERO: HMVLA reaches an average of 86%, against Dita 82%, CoT-VLA 81%, Octo 75%, DP 72%, and Tra-MoE 66%.
• Per-task results (Spatial / Object / Goal / LONG): HMVLA 90% / 96% / 89% / 69%; Dita 84% / 96% / 85% / 63%; CoT-VLA 87% / 91% / 87% / 69%; Octo 79% / 86% / 85% / 50%; DP 78% / 92% / 68% / 50%; Tra-MoE 69% / 77% / 88% / 31%.
• Strongest gains on the longest-horizon task: HMVLA ties CoT-VLA at 69% on LONG, improving 6 points over the Dita backbone (63%) and 19 points over DP and Octo (both 50%).
• Generalization claim without reported numbers: The authors state HMVLA shows "breakthrough improvements" on the reconstructed generalization validation set (Figure 3, compared against OpenVLA, DFP-OTTER, OTTER, and Dita), but the truncated content reports no numeric values for that comparison.
• Ablation findings: Introducing hyperbolic space into Dita "significantly improves task success," especially for complex instructions where the original Dita struggled. MoE routing helps avoid overfitting to a single fused representation. FiLM provides more stable and balanced modality conditioning than Cross-Attention. The ablation figure (Figure 5) shows these comparisons, but the truncated content gives no numeric ablation values.
• Qualitative result: Figure 4 shows HMVLA's grasping process on the LIBERO benchmark.

Methodology in Plain English

The model starts from a pre-trained VLM backbone (Dita is used as the backbone network). Images and text pass through their encoders, and the resulting features are linearly projected. Instead of comparing them in ordinary flat space, the authors lift these feature vectors onto a Lorentz hyperboloid — the mathematical surface used to represent hyperbolic space. Because the projected vectors live in the tangent space at the hyperboloid's origin, the exponential map simplifies, and the authors parameterize only the spatial components of vectors, computing the time component from the constraint equation.

Alignment then uses two losses. The first is the familiar contrastive loss. The second is an entailment loss: each text embedding defines a cone in hyperbolic space, and if the paired image embedding falls outside that cone's aperture, the model is penalized by the exterior angle minus the aperture. The two losses are combined with a balancing coefficient of 0.1.

For fusion, the Q-Former's feed-forward layers are replaced with a soft Mixture-of-Experts module. After self- and cross-attention produce query tokens, a gating network produces softmax weights over M experts (M = 6), and each token is updated as a weighted sum of all experts' outputs. A load-balancing loss, controlled by a coefficient beta, encourages balanced expert usage. Total training runs 80k steps with the Adam optimizer at a learning rate of 1×10⁻⁴, curvature 0.1, batch size 64, 256×256 input images, trajectory length 11, trajectory dimension 7, predicting 10 future actions per step, on an NVIDIA H200 GPU.

Why This Matters

Impact on research: The paper argues that the standard recipe of directly fine-tuning a pre-trained VLM for robot control perturbs the hierarchical structure of visual and semantic features. It offers an alternative — doing multimodal alignment in a geometry that natively represents hierarchy — and pairs it with expert routing. It also contributes a reconstructed LIBERO dataset intended as a cross-domain generalization test, and it echoes broader vision-language findings that fine-tuning linguistically aligned visual encoders can lead to overfitting.

Real-world applications:

• Robotic manipulation with long-horizon, multi-step instructions, where the LONG task gains are largest.
• Cross-domain deployment, where a robot trained in one environment must handle novel instructions or previously unseen objects and backgrounds.
• Instruction-following assistants for tabletop grasping and object handling, as illustrated by the grasping process in Figure 4.
• Reducing spurious-correlation failures in learned policies, such as a policy that associates "blue" with "cup" rather than with the object's semantics.

Industry relevance: Companies building general-purpose robot policies depend on fine-tuning large pre-trained VLMs affordably. If hyperbolic alignment plus expert routing preserves semantic structure better than direct fine-tuning, that is relevant to anyone shipping manipulation policies that must generalize beyond their training distribution. The MoE design is also framed as improving efficiency alongside comprehension.

Funding context: The work was supported by the National Natural Science Foundation of China (Grant No. 62277011), the National Key Research and Development Program of China (Grant No. GG-2024-01-02), a Project of Chongqing MEITC (Grant No. YJX-2025001001009), and the Open Research Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) (Grant No. GML-KF-24-18).

Future Directions

1. Report quantitative generalization results. The paper claims breakthrough improvements on the reconstructed Gen dataset but the truncated content provides no numbers; a full accounting of cross-domain accuracy would clarify how large those gains are.
2. Reconcile the MoE description. The abstract calls the mechanism "sparsely gated," while the method section describes a soft MoE in which every token accesses all experts with softmax weights. Clarifying whether sparse routing is used, and how it affects inference cost, would sharpen the efficiency claims.
3. Quantify the ablations. The ablation discussion describes directional effects of hyperbolic space, MoE, and FiLM versus Cross-Attention, but no per-component numbers appear in the truncated content. Reporting these would isolate how much each component contributes.
4. Study curvature and expert count. Curvature is fixed at 0.1 and the expert count at 6; whether these are tuned or arbitrary, and how sensitive the results are to them, remains an open question.

Target Audience

Robotics and embodied-AI researchers working on Vision-Language-Action models; machine learning researchers interested in non-Euclidean representation learning for multimodal alignment; and engineers building robot manipulation policies on top of pre-trained VLMs who need policies that generalize to novel instructions, objects, and environments. A background in hyperbolic geometry and transformer architectures is helpful, given the Lorentz-model formalism and the MoE routing equations.

Authors’ abstract

Vision Language Action (VLA) models have recently shown great potential in bridging multimodal perception with robotic control. However, existing methods often rely on direct fine-tuning of pre-trained Vision-Language Models (VLMs), feeding semantic and visual features directly into a policy network without fully addressing the unique semantic alignment challenges in the VLA domain. In this paper, we propose HMVLA, a novel VLA framework that exploits the inherent hierarchical structures in vision and language for comprehensive semantic alignment. Unlike traditional methods that perform alignment in Euclidean space, our HMVLA embeds multimodal features in hyperbolic space, enabling more effective modeling of the hierarchical relationships present in image text data. Furthermore, we introduce a sparsely gated Mixture of Experts (MoE) mechanism tailored for semantic alignment, which enhances multimodal comprehension between images and text while improving efficiency. Extensive experiments demonstrate that HMVLA surpasses baseline methods in both accuracy and generalization. In addition, we validate its robustness by reconstructing datasets to further test cross domain adaptability.