Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds

AI summary

Overview

Research area: Computer Vision and multimodal representation learning — specifically vision-language models (VLMs), hierarchical feature alignment, and geometric deep learning on hyperbolic manifolds.

Technical level: Advanced. The paper combines Riemannian geometry (Lorentz model of hyperbolic space), information geometry (KL divergence between distributions on curved manifolds), and transformer-based vision-language prompt tuning.

Scope: This paper proposes a method called Alignment across Trees that builds hierarchical feature trees for both images and text and aligns them across hyperbolic manifolds with distinct, learnable curvatures via an optimized intermediate manifold.

What This Paper Is About

Most vision-language models align a hierarchy of textual labels (coarse-to-fine concepts) against a single image embedding, producing an asymmetric and geometrically mismatched alignment. This paper constructs equivalent hierarchical feature trees for both modalities and aligns them symmetrically, embedding each tree in its own hyperbolic manifold so that curvatures can adapt to the intrinsic geometry of text versus images. The alignment is performed on an intermediate manifold whose curvature is proven to exist uniquely and is optimized for minimal geometric distortion.

Key Contributions

1. Alignment across Trees framework. The first method to construct symmetric (tree-to-tree) hierarchical features for image and text modalities, replacing the standard asymmetric single-vector-to-hierarchy alignment.
2. Semantic-aware visual feature extraction. A cross-attention module that uses textual features at each hierarchy level as queries and class tokens from intermediate Transformer layers as keys/values, producing coarse-to-fine visual features without discarding the semantic ordering of ViT layers.
3. Heterogeneous manifold alignment algorithm. A construction that embeds each modality into its own hyperbolic manifold with a separately learnable curvature, defines a KL-divergence-based distance between wrapped normal distributions on heterogeneous manifolds, and solves for an optimal intermediate manifold. The paper proves existence and uniqueness of this optimum.
4. Strong empirical gains on taxonomic open-set classification. Consistent improvements over MaPLe and PromptSRC baselines (with and without ProTeCt) across four datasets and four evaluation protocols.

Main Findings

• Symmetric beats asymmetric. Extracting a hierarchy of visual features (rather than one feature) and pairing them with the text hierarchy yields substantial gains; the paper argues this is the key structural change driving the improvements.
• Large HCA improvements. Hierarchical Consistent Accuracy improves by up to 7.72% under 1-shot and 28.83% under 16-shot settings. In the 16-shot regime, LA improves by up to 19.02%, HCA by 28.83%, and MTA by 8.48%.
• Generalizes to novel classes. Under base-to-novel evaluation on Cifar100, the method achieves +1.38% LA, +5.66% HCA, and +4.90% MTA on unseen classes, showing the alignment transfers beyond the base label set.
• Every component matters. Ablations compare Euclidean alignment (Ours-Euc), a single shared curvature (Ours-HypV1), and independently learnable curvatures (Ours-HypV2) against the full method. Hyperbolic modeling helps over Euclidean, and the intermediate-manifold search adds further gains on top of any curvature scheme.
• Negligible computational cost. Learning multiple curvatures adds only 0.5 s per batch (74.0 s → 74.5 s) and no extra memory (10,400 MB in both settings), so the theoretical machinery is practically cheap.
• Theoretical soundness. Proposition 1 shows that the distance is minimized when the intermediate curvature equals the modality curvature, and Proposition 2 shows the joint objective has a unique minimizer in the interval between the two modality curvatures.

Methodology in Plain English

The approach proceeds in four stages.

1. Build a visual feature tree. Instead of taking only the final-layer CLS token from a ViT, the method collects class tokens from several intermediate layers. Each intermediate token is passed forward with cross-token attention disabled (only linear projections, residuals, and MLP) so its coarse-grained semantics survive. A cross-attention module then uses the hierarchical text embeddings as queries and the stack of intermediate-plus-final visual tokens as keys and values, producing one visual feature per hierarchy level. The result is a visual tree that mirrors the text tree.

2. Embed each tree in its own hyperbolic space. Text and image trees are projected into two separate Lorentz-model hyperbolic manifolds with distinct curvatures c₁ (text) and c₂ (image). Hyperbolic space is chosen because its volume grows exponentially with radius, matching the exponential growth of concepts down a taxonomy.

3. Find an intermediate manifold. Since direct measurement of dissimilarity between differently curved hyperbolic manifolds is not well developed, the authors model data on each manifold as a wrapped normal distribution and define a distance between manifolds as an affine transform of the KL divergence between those distributions. The optimal intermediate curvature c₃* minimizes the sum of distances from c₃ to both c₁ and c₂. A golden section search solves this 1-D problem, and since that search is non-differentiable, the implicit function theorem supplies gradients to c₁ and c₂ during backpropagation.

4. Align via entailment cones. On the intermediate manifold, the visual feature at each level is constrained to lie inside the hyperbolic entailment cone of the corresponding textual feature (text is treated as the broader concept). Within each modality, finer features are similarly constrained to lie inside the cones of coarser features. These cone constraints plus the standard prompt-learning objective give the final training loss, which updates learnable prompt tokens and both curvatures jointly.

Why This Matters

Research impact. The paper reframes modality alignment from "map everything to one Euclidean space" toward "learn the right curved space per modality and bridge them." The KL-based distance between heterogeneous hyperbolic manifolds and the existence/uniqueness result for the intermediate curvature are general tools that could be reused wherever two structured distributions live in different geometries — not just in vision-language tasks.

Real-world applications.

• Biodiversity and species identification. The Rare Species dataset and the taxonomy-oriented evaluation directly target field biology, where classifiers must predict species, genus, family, and so on from photos with very few examples.
• Fine-grained product categorization in e-commerce. Products sit in deep category trees (department → category → subcategory); a tree-aware alignment helps zero-shot and few-shot tagging of new SKUs.
• Medical imaging and clinical coding. Diagnoses follow hierarchical ontologies (ICD codes), and symmetric hierarchical alignment could improve rare-disease screening where labels are sparse.
• Content tagging and moderation. Hierarchical tag systems for large media platforms benefit from models that respect parent-child label relationships instead of flattening them.

Industry relevance. The gains are demonstrated on top of lightweight prompt-tuning methods (MaPLe, PromptSRC) with essentially no added memory and less than 1% added time per batch. That makes the approach attractive for teams that already run CLIP-style models and want better hierarchical classification without retraining from scratch.

Future Directions

1. Extending beyond two modalities. The intermediate-manifold framework is formulated for a pair of manifolds; generalizing to three or more (image, text, audio, video) would require a new notion of a multi-way intermediate geometry.
2. Applying to full pre-training rather than prompt tuning. All experiments use frozen CLIP backbones with learnable prompts; testing whether the same alignment improves large-scale pre-training is an open question with substantially higher compute cost.
3. Tighter approximations of KL divergence on hyperbolic manifolds. The current manifold distance is an approximation with no closed-form KL; sharper or exactly computable distances could tighten the alignment and the theory.
4. Adaptive or learned hierarchy depth. The method assumes the textual hierarchy depth H is given; discovering the right granularity per sample or per dataset remains unresolved.
5. Broader structured domains. Hyperbolic entailment has proven useful for knowledge graphs and code; applying the heterogeneous-manifold alignment to these modality pairs is a natural extension.

Target Audience

This paper is most valuable to researchers and graduate students working on multimodal learning, vision-language models, and geometric deep learning who already have some familiarity with Riemannian geometry and hyperbolic embeddings. Practitioners building hierarchical or taxonomy-aware classification systems (biodiversity, e-commerce, medical coding, content tagging) will also benefit from the empirical results and the lightweight computational profile, even if they skip the proofs. Readers without a background in differential geometry can still follow the motivation and experimental sections, but the theory sections assume comfort with manifolds, tangent spaces, exponential/logarithmic maps, and KL divergence.

Authors’ abstract

Modality alignment is critical for vision-language models (VLMs) to effectively integrate information across modalities. However, existing methods extract hierarchical features from text while representing each image with a single feature, leading to asymmetric and suboptimal alignment. To address this, we propose Alignment across Trees, a method that constructs and aligns tree-like hierarchical features for both image and text modalities. Specifically, we introduce a semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers, guided by textual cues to extract visual features with coarse-to-fine semantics. We then embed the feature trees of the two modalities into hyperbolic manifolds with distinct curvatures to effectively model their hierarchical structures. To align across the heterogeneous hyperbolic manifolds with different curvatures, we formulate a KL distance measure between distributions on heterogeneous manifolds, and learn an intermediate manifold for manifold alignment by minimizing the distance. We prove the existence and uniqueness of the optimal intermediate manifold. Experiments on taxonomic open-set classification tasks across multiple image datasets demonstrate that our method consistently outperforms strong baselines under few-shot and cross-domain settings.