iStructTab: Structured Feature Sequencing for Multimodal Learning of Image and Tabular Data

AI summary

Overview

Research area: Multimodal machine learning (computer vision + tabular data), with roots in pattern recognition, feature ordering (Incremental Attribute Learning), and combinatorial optimization.

Technical level: Advanced. The paper assumes familiarity with the Column Permutation Problem (CPP), graph convolution, attention mechanisms, and Linformer-style efficient transformers.

Scope: A single paper proposing a feature-sequencing algorithm (GEDS) and an order-aware transformer backbone (OEMT) that together fuse image and tabular data, evaluated on six image-tabular benchmarks against 15 baseline methods.

What This Paper Is About

Multimodal models that combine images with tabular data usually concatenate or attend over features without ever deciding what order those features should sit in. Because tabular columns are inherently unordered while images have strong spatial structure, this mismatch can create redundant or dispersed representations that hurt cross-modal alignment and generalization. The paper casts feature ordering as a Column Permutation Problem and asks: if we deliberately search for a good ordering of fused image-plus-tabular features, does multimodal prediction improve?

Key Contributions

1. Reformulating multimodal fusion as a Column Permutation Problem (CPP). The authors define a dispersion cost over permutations and argue that feature ordering is an overlooked factor affecting representation learning in hybrid tabular-image settings.

2. Graph-Enhanced Descriptor Sequencing (GEDS). A sequencing algorithm that builds statistical descriptors (mean and variance) for each fused feature, refines them via cosine-similarity graph construction and graph convolution, and produces a permutation by sorting a per-feature ordering score.

3. Order-Aware Efficient Transformer with Memory Augmentation (OEMT). A backbone that enforces the GEDS ordering through order-aware pooling (compressing m features into k ≪ m tokens), M learnable memory tokens, and a dedicated sequencing loss that aligns predicted ordering scores with a linearly decaying target vector.

4. Evaluation across six image-tabular benchmarks covering accuracy, average rank, average regret, label-noise robustness, computational efficiency, calibration, and perturbation robustness, supported by Friedman, Nemenyi, and Wilcoxon-Holm significance tests.

Main Findings

• Best overall accuracy ranking. Across six datasets, iStructTab achieves the top Avg. Rank (1.50 ± 0.76) and lowest Avg. Regret (2.21 ± 4.59) among 16 methods ranked per dataset.

• Per-dataset scores. iStructTab reaches 99.29% on DVM, 85.23% on HAM, 83.75% on DL, 68.29% on Pok, 76.15% on CheX, and 87.75% on Pet. Strong competitors include STiL (99.27, 78.48, 81.35, 27.32, 88.60, 87.68) and TIP (98.27, 70.39, 69.17, 63.40, 87.50, 83.86). Notably, STiL scores higher than iStructTab on CheXpert (88.60 vs. 76.15), so the headline ranking reflects aggregate performance rather than a win on every dataset.

• Gains concentrated on multimodal-friendly benchmarks. The authors report the largest improvements on Pok and Pet, where metadata helps disambiguate visually similar or noisy samples, and describe a consistent ordering of multimodal fusion > image-only > tabular-only.

• Label-noise robustness. On HAM10000 under injected label noise, iStructTab scores 83.92, 81.92, 79.42, and 76.42 at noise levels 0.1, 0.2, 0.4, and 0.6, with a mean of 80.42, compared with CUFIT's mean of 78.3. At 60% noise the gap widens to 76.42 vs. 70.1.

• Best accuracy-efficiency trade-off. Using min-max normalized GFLOPs, parameters, latency, and composite complexity with weights (0.40, 0.25, 0.20, 0.15), iStructTab attains an efficiency score of 1.00, more than twice the next-best method, CHARMS at 0.40, and sits in the low-complexity/high-accuracy Pareto region on DVM.

• Ablation: sequencing and memory are both essential. On DVM, the full model reaches 96.85% untuned and 99.29% tuned, versus 90.42%/89.60% for plain concatenation. Removing memory tokens drops tuned accuracy from 99.29% to 87.60%; removing the sequencing loss drops it to 96.78%; removing feature sequencing entirely collapses performance to about 81-83%; and applying the memory-based sequencing loss without any ordering signal fails severely at 13.91-16.86%.

• Backbone choice is not the driver. Swapping ResNet-50 for ResNeXt-50 lowers accuracy (91.88%/93.33%), suggesting gains come from GEDS sequencing and OEMT rather than a stronger image encoder.

• Calibration and inference robustness. On HAM10000, Expected Calibration Error is 0.103 and fused-space kNN label agreement stays at 97.79% across all tested perturbation fractions. Accuracy degrades smoothly from 85.17% to 82.78% under full tabular shuffling and to 83.37% under full tabular dropping.

• Qualitative representation structure. t-SNE projections show large, smooth manifolds with contiguous regions on HAM10000, while CheXpert (Cardiomegaly) shows an elongated, continuum-like geometry that matches the clinical spectrum and label noise of that dataset.

Methodology in Plain English

The pipeline runs in stages.

First, images pass through a ResNet encoder and tabular columns pass through a transformer encoder, producing a d-dimensional image vector and a p-dimensional tabular vector per sample. These are concatenated into a single fused feature matrix with m = p + d columns.

Second, GEDS looks at each fused column and summarizes it with just two numbers: its mean and its variance. These two-number descriptors are scaled to unit length so that large-magnitude features don't dominate, and then every pair of descriptors is compared with cosine similarity to build a similarity graph. The graph is augmented with self-connections and symmetrically normalized, then passed through a graph convolution with a learnable weight matrix and a nonlinearity. This refinement lets each feature's descriptor absorb information from statistically similar features.

Third, each refined descriptor is scored by its Euclidean norm, and the features are sorted in ascending order of that score. The resulting permutation, π_GEDS, is the sequence the rest of the model must respect.

Fourth, the OEMT backbone takes the reordered feature vector, projects it into tokens, and compresses the m tokens down to k summary tokens using a learned pooling matrix. Because the input tokens were already reordered, the pooled tokens inherit that order. M learnable memory tokens are prepended to provide global context, and the whole sequence goes through a Linformer encoder.

Finally, training uses a two-part objective: ordinary cross-entropy on the classification output, plus a sequencing term that penalizes the squared distance between the model's predicted ordering scores and a target vector that decays linearly from 1 to 0 across the permutation. A weighting factor λ_FS balances the two. The intuition is that the model is pushed both to classify correctly and to internalize the feature order it was given.

Datasets were split 64/16/20 (train/validation/test) with stratification, and hyperparameters were tuned with Optuna over 20 trials per dataset on PyTorch 2.4.1+CUDA 12.1 with automatic mixed precision. Missing values were handled by median imputation and categorical/text encoding, except DVM, where CTGAN was used for extensive missingness.

Why This Matters

Impact on research. The paper reframes a mundane implementation detail — the order in which you feed features to a model — as a combinatorial optimization problem with measurable downstream consequences. It connects deep multimodal learning to the older Incremental Attribute Learning and Column Permutation Problem literature, and provides a concrete recipe for making permutation-sensitive architectures aware of feature semantics rather than treating columns as arbitrary.

Real-world applications (drawn from the paper's own evaluation domains):

• Dermatology triage. HAM10000 combines 10,015 dermoscopic images with 4 metadata fields across 7 lesion classes; the calibrated, low-ECE behavior reported here matters for clinical decision support.
• Cardiology screening. CheXpert Cardiomegaly classification using 9,999 frontal X-ray pairs with 17 metadata features, where the paper's t-SNE analysis shows the model organizing patients along a severity axis.
• Lesion detection from CT. DeepLesion provides CT slices with lesion and patient metadata, used here as 1,327 samples with 35 features.
• Attribute-conditioned fine-grained recognition. DVM's 176,414 car images with 17 attributes and 283 classes is a large-scale test of whether metadata can sharpen visual classification under heavy missingness.

The paper also names healthcare, remote sensing, and environmental modeling as fields where multimodal image-tabular learning is increasingly important.

Industry relevance. The efficiency results matter for deployment: iStructTab achieves near-maximum accuracy with the lowest FLOPs, parameters, and latency among tested multimodal methods on DVM, and its inference results show that latency and GPU memory stay nearly flat as the effective tabular sequence length varies. Combined with graceful degradation under tabular shuffling, tabular dropping, and Gaussian blur, this points toward models that can run under messy production data without retraining. The authors also release code on GitHub and a package installable via pip install istructtab.

Future Directions

• Scaling the affinity computation. GEDS requires an O(m²) affinity step. The authors suggest approximating it with sparse k-NN graphs or low-rank similarity estimation when m grows large, but this is left untested.
• Larger-scale pretraining and additional modalities. The conclusion states that extending iStructTab to larger-scale pretraining and beyond image-plus-tabular inputs is future work.
• Hyperparameter trade-off analysis. The authors explicitly flag the relationship among embedding dimension, pooling length, runtime, and performance as an open question.
• Handling spectrum-style and weakly supervised labels. The CheXpert results show iStructTab trailing STiL on that benchmark, and the t-SNE discussion attributes this to interleaved negative/uncertain/positive cases and label noise. Whether sequencing helps or hurts on such continuum tasks remains unresolved.

Target Audience

Researchers and practitioners working on multimodal fusion, particularly those combining imaging with structured metadata in medical, scientific, or industrial settings. It will also interest readers of the tabular deep learning literature concerned with permutation sensitivity, and engineers who need efficient, robust multimodal models deployable under compute or latency constraints. Familiarity with transformers, graph neural networks, and combinatorial optimization will make the technical sections substantially easier to follow.

Authors’ abstract

Multimodal learning of images and tabular data is often impaired by ineffective representations, resulting in redundancy, dispersion, and generalization problems. To tackle this challenge, we introduce Graph-Enhanced Descriptor Sequencing (GEDS), a structured feature sequencing algorithm grounded in principles from the Column Permutation Problem (CPP). GEDS refines statistical descriptors of the features through similarity graph-based computations, systematically determining an effective feature sequencing. We incorporate GEDS within an order-aware efficient transformer framework, utilizing order-aware memory tokens that explicitly adhere to the derived feature sequencing via a dedicated loss function. Experimental results across multimodal benchmarks demonstrate that iStructTab effectively minimizes feature dispersion, improving predictive performance and robustness, and highlighting the significance of structured feature sequencing in multimodal learning.