LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models

AI summary

Overview

Research area: Efficient multimodal machine learning — specifically, weight compression for large multimodal models (LMMs) using frequency-domain low-rank decomposition combined with quantization.

Technical level: Intermediate to Advanced. The paper assumes familiarity with singular value decomposition (SVD), low-rank adaptation (LoRA), quantization, and the discrete Fourier transform.

Scope: The paper proposes LLaVA-FA, a compression framework that performs joint low-rank plus quantized weight approximation in the Fourier domain, together with a polar-coordinate quantizer (PolarQuant) and an optional diagonal calibration (ODC) scheme, and evaluates the resulting models on standard multimodal benchmarks.

What This Paper Is About

Large multimodal models achieve strong vision-language performance but are expensive to store and run, so compression is essential for practical deployment. Existing methods typically treat low-rank decomposition and quantization as separate steps, which compounds reconstruction errors — and this problem is worse in multimodal architectures that carry an extra image encoder and cross-modal adapters with cross-modal redundancy. The paper asks whether the decorrelation, conjugate symmetry, and energy-compaction properties of the Fourier domain can be used to do low-rank plus quantization compression in a single shot, producing a smaller and more accurate compressed model.

Key Contributions

1. LLaVA-FA, an efficient LMM framework that decomposes each pretrained weight matrix into a low-rank component plus a quantized residual entirely in the frequency domain (W̃ ≈ Q̃ + L̃₁L̃₂), rather than in the spatial domain.

2. PolarQuant, an amplitude-and-phase polar-coordinate codec designed specifically for quantizing complex matrices. It discretizes amplitude and phase separately, preserving the complex structure and stabilizing low-bit reconstruction.

3. Optional diagonal calibration (ODC), a scheme that approximates the full Hessian with row and column means of the calibration matrix, enabling more robust compression without requiring large-scale calibration data.

4. Extensive experiments across comprehension-oriented and hallucination-oriented benchmarks showing that LLaVA-FA surpasses existing efficient multimodal models while maintaining minimal activated parameters and low computational cost.

Main Findings

• Frequency-domain truncation has smaller error at equal rank. Lemma 3.1 shows that if one adapter's singular values dominate another's for every index k ≥ r+1, its rank-r truncation has smaller Frobenius error. The authors observe that LMM weight matrices have a more compact spread of singular values in the frequency domain than in the spatial domain, so the accumulated low-rank approximation error is smaller in the frequency domain at the same rank.

• Conjugate symmetry halves stored parameters. A real matrix of shape [d₁, d₂] maps to a complex frequency-domain matrix of shape [d₁, d₂/2] without information loss, so approximating in the frequency domain can save nearly half of the learnable parameters compared to the spatial domain.

• PolarQuant separates amplitude and phase. Rather than quantizing real and imaginary parts directly, PolarQuant converts each complex entry to (r_ij, θ_ij) using r_ij = sqrt(X_ij² + Y_ij²) and θ_ij = atan2(Y_ij, X_ij), then uses step sizes Δr = max(r_ij)/(2^{b_r} − 1) and Δθ = 2π/2^{b_θ}. The paper states this preserves the phase structure critical for cross-modal alignment. Note that q_{θ,i,j} = round(θ_{i,j} + π/Δθ) is written in the equation statement while Algorithm 3 writes round((θ_{i,j} + π)/Δθ).

• Calibration data can be made optional. ODC replaces the intractable calibration-weighted objective with a diagonal approximation, using D_row (row means of sqrt(C)) and col-means, then solving via FourierSVD on D_row R̃ D_col. This avoids dependence on the availability and quality of calibration data — the paper notes calibration-based approaches usually require 256–2048 held-out samples.

• Parameter budget analysis. For a transformer layer with seven weight matrices (query, key, value, output, gate, up, down), average bits per parameter is B_avg = Σ(B_Q d₁ⁱd₂ⁱ + k B_L (d₁ⁱ + d₂ⁱ)) / Σ d₁ⁱd₂ⁱ. When k < (1 − B_Q/B_L) · Σd₁ⁱd₂ⁱ / Σ(d₁ⁱ + d₂ⁱ), B_avg is smaller than the full-precision budget B_L. For LLaMa3-8B the dimensions are 4096×4096 (query, output), 4096×1024 (key, value), 14336×4096 (gate, up), and 4096×14336 (down), and the paper states B_avg < B_L holds for common LLM configurations. The paper gives B_Q = 2 bits as an illustrative example of the quantized backbone.

• Iterative alternating optimization with a simple stopping rule. Algorithm 1 alternates between ODC for the low-rank factors and PolarQuant for the quantized residual, tracks the error ε_t, and terminates when ε_t > ε_{t−1}; Q̃ is initialized to 0.

• Benchmark performance of the two larger variants. On the reported table, LLaVA-FA-7B (built on InternLM-2-20B, 5M training samples) scores GQA 68.5, VizWiz 62.0, ScienceQA 76.0, TextVQA 68.0, MME 74.5, MMB 74.5, MMB^CN 69.5, for an average of 70.4. LLaVA-FA-3B (built on LLaMA-3-8B, 5M samples) scores GQA 65.0, VizWiz 62.5, ScienceQA 77.0, TextVQA 64.0, MME 71.0, MMB 70.5, MMB^CN 68.0, for an average of 68.3. The paper states LLaVA-FA achieves the best average result in both the 7B-class and 3B-class comparison groups.

• Comparison anchors from the same table. Among larger baselines, LLaVA-NeXT (Vicuna-1.5-13B) averages 68.5 and Deepseek-VL-7B averages 67.2, while VILA-7B averages 65.7 and LLaVA-1.5-7B averages 62.1. Among smaller baselines, MiniCPM-V-2 averages 66.9 and VILA-3B averages 61.8.

• The reported content is truncated. Table 1 cuts off after the row labeled "LLaVA-F", so results for LLaVA-FA-2B and LLaVA-FA-1B are not visible in the provided text. Results for the hallucination benchmarks the paper says it uses (POPE, Object HalBench, MMHal-Bench) are likewise not present in the provided excerpt, and the numerical comparison of training cost in Figure 1 is described only qualitatively.

• Multimodal motivation. The paper notes that training LLaVA 70B models requires over 800 GPU hours, calculated based on NVIDIA A100 GPUs, and that unlike pure-text LLMs, LMMs carry an extra image encoder whose cross-modal adapter ranks grow with each new visual domain.

Methodology in Plain English

The starting point is the standard compression recipe W ≈ Q + L₁L₂: a low-precision backbone matrix Q plus two full-precision low-rank factors L₁ and L₂. The authors move this entire problem into the frequency domain. They apply a 2D discrete Fourier transform to each real weight matrix, turning it into a complex matrix of half the width. Because the Fourier transform decorrelates the weights, the singular values of the transformed matrix decay faster, so keeping only the top r singular values loses less information than cutting off the same number of components in the original domain. Because the Fourier transform of a real matrix is conjugate symmetric, only half the coefficients need to be stored.

To find the decomposition, they alternate two steps until the error stops improving. First, ODC extracts the low-rank part using a complex-valued SVD (FourierSVD), optionally weighting the residual by a diagonal approximation of the calibration information built from row and column averages; ignoring this weighting gives the calibration-free variant. Second, PolarQuant compresses what is left: each residual complex entry is rewritten as an amplitude and an angle, those two quantities are uniformly quantized with separate bit widths, and the complex value is reconstructed from the quantized polar pair. The result is a drop-in replacement for the original weights — the architecture of the model is not changed and no tokens are pruned.

The experiments take pretrained CLIP-ViT-L/14 as the vision encoder, a two-layer MLP as the vision-language projector, and Qwen-2.5 as the language backbone. LLaVA-FA-2B and LLaVA-FA-1B are compressed from Qwen-2.5-7B and Qwen-2.5-3B respectively, while LLaVA-FA-7B and LLaVA-FA-3B are compressed from InternLM-2-20B and LLaMA-3-8B respectively. Training uses 8 NVIDIA RTX 4090 GPUs on a mixture of pretraining, VQA, text-centric, chart/document, reasoning, and web-scale datasets.

Why This Matters

Impact on research. The paper makes the first attempt (by the authors' account) to perform joint low-rank plus quantization approximation directly in the frequency domain. If the error argument holds generally, it suggests the spatial/frequency choice of basis is an underexplored degree of freedom in post-training compression, and that treating rank selection and quantization jointly rather than sequentially matters.

Real-world applications:

• Running multimodal assistants on consumer or edge hardware, where the compressed parameter count and reduced compute of LLaVA-FA-1B/2B/3B variants make local inference feasible.
• Document, chart, and OCR-heavy pipelines (the training mix includes ChartQA, DocVQA, OCR-VQA, and SynthDoG-EN), where teams need vision-language capability without full-scale serving costs.
• Deployment scenarios where calibration data cannot be collected or shared, which is the case ODC is designed for.
• Serving many multimodal models simultaneously, where per-model memory footprint directly determines how many models fit on a fixed GPU fleet.

Industry relevance. Model compression that keeps accuracy while cutting stored parameters and activated compute translates directly into lower inference electricity consumption and cheaper serving — the paper explicitly frames this as serving environmentally friendly AI and broadening accessibility. A method that is a drop-in weight replacement without architecture changes is easier to adopt than approaches requiring token pruning or structural edits.

Future Directions

• How sensitive are the results to the rank r and the bit widths b_r and b_θ? The paper states a condition under which B_avg < B_L but the provided content does not report a sweep over these settings or the accuracy cost of each.
• How far does calibration-free ODC get relative to the calibrated variant? The scheme is described as optional; the provided excerpt does not report an ablation isolating its contribution.
• Does the Fourier-domain advantage generalize beyond LLaVA-style stacks? The motivation is cross-modal adapter redundancy, but the reported experiments cover LMMs built on Qwen-2.5, InternLM-2, and LLaMA-3 — whether the same gains appear for other modalities or architectures is not addressed in the provided content.
• Hallucination benchmarks are listed but not reported in the provided text. The paper states it evaluates POPE, Object HalBench, and MMHal-Bench, and claims hallucination-oriented results; the numerical outcomes are not visible in the truncated content.

Target Audience

Researchers and engineers working on model compression, efficient inference, and large multimodal models — particularly those already familiar with LoRA, SVD-based decomposition, or post-training quantization. It is also relevant to practitioners who need to deploy vision-language models under tight memory or compute budgets and who cannot rely on large curated calibration sets. Readers without background in linear algebra and signal processing will find the methodology sections heavy going, though the motivation and experimental tables are accessible.

Authors’ abstract

Large multimodal models (LMMs) have achieved impressive performance on various vision-language tasks, but their substantial computational and memory costs hinder their practical deployment. Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors, especially in multimodal architectures with cross-modal redundancy. To address this issue, we propose LLaVA-FA, a novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain. By leveraging the de-correlation and conjugate symmetry properties of Fourier transform, LLaVA-FA achieves more compact and accurate weight representations. Furthermore, we introduce PolarQuant, a polar-coordinate quantization method tailored for complex matrices, and an optional diagonal calibration (ODC) scheme that eliminates the need for large-scale calibration data. Extensive experimental results demonstrate that our proposed LLaVA-FA outperforms existing efficient multimodal models across multiple benchmarks while maintaining minimal activated parameters and low computational costs, validating its effectiveness as a powerful solution for compressing LMMs.