Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization

AI summary

Overview

Research area: Post-Training Quantization (PTQ) for Large Vision-Language Models (VLMs), with a focus on error reconstruction using Mixture-of-Experts (MoE).

Technical level: Intermediate. The paper assumes familiarity with Transformer internals, weight/activation quantization bit-width notation (W4A6, W4A8, W3A16), low-rank adapters, and spectral clustering.

Scope: The paper proposes Quant Experts (QE), a token-aware adaptive error-compensation framework that splits quantization-sensitive channels into token-independent and token-dependent groups and reconstructs each group with a different kind of expert.

What This Paper Is About

Low-bit quantization compresses a VLM's weights and activations to save memory and compute, but it introduces numerical errors that hurt accuracy. Existing PTQ methods identify "important" or outlier channels once, using calibration data, and compensate for them globally with a single fixed correction. The authors show that the set of important channels actually shifts from input to input — across modalities and even between individual tokens inside one modality — so a single global correction is a poor fit. QE addresses this by learning one stable shared correction plus several routed corrections that are selected per token at inference time.

Key Contributions

1. An empirical observation about channel importance dynamics. The authors show that the positions and occurrence frequencies of important channels vary significantly across modalities and among tokens, even within the same modality, which undermines statically calibrated and globally compensated PTQ methods.

2. A token-aware MoE quantization framework (QE). Important channels are partitioned into token-independent and token-dependent sets. A fixed shared expert (a low-rank adapter) compensates global quantization error, while routed experts (multiple routed low-rank adapters) compensate local, token-specific error.

3. A co-occurrence-based construction procedure for routed experts. Token-dependent channels are clustered via normalized pointwise mutual information (NPMI), normalized Laplacian eigen-decomposition, and K-Means, with a lightweight router selecting the lowest-error expert per input token.

4. Broad empirical validation. Experiments across quantization settings and model scales report consistent accuracy gains, including up to a 5.09% accuracy improvement under W4A6 quantization for the 72B model.

Main Findings

• Important channels move per token and per modality. Using top-k importance scores computed from row-mean weight magnitudes (Eq. 1) and per-token activations (Eq. 2), the authors visualize value distributions and important-channel positions in a Qwen2VL-2B Transformer block. Positions shift both across modalities and among tokens within a modality.

• Importance frequency is highly uneven. Measured with Eq. 3, only a small subset of important channels appears across most tokens, while the majority are activated only for specific tokens. The authors note that low-frequency important channels can still carry large outlier magnitudes, making their contribution to error compensation non-negligible.

• W4A6 gains on Qwen2VL-2B. QE improves accuracy by 4.01% over the modality-aware baseline MBQ, with only a 4.23% drop from full precision. Its average score under W4A6 is 58.74 versus 62.97 at FP16, compared with 54.73 for MBQ, 55.92 for LQER, 50.27 for SmoothQuant, and 53.62 for RTN.

• W4A6 gains on InternVL2. The paper reports a 3.13% gain on InternVL2-2B; the accompanying table (Table 2) reports results for InternVL2-8B, where QE reaches 68.13 average under W4A6 versus 65.00 for MBQ and 65.29 for LQER, against a 70.60 FP16 baseline.

• W4A8 is close to full precision. The performance gap to full precision narrows to within 2%, which the authors describe as strong robustness. On Qwen2VL-2B, QE records 61.14 average under W4A8 versus 62.97 at FP16.

• The 72B result is the headline number. On Qwen2VL-72B under W4A6, QE achieves an average accuracy improvement of 5.09%, nearly matching full-precision performance. On individual tasks it scores 58.11 on MMMU, 76.60 on OCRBench, 90.33 on ScienceQA, 79.27 on TextVQA, and 73.91 on VizWiz, against FP16 values of 61.44, 78.70, 91.22, 82.26, and 76.27.

• Both expert types are necessary. Ablations on Qwen2VL-2B at W4A6 show MMMU falling from 36.89 (full QE) to 34.56 with routed experts only and 35.22 with the shared expert only. Replacing learned routing with random routing (35.89) or replacing co-occurrence clustering with random clustering (35.33) also degrades results, confirming that both the router and the clustering scheme matter.

• Refinement of routed experts helps. With the optional layer-wise refinement applied to routed experts and the router, Qwen2VL-2B under W4A6 improves MMMU from 33.78 to 36.89 and VizWiz from 59.62 to 60.58. On Qwen2VL-7B, OCRBench rises from 73.00 to 74.60 and TextVQA from 71.63 to 77.58, though MMMU dips from 45.44 to 44.00.

• More routed experts help, at a memory cost. Increasing N_r on Qwen2VL-2B under W4A6 raises the average of OCRBench, TextVQA and VizWiz from 67.08 (N_r=2) to 67.35 (4), 67.83 (8), and 68.06 (16), with higher memory overhead for routed experts.

• Overhead is small. Per-layer complexity grows from sd² to sd² + sd(2r + N_r) in computation and from d² to d² + rd(1 + N_r) in memory.

• Kernel-level speedups. Using an analytical performance model following the FlightLLM accelerator architecture with Qwen2VL-7B weight shapes and sequence length 128, QE shows 3.5× to 4.5× prefill-stage speedups on linear layers. For the 3584×3584 shape the speedups are 3.56× (W4A6), 3.50× (W4A8), and 4.10× (W3A16).

• Weight-only quantization results. Under W3A16, QE also outperforms MBQ and LQER across models. The authors attribute MBQ's limited benefit over AWQ to capacity limits and LQER's modest recovery to its static compensation.

Methodology in Plain English

The method starts from a calibration pass. The authors run 128 randomly sampled image-caption pairs from the ShareGPT4V enhanced COCO Caption dataset through the model and record the input activations of every layer.

For each layer, they score channels by combining the row-wise mean absolute weight with the token's activation magnitude, then take the top-k channels per token. Counting how often each channel appears in these top-k lists over the whole calibration set gives an occurrence frequency. Sorting channels by frequency splits them into two disjoint sets: the first k channels are "token-independent" (they matter for nearly every token), and the next N_r·k channels are "token-dependent" (they matter only for certain tokens).

Each set gets its own correction mechanism, both implemented as low-rank adapters:

• The shared expert reconstructs global error from token-independent channels using whitening SVD, with token-independent channels exempted from direct quantization, plus channel-wise scaling that shrinks activation magnitudes while proportionally amplifying weights.
• The routed experts handle token-dependent channels. Because the paper argues a separate correction per token is computationally infeasible, it instead groups channels that tend to be important for the same tokens. It builds a binary co-occurrence matrix, converts it to a similarity matrix with normalized pointwise mutual information, computes eigenvectors of the normalized Laplacian, and runs K-Means to form N_r clusters. Each cluster gets its own weighted, rank-truncated low-rank adapter.

At inference, a lightweight router — whose weights are set from the mean absolute residual error of each routed expert — picks the expert predicted to yield the lowest error. The shared expert always runs.

An optional refinement stage trains only the routed experts and the router, layer by layer, with all other parameters frozen. Experimental configuration: total SVD rank r = 64 split as 64/2 per expert type (matching LQER's total rank), k = 32, N_r = 8, refinement over 16 epochs with 100 iterations each using AdamW at learning rate 1×10⁻⁴ with no weight decay and a cosine annealing schedule, and coefficients τ = 0.5, α = 1.0, β = 0.05.

Why This Matters

Impact on research. The paper reframes quantization error compensation as a dynamic, input-conditioned problem rather than a static calibration problem. It connects PTQ to Mixture-of-Experts routing and spectral clustering of channel co-occurrence, offering a template that could be applied beyond VLMs to any network where the identity of outlier channels drifts with the input. It also argues directly against the assumption underlying SmoothQuant-, AWQ-, SpQR-, and LQER-style global compensation.

Real-world applications (implied by the evaluated tasks and settings):

• On-device or edge deployment of VLMs for document and infographic comprehension (DocVQA, InfoVQA) where memory is tight.
• Assistive and accessibility applications built on visual question answering over user-captured photos (VizWiz-VQA).
• Chart and scientific-diagram reasoning assistants (ChartQA, AI2D, ScienceQA).
• Multi-image and multimodal understanding agents (MMStar, MuirBench) that need to fit larger models into fixed hardware budgets.

Industry relevance. Recovering accuracy at 4-bit weights with 6- or 8-bit activations — and reporting prefill-stage kernel speedups of 3.5× to 4.5× in an analytical accelerator model — is directly relevant to teams trying to serve VLMs at reduced cost. The paper's emphasis on keeping overhead small (a marginal increase over the original linear layer's FLOPs and parameter count) speaks to practical deployability rather than benchmark-only gains.

Future Directions

• Scaling the routing design. The ablation shows accuracy improving monotonically as N_r grows from 2 to 16, but memory overhead grows with it. Finding the efficiency frontier — or compressing routed experts — is an open problem the paper does not resolve.

• Refinement stability. On Qwen2VL-7B, refinement improves OCRBench, ScienceQA (79.87 to 80.61), TextVQA, and VizWiz but lowers MMMU from 45.44 to 44.00, suggesting the refinement objective may trade off some capabilities. A more targeted refinement loss is a natural next step.

• Generalization beyond the tested families. Experiments cover Qwen2VL (2B, 7B, 72B) and InternVL2 (2B, 8B). Whether the token-independence partition and co-occurrence clustering hold for other architectures, encoder-only VLMs, or video-language models is not reported. Note also that the abstract describes a range of "2B to 70B parameters," while the tables list 72B for Qwen2VL.

• Interaction with QAT and joint quantization. The supplementary material mentions results for joint quantization of the visual encoder and the VLM, and an extended ablation on the number of important channels; extending the dynamic-compensation idea into quantization-aware training or into the visual encoder alone are open avenues.

Target Audience

This paper is most useful to researchers and engineers working on model compression and efficient inference for multimodal models — particularly those already familiar with PTQ pipelines such as GPTQ, AWQ, SmoothQuant, or LQER. It will also interest practitioners deploying VLMs under tight memory or latency budgets, and researchers studying Mixture-of-Experts architectures who want to see routing applied to quantization error rather than to model capacity. Readers with only a beginner-level background in quantization will find the observation sections (Section 2) accessible, but the method and ablation sections assume comfort with SVD, spectral clustering, and quantization bit-width conventions.

Authors’ abstract

Post-Training Quantization (PTQ) has emerged as an effective technique for alleviating the substantial computational and memory overheads of Vision-Language Models (VLMs) by compressing both weights and activations without retraining the full model. Existing PTQ methods primarily rely on static identification and global compensation of sensitive or outlier channels, yet they often overlook the distributional differences of these important channels across inputs, leading to unsatisfactory quantization. In this work, we observe that the distributions and occurrence frequencies of important channels vary significantly both across modalities and among tokens, even within the same modality. Accordingly, we propose \textbf{Quant Experts (QE)}, a token-aware adaptive error compensation with mixture-of-experts for VLMs quantization. QE divides the important channels into token-independent and token-dependent groups. For the former, a shared expert is designed for most tokens to compensate for global quantization error using a low-rank adapter. For the latter, routed experts including multiple routed low-rank adapters are elaborated to compensate for local quantization error related to specific tokens. Extensive experiments demonstrate that QE consistently enhances task accuracy across various quantization settings and model scales, ranging from 2B to 70B parameters, while maintaining performance comparable to full-precision models.