QuEPT: Quantized Elastic Precision Transformers with One-Shot Calibration for Multi-Bit Switching

AI summary

Overview

• Research area: Model compression for Transformers — post-training quantization (PTQ), specifically elastic multi-bit-precision quantization spanning vision Transformers (ViTs), large language models (LLMs), and multimodal LLMs (MLLMs). Filed under cs.CV.
• Technical level: Advanced. The paper assumes familiarity with quantization (scales, zero-points, clipping, bit-widths), LoRA/low-rank adaptation, and Transformer block-wise reconstruction.
• Scope: The paper proposes QuEPT, a post-training elastic quantization framework with two new components — Multi-Bit Token Merging (MB-ToMe) and Multi-Bit Cascaded Low-Rank Adapters (MB-CLoRA) — that lets one calibration run produce a model deployable at any bit-width in a predefined set without re-optimization.

What This Paper Is About

Most quantization methods fix a single bit-width at optimization time, so changing precision means re-running the whole optimization. Elastic precision quantization instead optimizes several bit-widths at once against shared weights, but it has been poorly studied for Transformers because attention creates dense inter-token dependencies, dynamic sparsity, wide activation ranges, and highly uneven per-layer bit-width sensitivity. The paper's goal is a one-shot calibration scheme that reconstructs block-wise multi-bit errors and switches between uniform and mixed precision in real time, while preventing the lowest precision setting from dragging down the higher ones.

Key Contributions

1. QuEPT framework: an efficient post-training elastic quantization framework for Transformers whose quantized model supports real-time configuration of different bit widths without repeatedly optimizing parameters; it adapts to predefined bit-widths by cascading different low-rank adapters.
2. Multi-Bit Token Merging (MB-ToMe): reconstructs mixed quantization error in each block by fusing tokens of different precision levels, selectively retaining the most robust high-bit tokens and softly merging the rest, which alleviates contention between bit-widths and improves robustness.
3. Multi-Bit Cascaded Low-Rank Adapters (MB-CLoRA): a hierarchical low-rank parameter-sharing structure where the effective rank grows as bit-width decreases, strengthening correlations between bit-width groups and improving overall quantization performance.
4. Validation across three model families: experiments on ViT, LLM, and MLLM models showing performance comparable to or better than current PTQ methods; the authors describe QuEPT as the first multi-bit-width PTQ method validated on ViTs, LLMs, and MLLMs.

Main Findings

• Vision Transformers (ImageNet, weight-activation): On ViT-S, QuEPT reaches 75.1 at W4A4 versus 68.9 for ERQ (stated as 6.2% higher) and 1.46% higher average accuracy across the five bit-widths, using 17 minutes versus ERQ's 9×N. On ViT-B it reaches 80.7 / 83.3 / 83.8 / 84.2 / 84.3 for W4A4 through W8A8, versus PTMQ's 77.7 / 78.6 / 79.1 at W6A6 / W7A7 / W8A8; the text states improvements of 6.1%, 5.6%, and 5.2% at those settings, with roughly 1/26 of PTMQ's time overhead on a single Nvidia RTX 3090 GPU.
• Single calibration, many bit-widths: single-bit-width baselines need separate optimizations per bit-width, whereas QuEPT requires only one training run; total training time is reported as less than half of ERQ's for ViT-S (17 minutes versus 9×N).
• Swin-S: QuEPT reports 81.9 / 82.8 / 82.9 / 83.1 / 83.2 across W4A4–W8A8, with FP32 listed at 83.2.
• LLaMA weight-activation quantization: On LLaMA2-7B W4A4, QuEPT achieves WikiText2 perplexity 6.33 and C4 perplexity 7.86 with 61.62 average zero-shot accuracy across PIQA, ARC-E, ARC-C, HellaSwag, and WinoGrande, which the paper says beats QuaRot and SpinQuant by 7.07% and 0.28% in average accuracy. At W6A6 it reports 5.53 / 7.03 / 65.48 and at W8A8 5.48 / 6.98 / 66.23, against an FP16 reference of 5.47 / 6.97 / 65.71.
• Larger LLaMA models: LLaMA2-13B yields 5.53 / 7.16 / 66.05 at W4A4 and 4.94 / 6.53 / 69.09 at W8A8 (FP16: 4.88 / 6.46 / 69.35). LLaMA3-8B yields 8.25 / 11.67 / 67.04 at W4A4 and 6.20 / 8.96 / 72.87 at W8A8 (FP16: 6.14 / 8.88 / 72.65).
• Multimodal LLaVA-OneVision-7B: Under W3, QuEPT scores 44.6 / 60.6 / 74.1 / 60.3 / 71.6 on MMMU, OCRBench, TextVQA, VizWiz, and SEED versus MBQ's 42.0 / 61.1 / 73.3 / 60.7 / 66.4 — stated as 1.5% higher average accuracy. Under W4A8 it scores 43.4 / 61.2 / 71.5 / 61.3 / 70.7 versus MBQ's 42.6 / 52.3 / 68.3 / 58.9 / 64.4 — stated as 4.3% higher. It also reports W2 results (32.2 / 47.8 / 64.0 / 55.2 / 67.0).
• Free mixed precision: Because the method is elastic, changing the per-layer LoRA configuration turns it into mixed precision with no extra training. On LLaMA2-7B at average bit-widths 2.25 / 3.00 / 4.00, QuEPT reports WikiText2 perplexity of 8.97 / 5.93 / 5.54, compared against SKIM, SqueezeLLM, and QuIP#.
• Ablation — MB-CLoRA: On LLaMA2-7B W4A4, MB-CLoRA reaches 61.6 versus 60.9 (fully shared) and 59.2 (independent), described as 2.4% and 0.7% higher than the Independent and Fully Shared strategies; at higher bit-widths it is comparable to the independent strategy.
• Ablation — MB-ToMe: Case 3 (selective merge) is best across all bit-widths; on LLaMA2-7B it shows gains of 5.9%, 0.8%, and 0.2% over Case 1 at 4, 6, and 8 bits.
• Ablation — module combination (ViT-S): Baseline (three independent LoRAs with r=4 and MSE loss) gives 69.2 / 80.1 / 81.1 at W4A4 / W6A6 / W8A8; adding clipping, MB-CLoRA, MB-ToMe, and MAE loss together gives 74.7 / 80.7 / 81.1.
• Loss choice: MAE reconstruction loss is used instead of MSE because the experiments report MAE generally yields superior results.
• Calibration cost: Only 1024 unlabeled ImageNet images for ViTs, 128 C4 samples for LLaMA models, and 128 image-caption pairs from the improved COCO Caption dataset proposed by ShareGPT4V for LLaVA-OV.

Methodology in Plain English

QuEPT treats quantization as a block-by-block reconstruction problem. For each Transformer block, it keeps the full-precision weights and the quantization scales frozen, and only learns two things: a low-rank compensation matrix and the weight clipping thresholds. Because the compensation is low-rank, it costs far fewer parameters than the model and, since the adapters sit inside the clipping operator, the LoRA branch adds no overhead at inference.

The target bit-width set (for example {4,5,6,7,8}) is split into three tiers — low, mid, and high. On each training step one bit-width is sampled from each tier, so the block repeatedly sees low, mid, and high precision together, and gradients flow only into the matching slice of the adapters and clipping thresholds. That is what turns five separate optimizations into a single calibration run.

Two components make this work. MB-CLoRA gives every bit-width the same pool of low-rank parameters, but slices it so that the rank used grows as bit-width drops: high-bit uses r_h, mid-bit uses r_h + r_m, low-bit uses r_h + r_m + r_l. Because the higher-precision compensations are leading submatrices of the lower-precision ones, low-bit models inherit what was learned at high precision and add their own correction.

MB-ToMe handles the features that feed the next block. The authors compared three strategies: randomly picking one bit-width's representation per token (Case 1), uniformly averaging low/mid/high features in a 1:1:1 ratio (Case 2), and selective merging (Case 3). Case 3 wins: tokens are ranked by the cosine similarity between their 8-bit and 4-bit representations — a measure of how stable they are to quantization — and the top p% most similar tokens are kept in their high-precision form, while the rest are blended as a weighted average of high, mid, and low features.

The optimization minimizes the mean absolute error between the full-precision block output and the quantized block output summed over the three sampled bit-widths. Large outlier errors are handled by the learned clipping thresholds, which produce a smoother error signal for the low-rank branch to correct.

Why This Matters

The paper targets a practical bottleneck: today a quantized model is usually locked to one bit-width, and moving to another precision means paying the optimization cost again. QuEPT shows that one calibration pass can cover a range of bit-widths across ViTs, LLMs, and MLLMs, and that the same artifact can be reinterpreted as a mixed-precision model per layer without retraining. Its reported training costs (17 minutes for ViT-S, 36 for ViT-B, 38 for Swin-S on a single Nvidia RTX 3090 GPU) are far below the multi-bit baseline PTMQ's 430, 950, and 430 minutes, and it accepts very small calibration sets (128 samples for LLaMA).

Real-world applications:

• On-device and edge inference for vision and language models that must run on resource-constrained devices, where one shipped artifact can be re-tuned to different memory or latency budgets on the fly.
• Dynamic server serving: a deployment could switch precision at runtime as load or power constraints change, rather than maintaining separately quantized model copies.
• Multimodal assistants such as LLaVA-OneVision-7B, where the same quantized backbone serves image-captioning and visual question-answering tasks under different accuracy targets.
• Hardware with heterogeneous support, where one model must satisfy several accelerator bit-width capabilities, and mixed precision is chosen per layer by measured sensitivity.

Industry relevance: the work is a collaboration between Anhui University and iFLYTEK Research, it releases code at https://github.com/xuke225/QuEPT, and it demonstrates the kind of single-GPU, low-calibration-budget PTQ pipeline that is attractive to teams serving LLMs and MLLMs.

Future Directions

• Explicit outlier handling: the authors state they have not explicitly handled outliers in LLMs and that integrating an outlier-mitigation technique such as SpinQuant would likely yield better results.
• Extremely low bit-widths: the paper notes that performance remains constrained at very low bit-widths and requires further exploration. This is visible in the reported W2 and W4A4 results for LLaVA-OneVision-7B and in the larger degradations at W4A4 for LLaMA3-8B.
• Better mixed-precision allocation: the current pipeline uses KL

Authors’ abstract

Elastic precision quantization enables multi-bit deployment via a single optimization pass, fitting diverse quantization scenarios.Yet, the high storage and optimization costs associated with the Transformer architecture, research on elastic quantization remains limited, particularly for large language models.This paper proposes QuEPT, an efficient post-training scheme that reconstructs block-wise multi-bit errors with one-shot calibration on a small data slice. It can dynamically adapt to various predefined bit-widths by cascading different low-rank adapters, and supports real-time switching between uniform quantization and mixed precision quantization without repeated optimization. To enhance accuracy and robustness, we introduce Multi-Bit Token Merging (MB-ToMe) to dynamically fuse token features across different bit-widths, improving robustness during bit-width switching. Additionally, we propose Multi-Bit Cascaded Low-Rank adapters (MB-CLoRA) to strengthen correlations between bit-width groups, further improve the overall performance of QuEPT. Extensive experiments demonstrate that QuEPT achieves comparable or better performance to existing state-of-the-art post-training quantization methods.Our code is available at https://github.com/xuke225/QuEPT