Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs

AI summary

Overview

Research area: Efficient machine learning — post-training quantization (PTQ) of transformer models, with a focus on activation outliers, plus systems-level deployment profiling.

Technical level: Intermediate. The paper assumes familiarity with quantization scale/zero-point mechanics and transformer residual structure, but its central arguments are explained in plain statistical terms (variance, kurtosis, energy concentration).

Scope in one sentence: A reproducible, single-model, single-GPU empirical study of why naive 8-bit activation quantization collapses BERT-base accuracy on QNLI, how activation statistics evolve with depth, and what four mitigation strategies cost in accuracy and in real deployment metrics.

What This Paper Is About

Quantizing transformer weights and activations to 8 bits (W8A8) is an attractive way to shrink and speed up models, but on transformers it often destroys accuracy in a way that does not happen with convolutional networks. The paper reproduces this collapse in BERT-base-uncased fine-tuned on QNLI, traces it to a small set of activation channels that dominate the tensor's dynamic range and get worse with depth, and tests whether mixed precision, Per-Embedding-Group (PEG) quantization, or a proposed percentile-based calibration can fix it. It then measures whether any of those fixes actually buy latency, VRAM, or model-size wins on a consumer RTX 3050 GPU.

Key Contributions

1. A fully reproducible experimental pipeline that validates the W8A8 collapse in BERT-base under controlled calibration conditions, with a single entry point (python oneclick/run_all.py) covering training, quantization, aggregation, profiling, and LaTeX artifact generation.
2. A depth-wise statistical characterization of activations, reporting mean per-dimension variance, kurtosis, and top-1% energy concentration for the embedding layer, layers 1, 2, 4, 10, and 11, and the pooler (layer 12).
3. A systematic ablation of mitigation strategies — mixed precision PTQ, PEG with permutation at three group counts, and a proposed percentile-based range estimation at four percentile thresholds — all evaluated under identical settings.
4. Deployment-oriented profiling on an RTX 3050 (6 GB VRAM), reporting p50 and p95 latency, peak VRAM, and serialized model size for every method, showing that statistical robustness does not translate into runtime gains on that hardware.

Main Findings

• Naive W8A8 collapses: Global W8A8 PTQ drops QNLI validation accuracy from 89.66% (FP32) to 54.33%, a change of -35.33 points.
• Mixed precision nearly recovers accuracy: Retaining FP16 for FFN output projections, residual summation inputs, and attention output projections yields 89.42%, only -0.24 points below FP32.
• PEG recovers only partially: Per-Embedding-Group quantization with K=3 and permutation reaches 66.12%, a drop of 23.54 points.
• Percentile calibration makes things worse: The proposed percentile-based scaling at p=99.9 gives 50.54%, a drop of 39.12 points — larger than naive W8A8.
• PEG is highly sensitive to group count: K=2 gives 49.46%, K=3 gives 66.12%, and K=4 gives 86.18%, indicating a non-linear dependence on whether grouping is fine-grained enough to isolate dominant channels.
• Percentile threshold does not matter: Sweeping p across 99.0, 99.5, 99.9, and 99.99 leaves validation accuracy at roughly 50.54% in all cases.
• Kurtosis explodes with depth: Mean activation kurtosis rises from 9 at the embedding layer to 14 (layer 1), 41 (layer 2), 73 (layer 4), 135 (layer 10), and 271 (layer 11), then falls to 73 at the pooler. A Gaussian has kurtosis 3.
• Variance roughly doubles across depth: Mean per-dimension variance grows from 0.25 (embeddings) to 0.58 (layer 11), with the pooler at 0.54.
• Energy concentrates in a few channels: The top 1% of channels hold 0.15 of total activation energy at the embedding layer, rising to 0.55 at layer 11 and 0.53 at the pooler.
• No latency benefit from INT8: p50 latency ranges only from 58.38 ms (FP32) to 59.12 ms (Percentile); p95 spans 59.08 ms to 59.81 ms.
• No meaningful memory or size benefit: Peak VRAM spans 483.7 MB (FP32) to 486.3 MB, and serialized model size spans 417.7 MB (FP32) to 419.6 MB (PEG).
• Outliers encode signal, not noise: Because clipping as little as the top 0.01% of activations damages accuracy, the dominant channels appear to carry structured, functional information rather than random extreme values.

Methodology in Plain English

The authors fine-tune BERT-base-uncased (110 million parameters) on QNLI, the question-answering subset of GLUE, using AdamW at a learning rate of 5×10⁻⁵, batch size 8, maximum sequence length 128, weight decay 0.01, gradient clipping 1.0, a linear warmup schedule, and a fixed seed of 1000, saving checkpoints every 500 steps. All quantization is post-training: weights go to INT8 everywhere, calibration runs in inference mode on the training split with static range estimation, and only the activation scheme changes between methods.

Four activation strategies are compared. The baseline uses one global min-max scale for the whole tensor. Mixed precision keeps three vulnerable layer types in FP16 while quantizing the rest. PEG splits embedding dimensions into K groups (default K=3), assigns each group its own scale, and permutes dimensions first so outliers are spread across groups rather than clustered; the permutation is inverted after quantization. The proposed percentile method replaces the maximum with a high percentile (default 99.9) when computing the scale, which clips rare extreme values but adds no runtime cost and requires no architectural change.

Accuracy is measured on the 5,463-example validation set, with 104,743 training examples used for calibration. Separately, the authors load the FP32 checkpoint and compute per-layer variance, kurtosis, and top-1% energy on a 64-sample slice of the validation set. For deployment, they profile on an NVIDIA RTX 3050 (6 GB VRAM) with CUDA 12.1 and PyTorch 2.2.2, using batch size 8 and sequence length 128, discarding 100 warmup iterations, timing 500 inference iterations with CUDA events, and reporting p50 and p95 latency alongside peak VRAM and serialized file size.

Why This Matters

Impact on research. The paper reframes transformer PTQ failure as a problem of structured channel dominance that accumulates through residual connections, rather than a problem of rare scalar outliers. This matters because it explains why an obvious fix — percentile clipping — fails in a way that is not just "insufficient tuning": sweeping the threshold from 99.0 to 99.99 leaves accuracy flat at about 50.54%, which is consistent with the clipped channels carrying functional signal. It also supplies a reproducible baseline (Bondarenko et al., EMNLP 2021, Qualcomm AI Research) against which new channel-aware methods can be measured with deployment numbers attached, not just accuracy.

Real-world applications.

• Deploying fine-tuned encoder models for question answering and sentence-pair classification where a 35-point accuracy drop is unacceptable but model size still matters.
• Edge and consumer-GPU inference, where the paper's profiling shows that INT8 arithmetic alone does not deliver speedups without hardware and kernel support.
• Selecting a mitigation strategy under a fixed accuracy budget: mixed precision when accuracy is paramount, PEG with K=4 (86.18%) when more compression pressure is present.
• Building calibration tooling for quantization pipelines, since the paper demonstrates that percentile-based range estimation can be actively harmful for transformer activations.

Industry relevance. The negative deployment result — median latency 58–59 ms and peak VRAM 484–486 MB essentially unchanged across all five configurations on the RTX 3050 — is directly relevant to teams planning quantization-driven cost reductions. It indicates that realized gains depend on whether the inference stack dispatches to INT8-optimized kernels, and that statistical robustness of a quantization scheme is a separate question from deployment efficiency.

Future Directions

1. Scale to decoder-only and billion-parameter models. All experiments used BERT-base-uncased with a largest checkpoint of about 418 MB, so it is unknown whether kurtosis growth and channel dominance intensify linearly, super-linearly, or level off with depth and width.
2. Hardware-aware quantization and edge deployment. The paper measured only one consumer GPU; it did not test data center GPUs (A100, H100), CPU-only inference, mobile NPUs or edge accelerators, or INT8 kernels with tensor-core acceleration. Profiling these would reveal whether the missing speedup is a hardware artifact.
3. Formal analysis of residual amplification. The paper offers empirical evidence (kurtosis rising from roughly 9 to 271, top-1% energy rising from 0.15 to 0.55) but no mathematical theory. Analytical modeling of residual amplification, error-propagation stability bounds, and spectral analysis of dominant channels remain open.
4. Task, seed, and sampling generalization. Results rest on a single task (QNLI), a single seed (1000), and a 64-sample validation slice for outlier statistics. Testing other GLUE tasks, additional seeds, and larger samples would establish whether channel dominance is architectural or task-specific.

Target Audience

Researchers and practitioners working on efficient transformer inference and post-training quantization, especially those who need reproducible baselines and honest deployment measurements rather than accuracy figures alone. It is also useful for systems and MLOps engineers deciding whether INT8 quantization will pay off on a given hardware target, and for graduate students looking for a controlled, well-documented empirical study that combines statistical analysis with a negative deployment result. Readers without prior exposure to quantization mechanics may find the problem-formulation section dense, but the accuracy and profiling tables and the ablation results are accessible on their own.

Authors’ abstract

Post-training quantization (PTQ) of transformers is known to suffer from severe accuracy degradation due to structured activation outliers, as originally analyzed by Bondarenko et al. (EMNLP 2021) in work associated with Qualcomm AI Research. This paper provides a reproducible empirical reproduction and systems-level extension of that phenomenon in BERT-base fine-tuned on QNLI. When global W8A8 quantization is applied, validation accuracy drops sharply from 89.66% (FP32) to 54.33%, a decrease of 35.33 points. Statistical analysis of FP32 activations shows strongly heavy-tailed behavior that intensifies with model depth: kurtosis reaches 271 in the final layers and approximately 55% of activation energy is concentrated in the top 1% of channels. We evaluate several mitigation strategies. Mixed precision PTQ restores accuracy close to the FP32 baseline (89.42%). Per-embedding-group (PEG) quantization shows strong sensitivity to grouping structure, improving accuracy from 66.12% with three groups to 86.18% with four groups. In contrast, percentile-based calibration, even at thresholds between 99.0 and 99.99, fails to recover accuracy (about 50.54%), indicating that large activation channels encode structured signal rather than rare noise. Deployment profiling on an RTX 3050 GPU shows minimal differences in latency and memory usage across methods (median latency about 58-59 ms; VRAM usage about 484-486 MB), highlighting the importance of hardware-aware evaluation. Overall, the results show that PTQ failure in transformers is primarily driven by structured channel dominance amplified through residual connections. Effective mitigation therefore requires channel-aware precision allocation rather than scalar clipping alone.