Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model

AI summary

Overview

Research area: Natural language processing, specifically extreme post-training quantization of large language models, applied to a non-English (Polish) model.

Technical level: Advanced. The paper assumes familiarity with weight quantization, Hessian-based error minimization, lattice codebooks, and rotation-based incoherence transforms.

Scope: A single-researcher comparative study applying six state-of-the-art 2-bit quantization methods to Bielik-11B-v2.3-Instruct, evaluated on 22 Polish benchmarks plus eq_bench.

What This Paper Is About

Nearly all published 2-bit quantization research evaluates English-language models, leaving open the question of whether those methods transfer to morphologically rich languages like Polish, where the model must distinguish fine-grained word forms (the paper gives dom/domu/domowi/domem as an example). This paper quantizes the leading open Polish LLM, Bielik-11B-v2.3-Instruct, using six different extreme-compression methods calibrated on a Polish corpus, and compares the results against the existing community IQ2_XXS baseline. The goal is to determine which method best preserves Polish-language capability at roughly 2 bits per weight, and to document failure modes that standard evaluation misses.

Key Contributions

1. First academic 2-bit quantization of a Polish (and Slavic) language model. Six distinct methods — QuIP#, SpinQuant+GPTQ, ButterflyQuant, QTIP, VPTQ, and AQLM — are applied to Bielik-11B-v2.3-Instruct, which uses a depth-upscaled 50-layer Mistral architecture, 4096 hidden dimensions, and grouped-query attention (32 heads, 8 KV heads).

2. Language-matched calibration on cloud infrastructure. Hessian matrices are generated from CulturaX-PL (512 sequences × 4096 tokens), producing 200 Hessian files totaling 25.6 GB (4 per linear sub-layer type across 50 layers).

3. A documented MC-generation dissociation. Two rotation-based variants preserved multiple-choice log-likelihood quality while producing incoherent autoregressive text, which the paper traces to specific missing runtime transforms.

4. Reproducibility under a small budget. The entire project — six quantization variants, cloud GPU computation, and full evaluation — was completed by one independent researcher for approximately $285 on vast.ai.

Main Findings

• QuIP# E8P12 reaches parity with the community baseline. It scores 71.92% raw average across 22 Polish benchmarks versus 72.07% for IQ2_XXS (a difference of −0.15 pp), with normalized averages of 61.10 versus 61.20. QuIP# wins 11 of 22 tasks, with 8 losses and 3 ties. FP16 quality retention is 93.2% for QuIP# and 93.4% for IQ2_XXS.

• A size premium accompanies the parity. QuIP# E8P12 is 3.26 GB versus approximately 2.6 GB for IQ2_XXS, a compression ratio of 6.7× versus roughly 8.5×.

• QuIP# leads on emotional reasoning and reasoning tasks. On eq_bench it scores 47.14 ± 2.12 versus 43.53 (+3.61), with first-turn responses at 44.70 ± 2.44 versus 39.83 (+4.87) and 99.42% parseable output. Adding eq_bench as a 23rd task flips the aggregate ranking: 60.49% versus 60.43% normalized, with 12 wins to 8.

• The two methods degrade different capabilities. QuIP# is strongest on ppc_mc (+7.23 pp), polqa_closed_g (+5.19 pp), psc_mc (+4.10 pp), polemo2-out_g (+3.84 pp), and dyk_mc (+3.22 pp). IQ2_XXS is strongest on cbd_g (−11.11 pp from QuIP#'s perspective), dyk_g (−5.90 pp), polemo2-out_mc (−5.45 pp), and klej_ner_mc (−4.88 pp).

• Perplexity degradation is well below the literature norm. QuIP# reaches 3.74 on held-out Polish text versus 3.39 for FP16, a +10.3% degradation, where the literature reportedly reports 20–30% as typical.

• QTIP gives the best per-bit efficiency. It achieves 79.4% acc_norm on the 10 multiple-choice tasks at approximately 2.4 bpw and 3.27 GB — matching VPTQ's quality while being 35% smaller. Its combined 22-task raw average of 76.50% exceeds QuIP# by 4.58 pp and IQ2_XXS by 4.43 pp, though the paper flags that MC was measured on the base model and generation on the e2e-finetuned model.

• QTIP's finetuning step contributes substantially. Knowledge distillation reduced cross-entropy loss from 2.2375 to 2.1916 (−2.05%) over 4 epochs, with the best checkpoint at epoch 1. Against IQ2_XXS on 12 generative tasks, QTIP wins 8 of 12 for an average advantage of +2.02 pp, dominating classification and NLI (cbd_regex +37.35 pp, dyk_regex +18.11 pp) but losing badly on open-book QA (poquad_open_book −27.17 pp, polqa_open_book −9.23 pp).

• VPTQ achieves the highest MC score at a higher bitrate. It reaches 79.41% acc / 79.64% acc_norm, but at 5.0 GB and an effective ~3.58 bpw — over 50% more bits than QuIP# or QTIP. The paper calls this "the best reconstruction fidelity of all methods tested," with residual quantization reducing proxy errors by 65–80% versus primary-only. It reports WikiText-2 PPL = 7.818 and C4-new PPL = 15.323 (context 2048).

• AQLM is competitive and uniquely uses adaptive bitwidths. It scores 79.3% acc / 79.3% acc_norm at 3.62 GB (~2.4 bpw, ~6.1× compression). Attention modules receive 2.5–3.0 bits while FFN modules receive 2.3–2.4 bits, averaging ~2.6 bits. It achieves the single highest task score in the study on psc_mc (97.40%).

• Rotation-based methods fail catastrophically at generation. SpinQuant+GPTQ produced incoherent text despite reasonable MC results (DYK MC: 62.88%); three repair attempts failed. ButterflyQuant + E8P produced loops ("Stolica Polski to Polski z Polski z Polski…") and dropped to 41.7% MC average (−14 pp versus Variant A) despite identical E8P codebooks, with a DYK MC anomaly of 75.3%.

• Four unrelated paradigms converge to a quality ceiling. Cross-method MC accuracy on 10 shared tasks is 78.10% (QuIP#), 79.11% (QTIP), 79.41% (VPTQ), and 79.42% (AQLM) — a 1.32 pp spread across lattice VQ, trellis coding, residual VQ, and additive codebooks operating at 2.4–3.58 bpw. Best-per-task wins distribute as QuIP# 2, QTIP 2, VPTQ 2, AQLM 4.

• Named entity recognition is universally the hardest task. klej_ner ranges from 45.72% (QTIP) to 52.38% (VPTQ), a 6.66 pp spread, while psc ranges only 1.20 pp (96.20% to 97.40%).

• Decompression to FP16 is nearly lossless. Variant A decompression achieved cosine similarity 0.999993–0.999995, identical top-1/top-5 predictions, and maximum absolute difference of 0.016–0.078.

Methodology in Plain English

The researchers took one Polish language model and compressed it six different ways, then ran the same Polish test suite against each compressed version.

The shared foundation was calibration data. For quantization to work, the algorithm needs to know which weights matter most, which it estimates from a Hessian matrix built by feeding real text through the model. All six variants used the same 512 sequences of 4096 tokens from CulturaX-PL, so any differences in final quality come from the quantization method rather than from the calibration text. This was done on an H200 GPU in about 40 minutes; an earlier local attempt with 128 samples on an RTX 5070 took roughly 12 hours overnight, an 18× speedup. Notably, the author initially generated Hessians from the base model (FP16 = 58.14%) rather than the Instruct variant (FP16 = 65.71%) and had to regenerate everything, since the IQ2_XXS baseline quantizes the Instruct model.

Each variant then applied a different compression strategy: QuIP# uses a lattice codebook with 2^16 codewords and a random Hadamard rotation; SpinQuant learns rotation matrices instead of using random ones; ButterflyQuant learns per-layer Givens rotation angles; QTIP replaces the lattice with trellis coded quantization over 256 dimensions; VPTQ uses residual vector quantization with a secondary codebook for errors; AQLM represents weight vectors as sums of entries from multiple codebooks and allocates bits adaptively per module.

Because QuIP# natively supports only the Llama model class, the author patched it to convert MistralConfig to LlamaConfig (mapping sliding_window to None, adding attention_bias = False, requiring 8 getattr fixes). The embedding layer and LM head (~0.52 GB) were kept in FP16 to preserve the full tokenizer vocabulary. Evaluation used the Open Polish LLM Leaderboard, covering 10 multiple-choice tasks scored by log-likelihood and 12 generative tasks scored by text generation and pattern matching, plus eq_bench, which a SpeakLeash collaborator evaluated on the decompressed model since dataset access was not granted. Scores were normalized with the leaderboard's official formula; an early pipeline error using random-chance baselines had inflated normalized scores by about 8 pp (69.3% → 61.10%).

Why This Matters

The paper matters less for proposing a new algorithm than for establishing what actually happens when established algorithms meet a non-English model — and for documenting a failure mode that standard benchmarks hide. Its central analytical claim is a "method-independent quality ceiling": four completely unrelated compression paradigms landing within roughly one point of each other on the same evaluation, which suggests that further algorithmic sophistication may not buy much without raising the bitrate or applying task-specific tuning. The MC-generation dissociation is arguably the more immediately useful finding: a model can look healthy on multiple-choice benchmarks while being unusable in production.

Real-world applications:

• On-device and edge deployment of Polish LLMs. Compressing Bielik-11B-v2.3-Instruct from 22.0 GB to 3.26 GB (6.7×) puts a Polish 11B model within reach of 4 GB consumer GPUs; AQLM's 3.6 GB inference footprint is cited as deployable on GPUs with as little as 4 GB memory.

• Task-specific method selection. Because QuIP# is better at reasoning and emotional understanding while IQ2_XXS is better at classification and NER, a practitioner whose application is, say, cyberbullying detection or named entity recognition can pick based on the error profile rather than a single average score.

• Low-resource language preservation. The demonstration that language-matched calibration is cheap (~40 minutes of H200 time, roughly $2–$4) provides a template for other morphologically rich or under-resourced languages.

• Quantization evaluation standards. The paper's recommendation that autoregressive generation testing be mandatory for weight-transformed quantized models is directly actionable for anyone benchmarking compressed models.

Industry relevance: The convergence finding is a resource-allocation signal — a quantization team choosing among these four methods for a comparable model is unlikely to gain MC accuracy by switching, which shifts investment toward bitrate, fine-tuning, or evaluation changes instead. The cost data (~$285 total, with individual stages ranging from under $1 to ~$50) also demonstrates that this class of research is now accessible outside well-funded labs.

Future Directions

• Complete the missing evaluations. Variant F (AQLM) generative evaluation was not completed, and 0-shot results are unavailable for all variants since only 5-shot was run. A fair 22-task comparison for QTIP also requires either generating on the base model or running MC on the finetuned model.

• Implement runtime inverse transforms. Variant B's failure traces to R3 (KV cache Hadamard) and R4 (FFN down-projection Hadamard), which must be applied at every decoding step and which no public engine implements. Building one could rescue the rotation-based approaches.

• Run the Polish versus English calibration ablation. The paper cites indirect evidence for language-matched calibration — the +10.3% perplexity degradation being below the 20–30% literature norm, Polish's 7 grammatical cases, and strong eq_bench results — but explicitly states the controlled comparison is future work.

• Test whether the ~79% MC ceiling generalizes. The authors propose investigating whether this convergence holds for other Slavic language models, and applying the pipeline to larger Polish models. Inference speed benchmarking is also listed as unaddressed.

Target Audience

Practitioners deploying compressed LLMs who need concrete guidance on which 2-bit method to pick and what size-quality tradeoff to expect. Researchers working on quantization will find the MC-generation dissociation and the cross-method convergence argument most relevant. NLP researchers focused on Polish or Slavic languages get the first systematic reference point for extreme compression in that domain. Independent researchers and small teams benefit from the detailed budget, hardware, and debugging record, including the four bugs fixed in Variant C, the three in Variant E, and the base-versus-Instruct Hessian mistake — though the paper explicitly warns of systematic error risk from having a single researcher.

Authors’ abstract

We present Bielik-Q2-Sharp, the first systematic academic evaluation of extreme 2-bit quantization applied to a Polish large language model. Using Bielik-11B-v2.3-Instruct (11B parameters, Mistral architecture) as our base model, we compare six state-of-the-art post-training quantization methods -- QuIP#, SpinQuant+GPTQ, ButterflyQuant, QTIP, VPTQ, and AQLM -- all calibrated on a Polish-language corpus (CulturaX-PL) with shared Hessian matrices. Our best variant (QuIP# E8P12) achieves 71.92% across 22 Polish benchmarks versus 72.07% for the IQ2_XXS baseline -- within statistical noise, at a modest size premium (3.26 GB vs. ~2.6 GB). On eq_bench, our method scores 47.14 versus 43.53 (+3.6pp), suggesting superior preservation of higher-order reasoning. QTIP achieves the best per-bit efficiency (79.4% MC acc_norm at ~2.4 bpw, 3.27 GB), matching VPTQ's quality at 35% smaller size. We additionally document a MC-generation dissociation phenomenon where rotation-based methods preserve log-likelihood quality but fail catastrophically at autoregressive generation. The entire project was conducted by a single independent researcher on cloud GPUs (vast.ai) within a $285 budget. All models, Hessians, and evaluation logs are publicly available.