Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them

AI summary

Overview

Research area: Software engineering for machine learning systems — specifically supply-chain quality assurance for quantized large language model (LLM) artifacts distributed through public model registries.

Technical level: Intermediate. Readers need some familiarity with LLM quantization formats and benchmarking, but the paper is structured around software-testing concepts (acceptance gates, provenance, reproducibility) rather than quantization mathematics.

Scope: An execution census of 305 eligible GGUF tags in the official Ollama library plus 22 Hugging Face community files, combining a 15-task screening suite, full 164-task HumanEval+ evaluation, and tensor-level recipe-matched rebuilds to determine whether published quantized files work and, when they do not, why.

What This Paper Is About

Developers routinely download quantized model files from public registries such as Ollama and Hugging Face to run LLMs locally, but none of the registries examined documents any functional testing of those files before publication. This paper measures how often published quantized artifacts simply do not work, then traces each confirmed failure back to its cause by rebuilding the file from the original checkpoint with the recipe its metadata implies. The central argument is that a file can be an exact, bit-faithful reproduction of its quantization recipe and still be functionally broken — so reproducibility checks alone are not an adequate quality gate.

Key Contributions

1. An execution census with a released audit dataset. The study executes 305 eligible tags of the official Ollama library — 291 of them quantized and 14 unquantized fp16 builds, spanning 35 model variants in 15 model lines — plus 22 community files from 12 popular GGUF repositories on Hugging Face, and releases every census transcript.

2. A recipe-matched attribution procedure. Introduced and demonstrated on five failing artifacts, it rebuilds a failing file from the original checkpoint using the uncalibrated recipe the file's metadata indicates (no quantize.imatrix keys), then compares the two tensor for tensor. This separates file defects from recipe, model, and backend effects.

3. Evidence that exact reproducibility does not imply function. Five failing registry files match the authors' independent uncalibrated rebuilds tensor for tensor (434 tensors for the four Qwen tags, 197 for phi3.5 once its 32 attn_qkv tensors are matched), yet score zero on the benchmark. For Qwen2.5-Coder-3B, adding an importance matrix lifts three of four failing levels from zero to 76–113 of 164 HumanEval+ tasks.

4. A screening gate with measured limits. The 15-task screen costs 12 to 105 seconds of generation per artifact (median 30 s) on one consumer GPU. None of the 66 sampled tags that passed it is non-functional, and the five rebuilt failures remain flagged under alternative generation settings and suite-size checks. The screen, scorer, and rebuild/calibration/tensor-comparison scripts are released as the quantcheck tool.

Main Findings

• Seven of 305 official tags (2.3%) are non-functional. All seven solve at most six of 164 HumanEval+ tasks; six solve none. They form three incidents: four Qwen2.5-Coder-3B tags at adjacent levels (q2_K, q3_K_S, q3_K_M, q3_K_L), one phi3.5-mini q2_K tag, and two llama3.2-1B tags (q2_K, q3_K_S). At the variant level, 3 of 35 model variants (8.6%) are affected.

• All failures are low-bit tags of small models. Every one of the seven sits among the 48 tags at q2_K to q3_K_L in models of at most four billion parameters, a subgroup rate of 14.6% described after the fact. Wilson 95% intervals are 1.1–4.7% at the tag level and 3.0–22.4% at the variant level.

• None of the five rebuilt failures is a corrupted file. Library files and independent rebuilds give byte-identical answers on all 15 tasks at every level, and files re-downloaded in October 2026 are byte-identical to the August census copies. Each is an exact, uncalibrated llama.cpp quantization of the released weights.

• Calibration rescues four of five, but only partly. With weights, toolchain and tensor mix held fixed, the authors' importance matrix takes Qwen q2_K from 0 to 106–113 of 164, q3_K_M to 76–89, and q3_K_L to 80–86. For phi3.5, the current llama.cpp mix alone lifts the plain rebuild from 0 to 21 of 164, and the imatrix lifts it from 21 to 37 (McNemar p = 0.007). Qwen q3_K_S stays weak under every calibration at 13–30 of 164.

• The same recipe that breaks 3B leaves 1.5B functional. Qwen2.5-Coder-1.5B tags from q2_K to q3_K_L match the authors' uncalibrated rebuild in all 338 tensors and score 13–15 of 15, although the 1.5B q2_K tag solves only 10 of the 49 RQ4 audit tasks against 28 for its q4_0 tag. Recipe alone therefore does not predict failure, and the toolchain does not guard against it: llama.cpp refuses its very lowest-bit types without an importance matrix but exempts Q2_K and the Q3_K mixes by design.

• The registry's 4-bit tags are markedly better. Qwen's 4-bit tag solves 132 of 164 and phi3.5's solves 74, against the best calibrated low-bit builds at 113 and 37 respectively.

• Failures are silent, not crashes. Non-functional builds stream text at normal speed; the Qwen q2_K answer to the one-line strlen task degenerates into repetitive text until it hits the token cap, and the q3_K_M answer is byte-identical to the plain rebuild's — fluent preamble, signature, docstring, and no body.

• Output statistics catch only some failures. Truncation at or above 0.2 flags 3 of 7 suspects but also 10 of the other 298 tags; mean 6-gram repetition at or above 0.2 flags 3 suspects and 4 other tags; either at 0.3 or above catches 4 suspects and 6 other tags, four of which are deepseek-coder-1.3B tags solving 2 to 16 of 164. The rule misses Qwen q3_K_M, q3_K_L and llama3.2-1B q3_K_S.

• Perplexity flags every measured non-functional build. Each of the five non-functional library builds measured has 27 to 5,700 times the WikiText-2 perplexity of its fp16 model (Qwen fp16 at 11.5, library q2_K at 66,391), while 4-bit tags stay within about 5% (4.2% and 5.1%). Thresholds of 10 to 25 times fp16 would flag all five without executing generated code, but the range was chosen after seeing the builds, needs an fp16 reference, and cannot distinguish a corrupted file from an uncalibrated one.

• The screen held up as an acceptance gate. All seven named suspects solve at most 6 of 164. Of 66 audited tags that passed the screen, none is non-functional — each tested file solves at least 17 tasks — giving an exact stratified 95% upper bound of 22 non-functional census files (9.9%) across the 223 other tags.

• One group escapes the relative screen entirely. All 12 deepseek-coder-1.3B tags, fp16 included, score 1 to 6 of 15; the fp16 tag solves 16 of 164 against the 65.2% on HumanEval its authors report. Their files lack a pre-tokenizer type, and 59% of the group's screen outputs do not parse (about half on indentation) against 2.4% for the other 286 official tags and the seven suspects. The group is left unattributed because deepseek-coder-6.7B and starcoder2-15B, the other 14 of the 26 tags without a pre-tokenizer type, work.

• A community pair fails for a different reason. Two TheBloke deepseek-coder-6.7B files at Q3_K_M and Q4_K_M score 5 and 6 of 15 without a supplied template, because the runtime falls back to generic ChatML; with the model's instruction template they score 13 and 15 of 15 and their full scores rise from 16 to 98 and 18 to 101 of 164. These are recorded as template-dependent screen failures, not file defects.

• Distributor identity is unreliable as a reference. The phi3.5 library file is byte-identical to two Hugging Face uploads under unrelated names and tensor-identical to a third, so one artifact reaches users through four storefronts; only a tensor comparison exposes the third. The paper's own first version, which chose reference conversions by distributor rather than by recipe, misdiagnosed these tags as defective conversions.

• A scoring fault was found and corrected. The census scorer marked HumanEval/15 as failed for every artifact; after serial re-evaluation it passes for 275 of the 305 official artifacts, reducing stage-one flags from 50 to 39 without changing any suspect or attribution verdict.

Methodology in Plain English

The authors treat a model registry like a software package registry and ask a simple question: does the thing users download actually work?

They first enumerate every eligible tag in a chosen set of Ollama model families — instruct, code-capable builds with an explicit quantization suffix and a file of at most 8 GB — giving 305 artifacts. Because that is the whole eligible population within those lines, it is a census rather than a sample. They add 22 files at Q2_K to Q4_K_M from 12 popular Hugging Face GGUF repositories.

Every artifact runs a short 15-task code screen derived from HumanEval+ tasks that all 23 healthy Qwen2.5-Coder configurations in a prior full-benchmark evaluation solved. The screen has two stages: flag anything passing fewer than 9 of 15, then compare against the median score of the same model variant at high precision (q6_K, q8_0 or fp16), classifying each artifact as healthy, suspect, low baseline, degraded, or no baseline. A suspect is called non-functional only if it also solves under 10% of all 164 HumanEval+ tasks.

Suspects are then investigated by attribution. The authors download the original checkpoint, convert it to GGUF, and quantize it themselves at the same level without calibration — the recipe the registry metadata implies. They compare the downloaded file and their rebuild tensor by tensor, checking type, shape and bytes. If the two match but both fail, the file is not corrupt; the recipe is the problem. They then build a calibrated version using an importance matrix (llama.cpp's activation-statistics-driven weighting, derived from 100 chunks of 512 tokens of WikiText-2 training data or 1.5 MB of Python source from the llama.cpp repository) and see whether scores recover. Third-party calibrated builds from bartowski and Qwen's Instruct-AWQ files inform the verdict but never decide a file defect.

Generation runs on Ollama 0.32.6 (bundled llama.cpp 96278e3), greedy decoding with seed 42, context 2,048 and at most 768 new tokens, on an NVIDIA RTX 4090 with CUDA, with replication on an Apple M2 with Metal. Scoring uses EvalPlus 0.3.1 plus-tests, one evaluation at a time. A repeat on a second RTX 4090 reproduced all 225 answers from 15 census tags byte for byte.

Finally, they test the screen itself: precision, misses (via a 49-task audit on sampled passing tags), population bounds, robustness to suite choice, robustness to generation settings, and cost.

Why This Matters

Impact on research. Studies that use registry tags as stand-ins for a model at a given precision can misread a recipe problem as a fundamental precision limit. The paper shows that a calibrated Q2_K build solves about two thirds of HumanEval+ where the published one solves nothing — so "2-bit quantization destroys code ability" may be a statement about an uncalibrated default, not about bit width. It also supplies a reusable attribution protocol: execution alone cannot separate a bad file from a bad recipe, but a recipe-matched rebuild can.

Real-world applications:

• Registry operators can add a ~30-second-per-tag screen at publication and withhold or recalibrate failures, rather than serving non-functional files unchanged byte for byte over months with no warning and no recorded recipe.
• Practitioners deploying locally can run the same cheap suite against the exact backend and chat template they will use, which is what caught the deepseek-coder-6.7B template mismatch that a generic ChatML fallback caused.
• Downstream tooling and CI pipelines can use the quantcheck scripts as an acceptance test for model artifacts, analogous to package-registry scanning and continuous-integration tests.
• Model publishers can decide per model-and-recipe pair whether a level is safe to ship — the 1.5B model stays functional at every level where the 3B model fails — and can record effective bits per weight, since nominally low-bit 0.5B files silently fall back to 4-bit types.

Industry relevance. Self-hosted LLM deployment is large and growing: prior measurements cited in the paper found more than 320,000 publicly reachable self-hosted LLM services, including 155,423 Ollama instances, and a year-long measurement saw 152,137 distinct IP addresses exposing Ollama. The failure mode described is silent — fluent, normal-speed output with no body — so users get no error signal. Download counts do not establish reliability, and Qwen's own quantization guide already recommends AWQ scales or an imatrix and notes that a mixture sometimes "just does not work," while Qwen's GGUF files for this model carry Instruct-AWQ labels and include only two of the four low-bit levels.

Future Directions

• Tune the perplexity pre-check on more models. The 10-to-25-times-fp16 threshold was chosen after inspecting these builds, needs an fp16 reference to exist, and has an unknown false-positive rate because the llama3.2-1B pair and other tags were not measured.

• Attribute the llama3.2-1B pair. Its two non-functional tags were not rebuilt because the checkpoint is license-gated, and the only available referees (unsloth's files) carry imatrix keys and fail too, at 1 and 4 of 15 — so recipe, model and file cannot currently be separated.

• Explain the residual gap in the Q3_K builds. Calibrated Q2_K builds have about 1.3 times the fp16 perplexity and solve 106–113 tasks, while calibrated Q3_K_M and Q3_K_L builds keep about 6 times the fp16 perplexity and solve fewer; the cause is not identified, and the imatrix comparison also bundles in llama.cpp's switch to a weighted scale search.

• Broaden beyond Python code generation and one registry. The screen tests Python only, its tasks are easy by construction and were selected from the model family where four of the seven failures occur, and the community sample is small and popularity-selected — so a mature gate needs per-language, per-backend and per-template coverage, plus a publicly preregistered attribution plan.

Target Audience

Model registry operators and platform engineers who decide what gets published; ML infrastructure and MLOps engineers who build artifact acceptance tests; practitioners who download GGUF files for local deployment and want to know which checks are worth running; and quantization researchers who need to distinguish a recipe failure from a precision limit before drawing conclusions about bit width.

Authors’ abstract

Developers download quantized large language models (LLMs) from public registries to run them locally. None of the registries we examined documented functional testing of their files. We tested all 305 eligible code-capable GGUF tags (the llama.cpp format) across 15 model lines in the official Ollama library, plus 22 popular Hugging Face community files. A 15-task code screen identified suspects, which we retested on all 164 HumanEval+ tasks. Seven official tags (2.3%) solved at most six of them. All were low-bit quantizations of models with at most four billion parameters. We rebuilt five suspect files from the original weights. All five registry files matched our uncalibrated rebuilds tensor for tensor, so none was corrupted. Exact reproducibility did not imply function. With the weights, toolchain and tensor mix held fixed, adding an importance matrix raised Qwen2.5-Coder-3B scores at three quantization levels from zero to 76-113 of 164 tasks. The same uncalibrated recipe left the smaller Qwen2.5-Coder-1.5B functional, showing that the recipe alone does not predict failure. None of 66 sampled tags that passed the screen was non-functional. We release the audit data and the quantcheck tool and recommend screening low-bit tags before publication, then calibrating or withholding failures.