Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them
Abstract
Quantized small language model files in public registries can be non-functional due to uncalibrated quantization recipes, and screening plus calibration is recommended before release.
Developers download quantized large language models (LLMs) from public registries to run them locally. None of the registries we examined documented functional testing of their files. We tested all 305 eligible code-capable GGUF tags (the llama.cpp format) across 15 model lines in the official Ollama library, plus 22 popular Hugging Face community files. A 15-task code screen identified suspects, which we retested on all 164 HumanEval+ tasks. Seven official tags (2.3%) solved at most six of them. All were low-bit quantizations of models with at most four billion parameters. We rebuilt five suspect files from the original weights. All five registry files matched our uncalibrated rebuilds tensor for tensor, so none was corrupted. Exact reproducibility did not imply function. With the weights, toolchain and tensor mix held fixed, adding an importance matrix raised three Qwen2.5-Coder-3B builds from zero to 76-113 solved tasks. The same uncalibrated recipe left the smaller Qwen2.5-Coder-1.5B functional, showing that the recipe alone does not predict failure. None of 66 sampled tags that passed the screen was non-functional. We release the audit data and the quantcheck tool and recommend screening low-bit tags before publication, then calibrating or withholding failures.
Community
We tested all 305 eligible code-capable GGUF tags across 15 model lines in the official Ollama library, plus 22 popular Hugging Face community files. A 15-task code screen flagged suspects, which we retested on all 164 HumanEval+ tasks. Seven official tags solve at most 6 of 164. All are low-bit quantizations of models with at most 4B parameters. The five we rebuilt match the registry files tensor for tensor, so none was corrupted. Exact reproducibility did not imply function. With the weights, toolchain and tensor mix held fixed, adding an importance matrix lifts Qwen2.5-Coder-3B from 0 to between 76 and 113 of 164 tasks at three quantization levels. The same uncalibrated recipe leaves Qwen2.5-Coder-1.5B functional, so the recipe alone does not predict failure. The audit data is on the Hub as aditiPatodiya/quantcheck-audit, and the screening tool and rebuild scripts are in the linked repository.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains (2026)
- Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit (2026)
- Saying, Not Knowing: Aggressively GGUF-Quantized Small Language Models Still Write Rare Words They Can No Longer Define (2026)
- Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving (2026)
- Shared Stopping Decisions Change Answers in HQQ Cache Quantization (2026)
- Measuring the Energy Cost of Completing Tasks with Quantized Language-Model Agents (2026)
- AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
I've had this bite me with a 4-bit of a 7B I grabbed for an edge box — it answered chat fine, then every tool call came back as near-JSON with a stray comment in it, so my parser threw and the agent looped. Nothing in the card hinted at it. The failure mode wasn't "can't code," it was "can't hold a schema," and no benchmark on that repo would have told me.
So the clustering you found matches what I'd expect, but a code screen is the wrong tripwire. Small quants lose structured output and mid-context recall well before they lose HumanEval ability, and those two things are what actually break a deployment. What I want from a registry is ugly and boring: a fixed prompt suite — JSON schema conformance, a needle at 50% context, a couple of tool-call round trips — scored per file at upload and shown as numbers, not a "quantized" badge. Even a pass/fail per prompt would save people a weekend.
Thanks for the note. The screen is built to catch files that are broken outright. The seven we found solve at most 6 of 164 tasks while still streaming text at normal speed, and the 15-task suite flags all of them, at a median of 30 seconds of generation per file on one consumer GPU. Schema-following and long-context recall need their own tests.
On whether tool use breaks first, a recent GGUF study of agentic tool use in Polish (arXiv 2609.32042, three 7B to 11B models) found no steady decline in task pass rates from Q8_0 to Q3_K_M and a collapse at Q2_K in all three, with format errors already present at Q8_0.
Any per-file suite also has to record the chat template it ran with. Two community deepseek-coder files in our sample carry no chat template. They scored 5 and 6 of 15 under the runtime's default ChatML and 13 and 15 with the model's own instruction template.
On the registry side, the paper recommends testing every low-bit tag before publication and reporting the result alongside its quantization recipe.
Get this paper in your agent:
hf papers read 2609.05881 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper