The Pulse
Hugging Face Brings llama.cpp Quants to Transformers
Hugging Face has added support for running llama.cpp quantized models through Transformers, allowing developers to load GGUF checkpoints with familiar Python and PyTorch APIs.
AI.info Team ·
Hugging Face has added support for running llama.cpp quantized models directly through Transformers, allowing developers to load GGUF checkpoints with familiar Python and PyTorch APIs instead of switching to a separate runtime. The company published the update on September 22, 2026, with an initial focus on local inference on Apple Silicon and the Qwen3.5 architecture.
The change connects two widely used parts of the open-model ecosystem. GGUF is the model format developed for llama.cpp, while Transformers is Hugging Face’s main library for defining, loading and evaluating models. Hugging Face says the new path reuses ggml kernels through its kernels library and reduces overhead in the generate method.
This is where we are right now. And i’m not gonna lie it feels pretty magical 🧙♀️
Julien Chaumond, Hugging Face CTO
Qwen3.6 27B running inside of Pi coding agent via Llama.cpp on the MacBook Pro
For non-trivial tasks on the Hugging Face codebases, this feels very, very close to hitting the latest Opus in Claude…
Qwen3.5 Fits More Easily Into Local Machines
GGUF packages model weights and metadata, including tokenizer information and, optionally, a chat template, in a single file. Its quantization levels reduce memory use by storing weights at lower precision, with variants such as Q4_K_M using mostly four-bit weights while preserving higher precision for selected tensors.
Hugging Face uses Unsloth’s Qwen3.5-4B GGUF checkpoint to show the effect. The unquantized BF16 file is listed at 8.42 GB, compared with 3.53 GB for Q6_K, 3.14 GB for Q5_K_M and 2.74 GB for Q4_K_M. The company recommends starting with Q4_K_M and moving to larger variants when more memory is available, while warning that the quality impact depends on the model and task.
One GGUF Argument, Then Standard Transformers Code
The new loading path keeps the GGUF-specific setup small. Developers pass a Hub repository and filename to from_pretrained, using gguf_file for both the tokenizer and model. The rest of the workflow uses standard Transformers calls such as apply_chat_template and generate.
Hugging Face’s example loads unsloth/Qwen3.5-4B-GGUF with Qwen3.5-4B-Q4_K_M.gguf. When the quantized weights remain packed on Apple’s Metal backend, Transformers automatically loads compatible ggml and Metal kernels and uses ggml-org/ggml-attn for attention. If the attention kernel cannot be fetched, the model falls back to sdpa with a warning. Separately, if a compatible quantization kernel is unavailable, the loader dequantizes the model, which uses more memory.
The same checkpoint can also run through transformers serve, which exposes an OpenAI-compatible API. The command accepts a model identifier in the form <model_id>:<filename>.gguf, allowing a repository with several quantization variants to select one specific file.
Hugging Face Measures Against llama.cpp
Hugging Face compares its implementation with llama.cpp across three GGUF checkpoints: a small dense model, a larger dense model and a mixture-of-experts model. The tests run on a MacBook Pro with an M2 Max processor, 32 GB of unified memory, macOS 26.6, PyTorch 2.12.1 and kernels 0.17.0.
The company says Transformers comes close to llama.cpp across all three checkpoints. The comparison is not an identical test: the Transformers measurement generates 128 tokens from a 12-token prompt and includes prompt processing, while llama.cpp’s llama-bench result reports decode-only throughput. Hugging Face also says its runs use three warmed repetitions and include long cooling intervals because back-to-back tests reduced performance by 10% or more on the test machine.
Hugging Face continues to recommend llama.cpp when efficient local inference is the main requirement. Its dedicated runtime, memory management and hardware support remain designed specifically for that job. The new integration instead targets developers who want to inspect activations, modify model code, evaluate quantized checkpoints or test generation changes inside Python.
GGML Kernels Extend Beyond GGUF
The work also brings several ggml-derived kernels into PyTorch model execution. The ggml-quantization kernel reads packed weights directly for matrix operations, while ggml-norm handles normalization and ggml-attn supplies Metal attention. A gated delta-network kernel supports linear-attention layers used by Qwen3.5 and Qwen3.8, and Hugging Face has added its own top-k implementation for mixture-of-experts routing.
That approach could eventually help models that do not have a dedicated llama.cpp implementation. Hugging Face says Transformers already contains PyTorch versions of many architectures, so compatible ggml kernels could accelerate selected operations without requiring a complete C++ implementation in llama.cpp. The same technique could also apply to computer vision, audio and multimodal models, although each architecture still requires integration and validation.
Apple Silicon Is the First Boundary
The initial packed inference path is limited to Apple Silicon’s MPS backend. GGUF import through dequantization remains available as a separate route, but support for the file format does not mean packed kernels work on every device.
Padding and batching also need additional work. Unpadded inputs benefit from an optimization that removes an unnecessary attention mask early in generation, while padded batches cannot use that shortcut and may run more slowly. Architecture coverage currently centers on Qwen3.5 dense and mixture-of-experts models, along with compatible Qwen3.8 checkpoints.
For now, the practical result is narrow but useful: a developer can take a compact GGUF checkpoint built for llama.cpp, load it through Transformers, keep the generation loop in Python and use PyTorch tools to inspect or modify the model. The first release supports Qwen3.5 packed inference on Apple Silicon, with wider architecture and hardware coverage still ahead.