The Pulse
Magnitude Releases Hardware-Tuned Inference Engine for Agents
Magnitude released its own open-source inference engine for local AI agents, replacing its previous llama.cpp-based backend. The company reports faster decoding in tests on an M4 Pro and Nvidia DGX Spark, while noting the results depend on
AI.info Team · Written with AI ·
“Sessions are long, several often run at once, and you still want to use your computer for other things.”
Anders and Tom, in Magnitude’s launch post
Magnitude released its own open-source inference engine on September 30, replacing the llama.cpp-based backend in its CLI 0.2.0 release. The project is aimed at developers running local models through coding agents, and its central pitch is that the engine tunes GPU kernels on the user’s machine rather than relying only on kernels compiled for broad hardware categories.
The release is available under the Apache 2.0 license and supports Apple Silicon, Nvidia and AMD GPUs, as well as CPU-only systems. A desktop app connects it to tools including Pi, OpenCode, Hermes and Codex; the new magnitude serve command also runs the inference service without opening the desktop window.
Magnitude’s gains come from a narrow benchmark
Magnitude says its engine can run open models up to twice as fast as llama.cpp. Its published comparison uses Qwen 3.6 35B A3B in 4-bit quantization, a 64,000-token context and no speculative decoding. On an M4 Pro with 48GB of memory, the company measured decoding at 57 tokens per second, up from 30 with llama.cpp, and prompt processing at 507 tokens per second, up from 466.
On an Nvidia DGX Spark, Magnitude reports decoding at 58 tokens per second versus 49 for llama.cpp, and prompt processing at 2,507 tokens per second versus 2,033. The company also reports 28% lower per-agent memory use on the M4 Pro and 27% on the DGX Spark. Those are Magnitude’s own results, not independent benchmarks across models and devices.
The team described its benchmark in the launch discussion as a prose-repetition task: it fed the text of Moby-Dick into the model up to the 64K context limit and asked it to repeat the final section. The team said it tried to keep llama.cpp’s settings comparable, but used 16-bit key-value cache for that baseline because its tests found that quantizing the cache to 8-bit keys and 4-bit values reduced llama.cpp’s decoding speed.
Kernel tuning targets long agent sessions
Magnitude’s design addresses a workload its developers say general inference engines do not prioritize: long-running agent sessions, often several at once, on a computer that remains in use for other tasks. Its engine tunes kernel parameters on the target device and focuses hand-written kernels on popular open-weight model families. A hybrid paged-attention design lets concurrent sessions share prefix caches, while memory allocated to an agent can be freed when that session ends.
The release also adds support for speculative decoding methods including DFlash, DSpark and DFlash2. In the release notes, Magnitude reports that speculative decoding on Qwen 3.6 35B A3B at 65K tokens was 10.7% faster than plain decoding on Apple Silicon, compared with a 7% slowdown previously; on Nvidia, it was 38.8% faster, up from 28.7%. The same release adds a server mode for macOS, Windows and Linux.
Early feedback exposes hardware gaps
Responses to the launch show why the headline speed claim needs its benchmark conditions attached. In the discussion, users questioned whether llama.cpp was the right comparison on Apple hardware and posted results where Magnitude ran slower than other tools or had weaker prompt-processing speeds. Magnitude’s team said it plans a fuller comparison against Apple’s MLX-based engines.
The team also acknowledged an optimization gap on M5 Macs, saying its kernels did not yet make full use of newer Metal 4 matrix-multiplication operations. In response to a report from a user with two Nvidia GPUs, it said multi-GPU setups were not supported at that point and that performance could vary by model and backend.
Magnitude’s open-source release puts device-specific tuning and local agent serving in one package, but its strongest published result remains a single-model comparison against llama.cpp. For now, the project’s own launch discussion says it handles one device at a time; broader claims will depend on results across more models, workloads and hardware configurations.