SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops

AI summary

Overview

Research area: Computer systems and hardware for machine learning (cs.AR) — specifically LLM inference serving on unified-memory desktop hardware.

Technical level: Intermediate. The paper is written for readers who know roughly how LLM serving works (prefill and decode, batching, KV cache), but every mechanism it relies on is explained in plain terms.

Scope (1 sentence): SiliconBench benchmarks nine Apple Silicon LLM serving engines, plus a CUDA reference track on DGX Spark, along three axes — speed, memory use, and output fidelity — under single-user agent workloads.

What This Paper Is About

Local LLM serving on unified-memory desktops is usually ranked by throughput alone, but on Apple Silicon the model weights, the inference caches, and the user's browser, IDE, and OS all share the same RAM pool. An engine can therefore post a top throughput number while leaving almost no memory headroom, or while silently producing worse task-level answers than a reference implementation of the same model. SiliconBench is a benchmark that measures all nine actively developed Apple Silicon serving stacks on concurrency scaling, system memory use, and task-level fidelity at once, with a CUDA reference track on DGX Spark to show what serving headroom looks like elsewhere.

Key Contributions

1. Three desiderata for Apple Silicon serving — serving architecture readiness (D1), memory discipline (D2), and multi-node scaling (D3) — each backed by measurements rather than stated as a principle.
2. A joint benchmark of concurrency scaling, memory use, and task-level fidelity across nine Apple Silicon stacks, with a complementary CUDA reference track on DGX Spark and an NVIDIA A100 fidelity reference.
3. A maintenance workflow for keeping the benchmark valid across rapid software updates: a Claude Code maintainer agent that updates frameworks, runs the benchmark, and proposes adapter fixes within a write allowlist covering framework adapters and model profiles, with human review of every agent change and community PR before official results are regenerated.
4. Released artifacts: benchmark code, per-run results, and maintenance journals, plus a workflow combining bounded agent fixes with human review.

Main Findings

• Only three stacks pass all three audit gates. The gates are completion of at least 90/100 requests at every concurrency level on both splits, reference-band fidelity, and serving all three model releases. llama.cpp, vllm-metal, and omlx pass. sglang, ollama, and mistral.rs pass one gate; mlx_lm and vllm-mlx pass the model gate but fail on memory behavior; hf_transformers passes two.
• Speed rankings hide agent-serving differences. On Qwen3-0.6B, vllm-metal alone more than doubles throughput on both chat and agent workloads from concurrency 1 to 16. Its chat scaling is 3.65x and its agent scaling 2.71x, making it the only stack above 2x on both splits. hf_transformers shares the same audited batching capabilities (packed queries, paged KV, mixed steps) yet scales only 1.23x on chat.
• Chat leaders are not agent leaders. At chat concurrency 16, vllm-metal and ollama finish within 1% of each other, but vllm-metal leads the agent split by 32%. Every padded or serial-prefill path either gains less than 30%, regresses, or fails by concurrency 16 on chat; on agent, only omlx stays near its single-stream rate.
• First-token latency separates the stacks. Only vllm-metal combines sub-quarter-second median TTFT with increasing agent throughput at concurrency 16 (219 ms). sglang also admits requests quickly (655 ms) but its agent throughput falls from 80.6 to 45.3 tok/s. The other agent survivors exceed one second.
• Completing every request does not mean memory discipline. Two stacks complete every request while memory use approaches physical capacity and throughput declines. ollama holds 94–97% of the Metal advisory working-set hint across both splits starting at concurrency 1. mistral.rs reaches 59.3 GB at agent concurrency 8 while completing only 13/100 requests. sglang and vllm-mlx each complete all 600 requests despite high memory usage. Unbounded mlx_lm nearly triples its chat footprint for a 7% throughput gain. By contrast, vllm-metal and llama.cpp keep nearly flat memory use as load increases; vllm-metal's agent peak system memory is 34.7–34.9 GB.
• Padding creates a deployment cliff, not an error. For a controlled mlx_lm workload of three prompts at 30K, 5K, and 10 input tokens, concurrent mode pads both query and KV paths to 90,000 token positions even though actual tokens total 35,010 — wasting 61% of query compute and KV memory I/O. On the 32 GB M1 Pro, the wasted memory raises system memory pressure enough to trigger macOS page compression, and wall time grows 2.5x relative to sequential execution. On the 64 GB M1 Max the same workload fits, and the concurrent penalty is negligible. Identical code degrades silently on the smaller machine with no error signal.
• Some failures are real, some are harness artifacts. mlx_lm's 70/100 completion at agent concurrency 1 and 8 reflects legitimate tool calls parsed to a structured field with zero content tokens, which the harness scores as failures; its collapse to 2/100 at agent concurrency 16 is real. mistral.rs degrades at concurrency 8 and crashes at concurrency 16, while hf_transformers exceeds the 1-hour wall-clock cap on agent.
• Fidelity mostly agrees, with two exceptions. On Qwen3-0.6B, six of eight measured stacks lie within roughly 1.5 percentage points of the A100 vLLM reference on both 0-shot and 5-shot; all five stacks measured on each newer model lie within 1.4 percentage points. ollama gains little from 5 shots and falls roughly 30 percentage points behind stacks that benefit from in-context examples. vllm-mlx scores 5.0 percentage points below the zero-shot reference despite zero request or parse errors, which the authors say may indicate an implementation bug. mistral.rs was too slow to finish.
• Newer architectures narrow engine support. Qwen3.5/3.6 combine recurrent attention state with full-attention KV caches, and Gemma 4 shares KV across layers. As of April 2026 only llama.cpp and vllm-metal supported both families; by the August 2026 campaign five of the nine stacks did. mistral.rs still lacks a DeltaNet kernel, and sglang's MLX backend serves neither family — it has no hybrid-SSM kernels for Qwen3.5, and its text-only loader rejects Gemma 4's multimodal checkpoint, even though upstream sglang serves both on CUDA. The coverage gap belongs to the Metal backend rather than the engine.
• The CUDA reference track shows the headroom. On Qwen3-0.6B, CUDA vLLM and SGLang reach 4.3–7.7x scaling across the two splits with flat TTFT, while llama.cpp's agent throughput flattens on both platforms (though its Metal build finishes ahead on chat at concurrency 16).
• On larger models, throughput gaps narrow but latency still separates. On Qwen3.8-27B (dense) and Qwen3.6-35B-A3B (MoE, 3B active parameters), vllm-metal's throughput rises with concurrency on the dense model and it has the lowest mean TTFT at concurrency 4. On MoE, omlx leads at concurrency 1 while vllm-metal maintains lower TTFT under load. vllm-metal's measured MoE advantage at concurrency 4 ranges from 4% to 17% depending on the archived omlx cache configuration.
• Parallelism and transport choice decide multi-node outcomes. On Qwen3.5-35B-A3B at 8-bit across two M4 Pro Mac minis connected by Thunderbolt 5 with RDMA, tensor parallelism scales while pipeline parallelism regresses. EXO reaches 87.0/85.1/78.3 decode tokens per second at context 512/4096/16384 on two nodes (1.30–1.37x speedup), mlx_lm reaches 63.3/62.7/57.7 (1.28–1.32x), and llama.cpp's sequential TCP pipeline falls to 32.4/31.2/29.9 (0.80–0.84x, a regression of 16–20%). Prefill scaling is roughly 2x at short contexts and 1.7–1.8x at 4K tokens.
• Maintenance is part of the method. Two incidents from five early M2 Max runs (2026-04-11 through 2026-04-23) illustrate why: the MLX 0.31.1 to 0.31.2 update broke mlx_lm, vllm-metal, and vllm-mlx simultaneously (two recovered in the same cycle, vllm-mlx remained broken), and ollama's bare-GGUF model import omitted Qwen3's ChatML delimiters, causing roughly 92% parse failures on the classification task until a registry pull replaced it.

Methodology in Plain English

The researchers picked one task that mirrors how people actually use local models: a single coding agent fanning out 4 to 16 parallel requests to a local server. They built two prompt splits of 100 prompts each — a chat split drawn from OpenOrca and CNN/DailyMail with prompts ranging from 21 to 3,500 tokens, and an agent split drawn from BFCL V3, Hermes, and ClawsBench with prompts from 0.3K to 8.9K tokens and pre-baked tool calls — and swept concurrency at 1, 8, and 16 on one Apple M5 Pro with 64 GB of RAM.

Three small models anchor the work: Qwen3-0.6B, which every stack can run, plus Qwen3.5-0.8B and Gemma-4-E4B-it, which deliberately exercise newer attention architectures and multimodal checkpoints so that support gaps themselves become a measurement. Small models also keep weights from dominating memory, which makes concurrency-dependent runtime allocations easier to see. Separate follow-up studies extend coverage to larger dense and MoE models and to two-machine clusters.

Speed is measured as throughput (total generated tokens over sweep wall time) and time to first token (from HTTP submission to the first streamed content), after three warmup requests, with temperature set to zero and thinking generation disabled. Failed requests count toward sweep wall time but not toward latency statistics or the output-token numerator. Memory is measured as peak system memory during serving, with per-second traces, against the machine's RAM and Metal's advisory recommendedMaxWorkingSetSize hint.

Fidelity is assessed separately, and deliberately not with a token-level comparison, because that comparison was dominated by chat-template differences. Instead the stacks run the GMRID supply-chain incident-classification task (1,146 items, 8 classes, 0-shot and 5-shot) and are scored by weighted F1 against a vLLM reference running on an NVIDIA A100 with identical weights and precision. The authors treat agreement within roughly 1.5 percentage points as a band rather than a formal noise estimate, and explicitly do not rank stacks inside it.

To keep results honest across fast-moving software, the authors run a maintainer agent under released instructions whose write allowlist covers only framework adapters and model profiles; workloads, scoring, and aggregation stay maintainer-controlled, and maintainers review all changes before official results are regenerated.

Why This Matters

Impact on research. The paper argues that evaluating local serving on throughput alone produces rankings that do not predict whether a stack will survive concurrent agent load, preserve memory headroom, or match a reference implementation's task quality. It supplies a reusable measurement design — joint speed/memory/fidelity gates plus a disclosed maintenance workflow — that other benchmark authors can adopt when their target software ships weekly. It also isolates a platform-specific problem: the Metal kernel-development ecosystem is less mature than the CUDA one, which helps explain uneven model coverage rather than treating it as incidental.

Real-world applications:

• Choosing a local inference engine for a desktop coding agent that issues 4 to 16 simultaneous requests, where the wrong choice either crashes under load or exhausts RAM the user's other applications need.
• Diagnosing memory pressure on unified-memory machines when identical code runs fine on a 64 GB laptop but thrashes on a 32 GB one, with no error message.
• Deciding whether to cluster two Macs: the results show tensor parallelism over Thunderbolt RDMA helps while pipeline parallelism over TCP hurts, which directly informs a hardware purchase.
• Auditing whether a locally served model produces the same task-level answers as a reference implementation, which matters for classification and other scored workflows.

Industry relevance. Unified-memory consumer hardware now reaches 512 GB, capacities historically confined to data-center accelerators, and the ecosystem is large: hundreds of thousands of annual Mac installs of Ollama, llama.cpp, and LM Studio; 1.45M monthly PyPI downloads of mlx_lm; and 4,710 community-converted models on Hugging Face's mlx-community hub (figures collected 2026-05-03). At least nine actively developed stacks compete on this hardware. For engine maintainers, the per-stack findings identify concrete gaps — missing kernels, loader restrictions, allocator retention — that are actionable rather than abstract.

Future Directions

• Foreground-aware memory adaptation. The paper states plainly that adaptation to changing foreground memory demand was not evaluated, even though D2 is defined around preserving headroom for macOS and the user's applications. A policy that reacts to live pressure rather than to a static at-startup budget is the obvious next measurement.
• Closing the Metal backend gap. sglang serves both newer model families on CUDA but neither on the Metal backend, and mistral.rs lacks a DeltaNet kernel. Whether these gaps are tooling-driven or effort-driven, and how quickly the coverage roster converges, is an open question the paper tracks across two dated campaigns.
• Scale and breadth of the follow-up studies. The larger dense/MoE and multi-node results cover selected engines, use later builds and different quantization formats, and rest on one measurement per configuration with a model that already fits on a single node. Extending them to the full engine roster, to capacity-unlocking models, and to more than two machines would test whether the tensor-versus-pipeline conclusion generalizes.
• Statistical rigor and additional fidelity tasks. The fidelity band is an observed agreement range, not a formal noise estimate, and the paper reports no run-to-run variance estimates; fidelity rests on a single classification task. Repeating runs and adding task families would establish whether the remaining outliers are bugs, drift, or measurement noise.
• Coverage of regimes the audit omits. The main splits set aside very long contexts (above 10K tokens) and long-form generation (above 1K output tokens), both of which local agents will increasingly hit; the authors name these as deliberate scope choices for this snapshot.

Target Audience

Engineers and researchers who run or maintain LLM inference locally on Apple Silicon or other unified-memory desktops, and who need more than a throughput leaderboard to choose a stack. It is also useful to serving-system developers looking for concrete gaps to close in Metal backends, to benchmark designers interested in the agent-assisted maintenance workflow, and to technically literate practitioners who deploy local coding agents and want to understand why the same setup behaves differently on a 32 GB and a 64 GB machine.

Authors’ abstract

Concurrent local LLM serving on unified-memory desktops must preserve memory headroom and output fidelity, which speed-only rankings overlook. We introduce SiliconBench, which evaluates nine Apple Silicon serving engines through three lenses: speed, memory, and fidelity. We evaluate chat and agent serving on Qwen3, Qwen3.5, and Gemma 4. We use a classification task to check for quality regressions against an NVIDIA reference. DGX Spark provides a complementary serving-performance reference. Three desiderata guide interpretation: serving architecture readiness, memory discipline, and multi-node scaling. On Qwen3-0.6B, vllm-metal alone more than doubles throughput on both workloads from concurrency 1 to 16. CUDA vLLM and SGLang show stronger concurrency scaling on the same prompts. Explicit memory budgets do not guarantee memory headroom: two stacks complete every request while memory use approaches physical capacity and throughput declines. The newer model architectures have narrower engine support. Their evaluated implementations match the fidelity reference. Only three stacks satisfy the completion, fidelity, and model-coverage gates. Comparisons on larger dense and MoE models reinforce the importance of scheduling prompt processing alongside ongoing generation: vllm-metal's packed prefill-decode path maintains lower first-token latency than omlx under concurrent load. In the tested two-machine configurations, tensor parallelism over Thunderbolt RDMA scales while pipeline parallelism over TCP regresses. We release benchmark code, per-run results, and maintenance journals, supported by a workflow combining bounded agent fixes with human review.