The Pulse
Strata Brings Qwen’s 125B Model to 12GB GPUs
Strata runs Qwen3.8-Flash-Next on a 12GB NVIDIA RTX 5070 by splitting model work across GPU memory, system RAM and SSD storage. Its October 4 release adds decoding and long-prompt changes, while an AMD ROCm port reports slower results on a
AI.info Team · Written with AI ·
Strata’s October 4 release adds speed and concurrency
Strata’s latest release, version 0.1.39, arrived on October 4 with changes to decoding, long prompts and simultaneous requests. The open-source inference engine runs Qwen3.8-Flash-Next, a model with 125 billion language-model parameters, on consumer PCs; Qwen lists 6 billion as active per token, alongside a 51-billion-parameter n-gram table and a 4-billion-parameter multi-token-prediction layer. The release notes report a 6% decoding gain for Q2_0 on an RTX 5070 in a comparison with version 0.1.38, from 68.4 to 72.7 tokens per second on a story prompt.
That benchmark is not a blanket speed promise for every 12GB setup. The notes say a separate verification-graph optimization only runs when a layer’s experts all fit in VRAM, a condition a 12GB card does not meet. The project’s README gives its own RTX 5070 results by model size and context, and says speed varies with those settings.
“The output is byte-identical to 0.1.38’s (10/10 on all four quants with a fixed cache).”
Niko1221, Strata release notes
GPU memory is only one part of the setup
Strata does not load all of Qwen’s weights into the graphics card. It keeps frequently used model components and experts on the GPU, stores other experts in system memory for CPU computation, and reads a large lookup table from the SSD. The approach uses Qwen’s mixture-of-experts structure: only a selection of experts contributes to each token, rather than every parameter taking part in every step.
The project lists 12GB or more of GPU memory as a requirement and says 32GB of system RAM is the minimum, with 64GB enough for every offered size. The model download is about 70GB and the installer recommends roughly 80GB of free disk space. Those demands shift the bottleneck: a modest graphics card can run the model, but it needs substantial host memory and storage to do so.
AMD support has a different performance profile
A separate ROCm port adapts Strata for AMD cards. Its documentation says a bounded-memory mode can let a 12GB GPU and 48GB system-memory PC run the model by keeping a fixed expert pool in RAM and evicting less-used entries. The public benchmark, however, uses an RX 9070 XT with 16GB of VRAM and 64GB of RAM, not a 12GB AMD card.
On that 16GB system, the port reports 14–17 tokens per second for short responses in its default configuration, and 11–14 tokens per second with a bounded expert pool. Those tests use Swift 1.5 IQ2_XS, a fine-tuned version of Qwen3.8-Flash-Next, so they are not a direct comparison with the RTX 5070 figures. The port’s documentation also describes the bounded path as slower than its all-experts-pinned mode.
Local access comes with practical trade-offs
Strata’s appeal is concrete: it makes a large model available through a local setup rather than requiring a hosted inference service. Its README says it exposes OpenAI- and Anthropic-compatible endpoints on the user’s machine, and supports applications and coding agents that can connect to those interfaces. The current release also adds opt-in parallel requests, though requests still wait when the configured slots are occupied.
For a 12GB NVIDIA card, the supported path has a published RTX 5070 benchmark, but the latest release’s most prominent speed gains do not all apply to that memory size. AMD support extends the project’s reach, yet the cited ROCm performance figures come from a higher-memory card. Strata’s practical threshold is therefore not just the GPU: it is the combination of VRAM, system RAM, storage capacity and patience for a first load that can use tens of gigabytes of memory.