The Pulse
InferenceX Opens TPU-vs-Nvidia Inference Comparisons
SemiAnalysis has published the first third-party InferenceX results for Google’s TPUv7 Ironwood against Nvidia’s B200 and B300. The benchmark shows Ironwood reaching up to 50% better performance per dollar in selected FP8 serving workloads,
AI.info Team ·
A new public benchmark puts Google’s latest TPU in the same frame as Nvidia’s leading inference systems, using the metric that increasingly determines whether an AI service makes money: tokens delivered for each dollar spent.
On September 7, SemiAnalysis published the first third-party inference results for Google’s TPUv7 Ironwood through the InferenceX comparison platform. In tests serving the 397-billion-parameter Qwen3.5 model in FP8, Ironwood reaches up to 50% better performance per dollar than Nvidia’s B200 and B300 under the benchmark’s selected serving conditions.
The release gives outside operators a public reference point for a chip that Google has historically used mainly inside its own services. It also makes clear that Ironwood’s advantage depends on workload shape, cost assumptions, precision, serving topology and the maturity of the software stack.
Ironwood’s first public contest with Blackwell
SemiAnalysis says the new results are the first third-party measurements for TPUv7 Ironwood in its official InferenceX preview. The comparison uses aggregated serving, FP8 precision and single-token prediction, with Qwen3.5 397B as the initial model for Google’s native TorchTPU serving path.
At an interactivity target of 100 tokens per second per user, the analysis estimates Ironwood at approximately $0.181 per million total tokens. The corresponding estimates are $0.222 for Nvidia B200 and $0.276 for Nvidia B300. On those figures, Ironwood costs about 19% less than B200 and 34% less than B300 while delivering the same stated per-user generation speed.
The comparison does not claim that Ironwood produces the highest raw throughput across the entire test range. SemiAnalysis says Nvidia systems remain ahead across much of the raw-performance curve. Ironwood’s result comes from combining competitive throughput with a lower modeled hourly cost, which changes the ranking when the calculation shifts from tokens per second to tokens per dollar.
At 20 tokens per second per user, Ironwood reaches 9,364 total tokens per second per chip in the reported runs. B200 reaches 8,903, while B300 reaches 8,925. The resulting performance-per-dollar advantage is 50.4% over B200 and 96.0% over B300 under the external total-cost assumptions used by SemiAnalysis.
The benchmark’s economics depend on who owns the hardware
SemiAnalysis presents two cost views rather than one. The external comparison models the economics of a hyperscaler laboratory buying TPU systems, while a separate calculation uses an estimated internal Google cost of $1.03 per chip-hour.
At concurrency 256, the internal-cost scenario increases Ironwood’s modeled performance-per-dollar lead to 76.7% over B200 and 130.2% over B300. That result comes with a latency penalty: Ironwood’s mean time to first token is listed at 5.41 seconds, compared with 3.75 seconds for B200 and 2.40 seconds for B300 at the same concurrency.
Those numbers limit what the headline result means for production deployments. A service selling fast responses to interactive users may value first-token latency more heavily than maximum aggregate throughput. A service processing large batches or operating under a strict infrastructure budget may accept slower initial responses in exchange for a lower token cost.
At a 20-second median response-time target, SemiAnalysis places Ironwood at about $0.098 per million total tokens, versus $0.106 for B200 and $0.132 for B300. Ironwood can still lose on a small section of the comparison curve, including a point around a 30-second median response time, before regaining a cost advantage at longer response times.
Native TorchTPU is the real product story
The hardware figures arrive alongside a software transition. Google and outside contributors are moving TPU inference away from the earlier TorchAX path toward TorchTPU, a native PyTorch backend intended for vLLM and SGLang.
TorchAX allows PyTorch model code to execute through JAX. TorchTPU instead lets serving frameworks treat TPUs as native PyTorch devices while still using TPU-specific kernels and compiler components where they matter. That distinction affects how easily developers can bring existing model code and serving infrastructure to Google hardware.
Inference performance does not appear automatically after a framework supports a device. SemiAnalysis describes hundreds of engineering hours and numerous code changes for the first Ironwood bring-up model. The work includes data-parallel attention, mixture-of-experts routing, custom kernels, communication tuning and reductions in padding for gated architectures.
Qwen3.5’s grouped-query attention creates one example of the problem. The model has 32 query heads but only two shared key-value heads, which do not divide evenly across eight logical devices. TPU-specific mapping and communication work is needed to keep those devices busy rather than allowing the uneven attention pattern to become a bottleneck.
“Vendor-neutral, continuously updated benchmarking is essential as models and inference stacks co-evolve,” Ryan Lee, head of developer relations at MiniMax, says on InferenceX’s supporter page. “InferenceX provides the kind of transparent, reproducible data the ecosystem needs.”
The quote reflects why a public comparison matters beyond the ranking of two chips. Inference results change when a kernel improves, a scheduler changes, a new serving topology appears or a model receives a different parallelism strategy. A static specification sheet cannot capture those changes.
FP8 gives Ironwood a fair opening, but not the whole market
Ironwood has native FP8 hardware support, which allows the comparison with B200 and B300 to use the same FP8 precision on both sides. TPUv7 does not have native FP4 computation, however, and that gives Nvidia a clear position when operators use FP4 serving.
SemiAnalysis says Nvidia retains the lead in FP4 comparisons, although it also warns that serving a model in FP4 can reduce quality compared with FP8. The distinction matters because Nvidia’s Blackwell systems can use lower-precision paths that Ironwood cannot currently match in hardware.
The analysis points to TPUv8i, identified as Boardfly, as Google’s answer to that gap. TPUv8i is expected to add native FP4 support, and SemiAnalysis believes it could compete with Nvidia’s Rubin NVL72 systems. That is a forecast rather than a result from the current public Ironwood benchmark, so it should not be read as evidence that TPUv8i has already matched Nvidia hardware.
Google’s current advantage also does not extend equally to every serving design. The public Ironwood results focus on aggregated serving, where prefill and decode run together. Nvidia’s GB200 and GB300 NVL72 systems are compared in some cases using disaggregated serving, which separates prefill and decode onto different pools and can improve the system’s response to particular workloads.
Disaggregation and KV-cache work will decide the next comparison
SemiAnalysis says Google has run disaggregated serving internally for years, including in Gemini production systems, but the external TPU path is not yet fully optimized. In a disaggregated comparison against GB300 NVL72, the Nvidia system can therefore appear more competitive than it does in the aggregated Ironwood comparison.
The next stage of TPU externalization involves moving those internal techniques into public software. Google is working on TPU support for llm-d and on TPU-Sync, a library for transferring key-value cache between workers. The cache contains the intermediate attention state needed to continue generating a response, so moving it efficiently can determine whether a split prefill-and-decode design saves time or adds communication overhead.
KV-cache offloading becomes more important as models serve long prompts, large batches and multi-turn agent sessions. Such workloads reuse system instructions, tool definitions and conversation history across many requests. Keeping that state in high-bandwidth memory is expensive, while moving it to host memory or storage can reduce cost but introduces new latency and scheduling problems.
SemiAnalysis also says Google plans to optimize speculative decoding, multi-token prediction, disaggregated prefill and agentic workloads. The initial benchmark deliberately focuses on an 8,000-token input and 1,000-token output configuration to keep the first bring-up manageable. Results from longer context, multi-turn traces and cache-heavy sessions could produce different rankings.
Public data narrows Google’s TPU software gap
Google’s chips have long been difficult for outside developers to assess because the company’s strongest TPU deployments run inside Google-controlled systems and software. Ironwood changes that commercial position: SemiAnalysis describes TPUv7 as the first generation Google is offering for other companies’ inference workloads through direct purchases or Google Cloud rentals.
The benchmark release also arrives as Google’s external software stack begins to connect with projects that already dominate open model serving. vLLM and SGLang have extensive Nvidia support and growing AMD support. SemiAnalysis expects TorchTPU to move out of private beta and become open source around October, which would give framework maintainers a public base for adding TPU support closer to a model’s release date.
That timing matters because model support often determines hardware choice. A chip can have favorable cost and power characteristics, but an operator may still choose Nvidia if the latest model, quantization method, scheduler or speculative-decoding implementation works there first. TorchTPU’s success will be measured not only by the Qwen3.5 result, but by how quickly it supports models such as Kimi K3, GLM5.3 and Google’s own open-weight systems.
The September 7 release does not establish that Google has displaced Nvidia in inference. It establishes a narrower and more useful finding: under a public FP8 workload using external hardware-cost assumptions, Ironwood can deliver more tokens for less money than B200 and B300, while Nvidia still leads in parts of the raw-speed curve, latency-sensitive scenarios and FP4 serving.
For infrastructure buyers, the immediate change is practical. TPUv7 is no longer judged only through Google’s internal claims or theoretical specifications. Operators can now compare a named open model, a named serving path and explicit token-cost estimates against Nvidia systems, then decide whether the remaining software work fits their own production requirements.