# Lambda reports RoCE throughput gap in AI inference test

Published 2026-11-20 · AI.info Team · AI.info

> Lambda says RoCE delivered roughly 70% lower throughput than NVIDIA Quantum InfiniBand near hardware saturation in an inference experiment. The comparison used two-node HGX B200 configurations that differed in network fabric, placement and settings.

An inference test found roughly 70% lower throughput and a 2.5-times slower time to first token with RoCE than with [NVIDIA](https://ai.info/companies/nvidia) Quantum InfiniBand near hardware saturation, according to [Lambda](https://lambda.com/blog/roce-and-nvidia-quantum-infiniband-and-their-quantified-tradeoffs). The comparison, published November 20, used two-node HGX B200 configurations that differed in placement and network settings as well as interconnect.

The company says the throughput gap appeared as concurrency reached about 250 users. It sells access to both configurations tested: On-Demand instances connected through RoCE and 1-Click-Clusters with a dedicated InfiniBand fabric.

## How the test was run

For the InfiniBand test, Lambda used two 1-Click-Cluster NVIDIA HGX B200 nodes connected through NVIDIA Quantum InfiniBand switches and adapters. For RoCE, it used two On-Demand HGX B200 nodes. The On-Demand instances were in the same cloud region, but not necessarily in the same data center.

Both setups served nvidia/GLM-5.2-NVFP4, an NVIDIA Blackwell-optimized variant of [Z.ai](https://ai.info/companies/zai)’s GLM 5.2 model. The serving framework was SGLang, with TP16/DP16/EP16 parallelism, and the benchmarking tool was NVIDIA’s AIPerf.

Each request had 8,192 input tokens and 1,024 output tokens. The test ranged from 10 to 300 simulated concurrent requests. The servers shared a core set of SGLang flags, including a maximum of 512 running requests and a static memory fraction of 0.70.

The network-specific settings differed between the configurations. The RoCE setup used RoCEv2 and disabled IBGDA in its NVSHMEM setting, while the InfiniBand setup enabled it and listed a different set of host channel adapters. The comparison therefore covers the configurations Lambda ran, rather than network fabric alone.

## Throughput and latency

Lambda attributes a throughput drop at 250 concurrent users to the hardware bandwidth limit on the NVIDIA Blackwell GPUs. It also reports that, as the number of users fell, low concurrency showed the slowest inter-token latency and tokens per second overall.

## Different network fabrics

RDMA over Converged Ethernet, or RoCE, includes RoCEv1 and RoCEv2. RoCE operates at layer 2: it uses Ethernet and MAC addresses to communicate across the same network, and nodes can span multiple switches but must stay within the same base network. RoCEv2 builds on this with UDP/IP packets, allowing nodes to communicate across different subnets, buildings or data centers.

NVIDIA Quantum InfiniBand uses a dedicated networking fabric rather than Ethernet/IP and can cross switches and locations. Its switches and fabric must be in place to obtain maximum performance with minimal latency, according to Lambda.

The company estimates typical InfiniBand latency at around 0.5–2 microseconds, compared with roughly 1–3 microseconds for RoCE on a well-optimized local fabric. It says RoCE is more sensitive to configuration, congestion and topology. Under less favorable conditions, latency can be several times higher and potentially reach tens of milliseconds during congestion.

## Sources

1. Original source · Blog post [Lambda](https://lambda.com/blog/roce-and-nvidia-quantum-infiniband-and-their-quantified-tradeoffs) lambda.com

## Related stories

- [Lambda connects more than 10,000 GPUs with photonics switches](https://ai.info/news/lambda-photonics-switches-10000-gpus-quebec)
- [Lambda Targets Up to $4 Billion Before a Planned 2027 IPO](https://ai.info/news/lambda-targets-4-billion-pre-ipo-raise)
- [Nvidia’s private-company stakes reach $65.5 billion, TechCrunch reports](https://ai.info/news/nvidia-private-startup-stakes-65-billion)
- [NVIDIA reports Vera Rubin gains in MLPerf Training debut](https://ai.info/news/nvidia-vera-rubin-mlperf-training-endpoints-results)
- [Cerebras reports 55% third-quarter revenue growth, raises outlook](https://ai.info/news/cerebras-third-quarter-2026-record-revenue)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/lambda-roce-infiniband-ai-inference-comparison
