Skip to content
AI.info

The Pulse

Lambda reports RoCE throughput gap in AI inference test

Lambda says RoCE delivered roughly 70% lower throughput than NVIDIA Quantum InfiniBand near hardware saturation in an inference experiment. The comparison used two-node HGX B200 configurations that differed in network fabric, placement and settings.

Lambda reports RoCE throughput gap in AI inference test

An inference test found roughly 70% lower throughput and a 2.5-times slower time to first token with RoCE than with NVIDIA Quantum InfiniBand near hardware saturation, according to Lambda. The comparison, published November 20, used two-node HGX B200 configurations that differed in placement and network settings as well as interconnect.

The company says the throughput gap appeared as concurrency reached about 250 users. It sells access to both configurations tested: On-Demand instances connected through RoCE and 1-Click-Clusters with a dedicated InfiniBand fabric.

How the test was run

For the InfiniBand test, Lambda used two 1-Click-Cluster NVIDIA HGX B200 nodes connected through NVIDIA Quantum InfiniBand switches and adapters. For RoCE, it used two On-Demand HGX B200 nodes. The On-Demand instances were in the same cloud region, but not necessarily in the same data center.

Both setups served nvidia/GLM-5.2-NVFP4, an NVIDIA Blackwell-optimized variant of Z.ai’s GLM 5.2 model. The serving framework was SGLang, with TP16/DP16/EP16 parallelism, and the benchmarking tool was NVIDIA’s AIPerf.

Each request had 8,192 input tokens and 1,024 output tokens. The test ranged from 10 to 300 simulated concurrent requests. The servers shared a core set of SGLang flags, including a maximum of 512 running requests and a static memory fraction of 0.70.

The network-specific settings differed between the configurations. The RoCE setup used RoCEv2 and disabled IBGDA in its NVSHMEM setting, while the InfiniBand setup enabled it and listed a different set of host channel adapters. The comparison therefore covers the configurations Lambda ran, rather than network fabric alone.

Throughput and latency

Lambda attributes a throughput drop at 250 concurrent users to the hardware bandwidth limit on the NVIDIA Blackwell GPUs. It also reports that, as the number of users fell, low concurrency showed the slowest inter-token latency and tokens per second overall.

Different network fabrics

RDMA over Converged Ethernet, or RoCE, includes RoCEv1 and RoCEv2. RoCE operates at layer 2: it uses Ethernet and MAC addresses to communicate across the same network, and nodes can span multiple switches but must stay within the same base network. RoCEv2 builds on this with UDP/IP packets, allowing nodes to communicate across different subnets, buildings or data centers.

NVIDIA Quantum InfiniBand uses a dedicated networking fabric rather than Ethernet/IP and can cross switches and locations. Its switches and fabric must be in place to obtain maximum performance with minimal latency, according to Lambda.

The company estimates typical InfiniBand latency at around 0.5–2 microseconds, compared with roughly 1–3 microseconds for RoCE on a well-optimized local fabric. It says RoCE is more sensitive to configuration, congestion and topology. Under less favorable conditions, latency can be several times higher and potentially reach tens of milliseconds during congestion.

Sources

  1. Original source · Blog post Lambda lambda.com

Related stories

Explore

More articles