The Pulse
NeMo-DCR Cuts 1T Policy Sync From 87.5 Minutes to 150 Seconds
A paper posted October 6 describes NeMo-DCR, a method for syncing updated policies in agentic reinforcement learning. Its authors report a 150-second refit for a trillion-parameter model, compared with 87.5 minutes for transferring a full c
AI.info Team · Written with AI ·
A full checkpoint transfer for a trillion-parameter model takes 87.5 minutes between two AWS regions; the NeMo-DCR method reports completing the same policy refit in 150 seconds. The gap matters for agentic reinforcement learning, where a newly trained policy must reach rollout clusters before the next batch can begin. In a paper submitted to arXiv on October 6, eight authors propose sending changes instead of retransmitting the entire model.
Full checkpoints are the costly baseline
Agentic RL often separates training from rollout: one cluster updates the policy while another generates the experience used to train it. Each update must be synchronized across the clusters. The paper says measurements of BF16 training show that about 1% of weights change their stored values per step, suggesting a full-checkpoint transfer moves much more data than most updates require.
Sending only changed values is not enough by itself. The authors say earlier approaches can fall short on correct placement, bit-level exactness or transfer efficiency, and do not fully recover from failures during a refit. Their method is designed to address those problems while preserving the result of a dense transfer.
NeMo-DCR sends deltas without changing stored bits
NeMo-DCR, short for Delta-Compressed Refit, projects changes from training shards into the checkpoint’s canonical coordinates and uses the serving runtime’s native loader to place them on the receiving side. It represents some changes as XOR masks and sends overwrites for others. Receivers apply the updates in place; retries overwrite partial writes, and a joint commit links the new policy to its baseline for the next update.
“We present NeMo-DCR (Delta-Compressed Refit), which sends only changes yet is bit-exact: receivers obtain the same parameter and buffer bits as a dense refit.”
Songlin Jiang et al., “NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale”
For transport, the authors describe streaming payloads through object storage or a relay tree while delta construction proceeds, avoiding a cross-cluster collective. The design’s central trade-off is precision: it aims to reduce what travels without changing the exact parameter and buffer bits the receiving model gets.
The speedup is a paper result, not a deployment guarantee
The authors report that refits for models ranging from 30 billion to 1 trillion parameters run 12 to 40 times faster than a transport-only full-checkpoint reference at change rates of 3% and 5%. For a 1T model at a 3% change rate, their relay-tree test takes 150 seconds, compared with 87.5 minutes for a full transfer. The 1% BF16 change figure is a separate measurement in the paper, not the change rate used for that headline result.
Those figures are the authors’ experimental claims, not an independent evaluation or a promise of the same timing in every deployment. The abstract does not give enough information to generalize the 150-second result across different hardware, network conditions or serving setups. Teams considering the method would need to assess the actual transfer path and failure handling in their own clusters.
Code is linked from NVIDIA NeMo RL
The paper points to code in a pull request in NVIDIA’s NeMo RL repository. That pull request, merged on July 19, 2026, describes optional delta-compressed weight transfer for non-colocated vLLM collective refit and lists Amazon S3 or ZeroMQ as transport backends. The repository entry is relevant implementation context, but the paper’s 150-second trillion-parameter result should be read as the result reported in the paper—not as a general benchmark of the merged pull request.
For distributed RL systems, the concrete claim is narrower than “faster model updates” in general: under the paper’s stated setup, NeMo-DCR moves a 1T policy update in 150 seconds at a 3% change rate, while preserving the bits produced by a dense refit.