The Pulse
Google details bitwise-matched RL training across 64 TPU chips
Google says its prototype eliminated a numerical mismatch between sampling and training in a reinforcement-learning test across 64 TPU v5p chips. The single-seed test used a synthetic environment and a Qwen3-8B checkpoint.

A Google prototype produced bitwise-identical probability calculations across sampling and training paths in a reinforcement-learning test using 64 TPU v5p chips. In a November 16 developer blog post, the company said the approach addresses a numerical mismatch that can destabilize training of large language models.
The work was developed in Tunix, Google’s open-source JAX library for language-model reinforcement learning and post-training on Cloud TPU. The prototype, called Zero train–inference mismatch, or Zero-TIM, keeps separate serving and training pipelines rather than combining them into one engine.
A test of numerical alignment
In the test, Google ran a 200-step numerical verification on the synthetic FrozenLake environment. The comparison used a public Qwen3-8B checkpoint and a multi-host mesh of 64 TPU v5p chips, configured as DP8×TP8.
The Zero-TIM run recorded zero mean absolute difference between sampler and trainer log probabilities at every logged step, according to Google. Standard training and a token-level truncated importance sampling method retained nonzero differences; the latter adjusted their contribution to the update without removing the mismatch.
Google reported higher observed solve rates for Zero-TIM in the runs. The runs used a single seed on FrozenLake grids ranging from 2×2 through 9×9, with up to five interaction turns, and the displayed results covered the first 200 training steps.
The Zero-TIM configuration also preserved sampled token IDs across environment turns using token-in/token-out handling. The comparison runs kept their existing tokenization paths.
Where the mismatch begins
In large-language-model reinforcement learning, the model acts as a policy that selects actions based on preceding tokens. For on-policy learning, the sampler and trainer must evaluate that policy identically when using the same weights and token history.
The sampler selects action tokens during incremental decoding and records their log probabilities. The trainer evaluates those tokens in batched forward and backward passes to calculate current log probabilities and gradients.
Google says differences in execution paths can change the order of floating-point operations and their intermediate precision, producing different probability bits even when weights and model equations match. Its prototype checks three paths: sampler decoding, serving prefill and the trainer’s forward pass.
Comparing decoding with prefill checks whether the serving engine agrees with itself when generating and re-scoring tokens. Comparing prefill with the trainer checks agreement across the serving and training boundary.
Controlling the arithmetic
The implementation uses Pallas kernels to control accumulation order and intermediate precision within each tensor-parallel rank. Partial products accumulate in float32 before explicit output casts, while dedicated kernels control normalization reductions and intermediate casts.
Across ranks, the approach separates communication from addition: partial outputs are transported and then added in the same source-rank order. The paths also use matching sampling transforms and the same log-softmax construction before selecting the recorded action token.
The trainer reuses the serving prefill computation, with custom differentiation rules providing the backward pass. Long trajectories are evaluated in fixed-size chunks, with cache state carried forward and gradients passed back through the cache connections before the optimizer update.
Google is extending support to linear-attention and Mixture-of-Experts architectures. The current implementation is available to preview as part of the Tunix work.
Source
Companies in this story
Related stories
- RadixArk team describes serving Meta’s Muse Spark Open on one TPU host
- Google reports Ironwood TPU results in MLPerf training tests
- Adobe and Google expand AI partnership across apps
- Google launches benchmark for visual knowledge-work agents
- Google’s Suncatcher Satellite Sends Its First AI Response to Earth