The Pulse
VIDRAFT Trains a 46M Model on Inference-Only NPUs
VIDRAFT says it pretrained VKORE-46M-NPU from scratch on 32 Rebellions ATOM-Max inference chips. Its model card reports close per-layer gradient agreement with PyTorch, but says the multi-chip NPU and GPU training curves diverged.
AI.info Team · Written with AI ·
Per-layer gradients from a single Rebellions ATOM-Max NPU matched a PyTorch autograd reference with an average cosine similarity of 0.99986, according to VIDRAFT’s model card. But that single-chip correctness test is separate from the 32-chip pretraining comparison: the card says the NPU and GPU runs followed different loss trajectories, qualifying an earlier headline claim of a 0.99 correlation.
VIDRAFT released VKORE-46M-NPU on October 14 as a demonstration that an inference-oriented accelerator can run backpropagation and train a language model from random initialization. The experiment is small by design, and its detailed comparison points to both a working training path and limits in what the GPU comparison proves.
What the 32-chip run tested
VKORE-46M-NPU has 45.8 million parameters and a decoder-only transformer architecture using RoPE, RMSNorm, SwiGLU and grouped-query attention. It has 12 layers, a hidden width of 512, and a 256-entry character-level vocabulary. The team trained it on about 22.5 million tokens from the public TinyStories dataset, using a context length of 256.
The training ran on 32 first-generation Rebellions ATOM-Max chips, released in 2025. The model card says the run took a few hours. It reports throughput of about 260,000 tokens per second on the NPUs, compared with about 275,000 tokens per second on two Nvidia B200 GPUs.
Those throughput figures are close, but they do not mean the runs followed identical optimization paths. The model card says the NPU run averaged parameters every eight steps, in a DiLoCo-style setup, while the B200 run synchronized gradients every step. That difference matters when interpreting the loss comparison.
The 0.99 comparison has a qualification
VIDRAFT’s technical write-up presents the NPU and GPU runs as using the same initialization, data order and schedule, and says their loss trajectories correlate at 0.99. The model card’s expanded account says the loss gap grew during training: at step 2,950, the NPU’s single-batch training loss was 0.42, while the B200 run’s was 0.25.
“A high correlation between two falling loss curves is weak on its own, so we do not lead with it.”
VIDRAFT, VKORE-46M-NPU model card
The card says a controlled GPU rerun that separates the effects of the different synchronization schemes is in progress. Until that comparison is available, the reported throughput is a useful measurement, but the 0.99 correlation should not be read as proof that multi-chip NPU training reproduced the GPU’s optimization dynamics.
Gradient checks support a narrower claim
For a separate test, VIDRAFT compared gradients from one NPU against an autograd reference using the same seed and batch, without synchronization across chips. It reports an average per-layer cosine similarity of 0.99986, with a minimum of 0.99972 and a maximum of 0.99996. The hand-written backward implementation’s worst-case relative gradient error against PyTorch autograd was 8.96×10-7 in that test.
The model card also reports a check on all 110 parameter tensors for a TinyStories batch: relative L2 error against CPU fp32 autograd was 0.96% at the median and 2.21% at the worst. A CPU bf16 control had higher errors—1.67% at the median and 3.42% at the worst. Those measurements support numerical agreement for the tested gradients; they do not settle how the system scales across chips or larger training runs.
Weights are public; the training method is not
VIDRAFT has published the model weights, model card and comparison materials on Hugging Face, but says the core training technique is under patent application and the training code is withheld. The company says it plans to describe the method in a future paper. That leaves outside teams able to inspect the model and reported results, but not to reproduce the training stack from the released artifacts.
The model itself is a proof of execution, not evidence of broad language ability: its character-level tokenizer and TinyStories data are aimed at producing short English stories. The remaining test is whether the gradient behavior and throughput hold under a controlled multi-chip comparison, then on larger models and less restricted workloads.