Skip to content
AI.info

The Pulse

NVIDIA releases Sigma diffusion language models in two sizes

Nemotron-Labs-Sigma generates text in blocks by denoising continuous token representations. NVIDIA reports a 3.46-fold average decoding speedup for its 8B model in tests on a GB200.

NVIDIA releases Sigma diffusion language models in two sizes

NVIDIA has released a family of language models that generate text in blocks by denoising continuous representations of tokens, with versions at 3 billion and 8 billion parameters. The models, called Nemotron-Labs-Sigma, are designed for reasoning and coding, according to NVIDIA Research.

MarkTechPost reports that the collection contains 14 open checkpoints, including base, instruction-tuned, distilled and self-speculative variants. The 8B base model's card lists its Hugging Face release date as November 12, 2026.

Generating text in blocks

Sigma begins each block with Gaussian noise and repeatedly denoises a representation conditioned on the preceding text. Once the block has been refined, the model reads out discrete tokens and moves to the next block.

The representation uses 16-dimensional token embeddings, learned together with the denoising network. Training starts from pretrained autoregressive weights and uses a shared backbone for an autoregressive stream processing clean tokens and a diffusion stream denoising their embeddings.

During generation, the model can adjust the trajectory using classifier-free guidance and score temperature. It also feeds the previous conditional prediction into the next denoising step, a technique the project calls self-conditioning.

Fewer steps and speculative decoding

NVIDIA says distilled models generate eight tokens per forward pass on math tasks and four on coding, with negligible accuracy loss and preserved diversity. Distribution matching is used to distill the model for generation with fewer denoising steps.

In a separate method, the distilled diffusion model drafts blocks that its own autoregressive stream verifies. NVIDIA says this self-speculative decoding reproduces the autoregressive model's greedy output.

On an NVIDIA GB200, the 8B model achieved an average throughput speedup of 3.46 times over its autoregressive decoding, according to the research project. That figure describes the tested self-speculative method rather than a general speedup for every use of Sigma.

Results and availability

In NVIDIA's evaluations, the 8B base model averaged 71.16 across NVIDIA's six-task suite, ahead of NLD-8B's 70.22, MarkTechPost reports. The same report says Sigma-8B scored 65.12 on HumanEval, against 62.80 for the comparison model, but scored 65.74 on MBPP against 67.00.

The 8B base model's maximum context is 4,096 tokens, including the prompt and generated text. Its model card describes the base version as a starting point for research and fine-tuning rather than an instruction-following assistant; a corresponding instruct variant is available.

The 8B base model runs on a CUDA-capable NVIDIA GPU using bfloat16, and CPU inference is not supported. Its use is governed by the NVIDIA Nemotron Open Model License, and the model card says it is ready for commercial or non-commercial use.

Sources

  1. Original source NVIDIA Research research.nvidia.com

Other pages read

  1. NVIDIA on Hugging Face huggingface.co
  2. MarkTechPost marktechpost.com

Companies in this story

Related stories

Explore

More articles