The Pulse
Apple Details a 3B Tri-Modal Diffusion Model Trained on 6.4T Tokens
Apple’s research listing spotlights a masked diffusion study spanning text, images and audio, with a preliminary 3-billion-parameter model trained on 6.4 trillion tokens. The paper examines scaling laws, data mixtures and batch-size choices
AI.info Team · Written with AI ·
Apple’s October 2026 research listing features a study of a 3-billion-parameter model trained on 6.4 trillion tokens to work across text, images and audio. The paper, first posted to arXiv on February 25, examines how to train masked diffusion models across the three modalities and how their performance changes with data, model size and sampling choices. Its authors describe the work as a study of the design choices behind tri-modal systems, not a product announcement.
A single model for text, images and audio
The researchers introduce what they call the first tri-modal masked diffusion model pretrained from scratch on text, image-text pairs and audio-text pairs. Rather than generate text one token at a time, as autoregressive language models do, a masked diffusion model repeatedly fills in masked portions of a sequence. In this system, text, image and audio tokens share one vocabulary and one transformer backbone.
The unified design supports tasks including text-to-image generation, image captioning, text-to-speech and automatic speech recognition. The authors write, “Our work represents the largest scale systematic open study of multimodal discrete diffusion models conducted to date, providing valuable insights into scaling behaviors across multiple modalities.” The paper’s authors say the model can generate across modalities without separate modality-specific heads or adapters.
262 models inform the scaling analysis
The study reports training 262 tri-modal models to derive scaling laws, alongside a separate preliminary 3B model trained on 6.4T tokens. The authors vary model size and token budgets, and examine modality mixtures, masking schedules, batch-size effects and inference settings. Their proposed stochastic differential equation reparameterization separates the physical batch size used to make efficient use of compute from the logical batch size used to control gradient variance.
That separation matters because the researchers say it removes the need to search for an optimal batch size, up to a critical threshold. Their experiments also suggest that inference settings such as guidance, temperature and sampling steps need to be selected differently for text, image and audio generation. The result is a set of training and inference recommendations, rather than a claim that one setting works best across all three modalities.
The modalities compete for capacity
The paper’s mixture experiments offer a qualification to the promise of joint training. In tests using 320-million-parameter models trained on 13 billion tokens, the researchers report no measurable synergy between modalities: each modality’s loss improved when its share of the training mixture increased, while the other modalities’ shares did not produce a corresponding benefit. They conclude that the modalities compete for model capacity and tokens at those tested scales.
The team used a one-third share for text, image-text and audio-text in most experiments, except for the dedicated mixture ablation. The paper says this balanced split appeared reasonable in that setup; it does not establish that equal weighting is best for larger models or different data collections. That distinction matters when interpreting the 3B model: its unified architecture demonstrates feasibility, but does not show that adding modalities automatically improves each task.
Benchmarks show breadth, not a product release
The 3B model was evaluated after pretraining, without a separate fine-tuning stage described in the paper’s results section. On GenEval, its overall score was 48.89; on the LibriSpeech-PC speech test, the authors report a word error rate of 0.164. Those results cover different tasks and evaluation methods, so they do not by themselves establish that the model outperforms specialist image, speech or language systems.
Apple’s research page and the paper present the work as an empirical study of masked diffusion’s design space. The paper provides evidence that one backbone can handle text, image and audio generation tasks, while also documenting the training trade-offs that come with combining them. Its most concrete caution is in the mixture study: at the scales tested, sharing a model did not create cross-modal gains.