Think Twice: The Metis-1.6 Technical Report

AI summary

Overview

Research area: Machine learning — large language model architecture, pretraining, and post-training (mixture-of-experts, hybrid state-space/attention models, looped depth, reinforcement learning with verifiable rewards).

Technical level: Intermediate. The report is written in plain, first-person prose and explains its own failures clearly, but it assumes familiarity with mixture-of-experts routing, Mamba-2 blocks, GRPO-style group-normalized policy gradients, and supervised fine-tuning pipelines.

Scope (one sentence): A single-researcher technical report documenting two from-scratch looped mixture-of-experts language models (Praxis and Logos), their full training lineage on 1.02T tokens, an honest multi-benchmark evaluation, and an unusually candid catalogue of pretraining and post-training mistakes.

What This Paper Is About

The paper argues that better data, better training methods and new architectures can buy capability that a larger parameter count cannot, at a fraction of the cost — and tests that bet with two models built entirely from scratch by one researcher. Praxis (2.94B parameters, 0.46B active per pass) and Logos (10.23B parameters, 1.18B active per pass) both run a hybrid Mamba-2 and attention mixture-of-experts stack twice for every token, so each token gets a second look. The goal is not to beat frontier models but to measure honestly where a small, cheap, heavily-iterated model lands, and to record what went wrong along the way so it is not repeated.

Key Contributions

1. Two open models with complete training lineage. Praxis and Logos (plus -Base counterparts) are released with checkpoint hashes and exact parameter accounting, evaluated by a single harness against open models of similar size — 27 at the base stage and 28 after post-training.

2. An audit of the pretraining optimizer. The declared AdaMuon optimizer was never instantiated: the trainer never read the optimizer name, and a parameter-routing mismatch meant every attention matrix trained with AdamW for all 804.76B pretraining tokens.

3. A negative result on hashed n-gram memory. In both models the learned injection gates closed on their own; deleting the 1.77B-parameter table from Logos changed held-out loss by less than 10⁻³ nats, and the released checkpoints ship without it.

4. Post-training lessons and a proved bug. Five supervised fine-tuning failure modes are documented, along with a verifier imbalance that shortened reasoning on unseen science questions, and a sign inversion of length rewards under group-mean normalization that is proved and fixed with a one-line clamp.

5. Controlled ablations for the next model. 10B-token ablations of the loop design and optimizer behind MoRE (Mixture of Recursive Experts), the architecture planned for Metis-1.7.

Main Findings

• Logos averages 52.93 across 40 post-trained benchmarks. That is seventh of 17 models in its comparison set and 0.75 points behind Qwen3-4B — not a frontier result, but achieved on 1.02T training tokens against a median of 12T for comparators that disclose a count, with every model above it that discloses its data trained on at least 28T.

• It sits on the Pareto frontier for both data and compute. Logos activates 2.36B parameters per token, while every model that beats it activates between 1.5 and 4.6 times as many. Praxis averages 36.62 and also sits on the compute frontier. As base models, both trail open models of similar size, which the author attributes mainly to training on one to two orders of magnitude fewer tokens.

• Mathematics is the standout strength. Logos ranks second of 30 models, behind only Gemma 4 12B, with 67.50% on AIME 2025 and 63.64% on HMMT February 2026, and the highest score in its set on HMMT February 2025 (62.92), HMMT February 2026 (63.64) and IMO-AnswerBench (57.00).

• Code, instruction following, long context and factual recall are weak. These trail models trained on a small fraction of the data used by large labs.

• The optimizer that ran was not the optimizer configured. Four discrepancies held for the whole of pretraining: the optimizer name was never read (plain Muon with Nesterov 0.95 and five Newton–Schulz steps, no second-moment state); attention fell through to AdamW; the fused Mamba-2 input projection was orthogonalized as one block, coupling five functionally different projections; and weight RMS drifted (by factors of 0.57 for convolutions to 1.49 for the final norm between 100B and 600B tokens). The loss curve never signalled any of it.

• The n-gram memory switched itself off. Mean gates after pretraining were between 0.0001 and 0.0031 at every injection site, and the memory's update to a residual stream stayed below 0.85% of the stream's norm. Removing it moved bits per byte from 0.733898 to 0.733894 on Logos, and shrank the BF16 checkpoint from 24.35 GB to 20.80 GB. Two suspected causes: redundancy with early Mamba-2 convolutions, which already compute local 2-, 3- and 4-gram compositions, and hash-collision noise across roughly 1.73M rows per table over 805B tokens.

• Context extension worked. On Logos, needle-in-a-haystack top-1 retrieval at 131,072 tokens rose from 19.53% to 68.75% at the 12B-token gate and 66.93% at 18B, tail loss on full-length documents fell from 5.689 to 1.495 nats, and RULER reached 84.48% at 4K, 73.25% at 32K and 60.14% at 128K, while the seven-task short-context mean barely moved (62.65 to 62.24). Praxis rose from 17.97% to 65.63% at 18B, with RULER at 39.01% at 128K.

• A verifier imbalance shortened reasoning on science. In the GSPO stage, 75.3% of prompts were formatting and tool-calling tasks, 24.7% were mathematics, and none were science. Think-mode GPQA Diamond fell from 52.02% to 43.43% and pooled AIME from 60.00% to 40.00%, with responses 5.2 and 6.3 times shorter respectively, while think-mode IFEval rose 13.41 points to 84.48%.

• Length rewards inverted sign once prompts got easy. Under group-mean centring, with a median length split and λ = 0.5, every long correct sample has negative advantage if and only if the group pass rate p > 2/3 (at least 11 of 16 samples). The reasoning specialist followed the predicted pattern: gains at step 40 and step 50, then regression at step 78 once mean group pass rate crossed 56.7%. The fix is clamping the advantage of correct samples at zero, max(0, A_i).

• The distillation step was too small to matter. One optimizer step at warm-up speed moved the weights about 100 times less than a real run, and the best teacher (the reasoning specialist's step-40 and step-50 checkpoints) had already been removed by automatic checkpoint pruning. It still nudged several results upward, e.g., HMMT February 2026 from 12.12% to 21.21%.

• Five supervised fine-tuning choices cost capability, including direct-mode examples made by deleting reasoning blocks from 15–25% of traces, inherited verification loops (median 42,377 tokens on HMMT February 2026, with 36.36% truncated at 64K), a hidden teacher mix (74.2% of one SFT dataset came from early-2025 teachers), almost no identity data (4,800 rows), and off-policy token-level cross-entropy in Stage 2.

Methodology in Plain English

The author builds everything himself — data acquisition and decontamination, tokenization, training kernels, reinforcement-learning environments and evaluation — so any part can change as soon as a better idea arrives.

Architecture. Rather than stacking more layers, the model runs its hybrid trunk twice per token with all weights shared. Logos has 20 layers of width 2,560 (17 Mamba-2 blocks and 3 attention layers at layers 5, 10 and 15), so each token passes through 40 sequence-mixing and expert stages while only half as many are stored. Every layer ends in a mixture-of-experts block whose experts operate in a 1,024-dimensional latent space: one always-on shared expert plus 4 of 192 routed experts in Logos, 4 of 128 in Praxis. Four residual streams are read, written and mixed at each sub-layer by a controller conditioned on the current pass. Attention uses no positional encoding, which is what allows the models to stretch from 4,096 to 131,072 tokens without changing position handling, and a cross-pass depth memory lets the second pass consult notes written in the first.

Data and pretraining. Both models trained on a sealed corpus of 806 provenance-hashed shards totalling 804,755,808,835 tokens, served in three curriculum phases (broad web/books/code at 68.9%, up-weighted maths and science at 26.1%, and an annealing mix at 5.0%), with every shard decontaminated against evaluation benchmarks by 13-gram shingle matching. Logos took 219,280 optimizer steps on AMD Instinct MI300A accelerators.

Context extension. From the 4,096-token checkpoints, both models continued pretraining on 18B tokens of long documents packed to at most 163,840 tokens, gated on needle retrieval, document-tail loss, short-context retention on seven zero-shot tasks, and the 13-task RULER suite.

Post-training. A cold-start stage on 120.01B tokens of long reasoning traces, then an overall stage on 80.00B tokens spanning instruction following, tool calling and conversation — the released model is the end of Stage 2. From there, a six-stage study of hybrid-mode GSPO and five domain specialists, followed by on-policy distillation of the specialists back into one model (none of which became the released checkpoint). Comparisons use one harness that ran Metis and every comparison model with identical prompts and scorers, with every table generated from benchmark data published alongside the models.

Why This Matters

Impact on research. The most distinctive contribution is the negative-result record. Release reports usually describe the recipe that worked; this one documents an optimizer that silently differed from its configuration for an entire pretraining run, a 1.77B-parameter component the model learned to switch off, and a provable sign inversion in a widely used reward-normalization scheme. The length-reward proposition is a clean analytical result that applies to any group-mean-centred objective with a length penalty, not just to these models. The report also gives a concrete existence proof for the "out-iterate, don't outspend" thesis: a Pareto-competitive model at roughly a twelfth of the median training data of its comparators.

Real-world applications:

• Small teams and independent researchers deciding where to spend a limited compute budget between data, method and parameters.
• Practitioners building reinforcement-learning pipelines with verifiable rewards, who need to know that length rewards invert once prompt pass rates exceed 2/3.
• Engineers deploying models at long context, where a hybrid stack with only 3 of 20 layers keeping a key-value cache makes memory grow slowly.
• Anyone auditing a training run, for whom the "assert every parameter's optimizer group at start-up" lesson is directly actionable.

Industry relevance. The failure catalogue is a checklist for anyone running multi-hundred-billion-token campaigns: silent optimizer-group fallthrough, checkpoint pruning that deletes the best teacher, verifier prompt mixes that quietly teach brevity on graduate-level science, and container orchestration that turns 15-minute rollout rounds into hours. The architecture, released checkpoints and per-benchmark data make the results reproducible.

Future Directions

• MoRE for Metis-1.7. The report closes with 10B-token controlled ablations of the loop design and optimizer behind Mixture of Recursive Experts; the excerpt provided does not include those results or the measured value of the corrected optimizer.

• A larger, off-accelerator n-gram memory. Rather than abandoning the idea, the plan is a 50B-parameter table stored off-accelerator in a per-row ternary format (BitNet b1.58, 1.85 bits per value) and trained with quantization-aware straight-through estimation, so it can hold long-tail facts instead of competing for accelerator memory.

• Fixing the post-training pipeline. Concrete planned changes include 12.0B tokens of natively generated direct-mode answers, filtering traces with more than two post-solution verification passes, removing all 6.93M early-2025 teacher rows, adding 2.5B tokens of house-voice writing plus 600M epoch-weighted identity tokens, a reference-anchored sampling objective for Stage 2, and a 48- to 64-step domain-balanced bridge in place of the imbalanced GSPO stage.

• Open questions. Whether the reported architecture conclusions survive a correctly configured optimizer; whether the length-reward clamp holds up across specialist domains beyond the four that trained with it; and how much of the remaining gap to larger models is data volume versus method, given that the base models trail similar-size open models attributed mainly to token count.

Target Audience

Independent researchers and small teams training language models without a frontier budget; ML engineers responsible for large pretraining or reinforcement-learning runs who want a checklist of silent failure modes; architecture researchers interested in looped depth, hybrid state-space/attention stacks, latent mixture-of-experts and conditional memory; and anyone evaluating small open models who wants per-benchmark results measured under a single harness rather than assembled from model cards.

Authors’ abstract

Metis is a bet that better data, better training methods and new architectures can buy what a bigger parameter count cannot, at a fraction of the cost. Metis-1.6 is its latest step: two language models built from scratch by a single researcher, Praxis (2.94B parameters, 0.46B active per pass) and Logos (10.23B parameters, 1.18B active per pass). Both run a hybrid Mamba-2 and attention mixture-of-experts stack twice for every token, so each token gets a second look. Each was trained on 1.02T tokens in total: 804.76B of pretraining, 18B of context extension to 131,072 tokens, and 200B of supervised fine-tuning. Metis-1.6 was never meant to challenge frontier models, and across 40 post-trained benchmarks it mostly does not: Logos averages 52.93, seventh of 17 models in its comparison set and within a point of Qwen3-4B. But it gets there on roughly a twelfth of the median training data of the comparators that disclose it, and with fewer active parameters per token than any model that beats it, which puts it on the Pareto frontier for both. In mathematics it ranks second of 30 models, with 67.50% on AIME 2025 and 63.64% on HMMT February 2026. It is weak at code, instruction following and long context. The rest of the report is the part I found most valuable: what went wrong. The optimizer I configured never ran, and attention trained with AdamW for all of pretraining. A 1.77B-parameter n-gram memory learned to switch itself off. Five fine-tuning choices cost capability. And a length reward under group-mean normalization punished correct solutions for being long once a group's pass rate passed 2/3, a sign inversion I prove and fix with a one-line clamp. I close with the 10B-token ablations behind MoRE (Mixture of Recursive Experts), the looped architecture of Metis-1.7.