Hunyuan-A13B Technical Report

AI summary

Overview

Research area: Large language models, specifically sparse Mixture-of-Experts (MoE) architectures, pre-training data curation, and post-training (supervised fine-tuning plus reinforcement learning).

Technical level: Advanced. The report includes detailed architecture hyper-parameters, learning-rate schedules, and multi-stage RL reward designs.

Scope: This paper describes the design, training pipeline, and benchmark evaluation of Hunyuan-A13B, an open-source MoE language model with 80 billion total parameters that activates 13 billion parameters per input.

What This Paper Is About

Deploying state-of-the-art large language models typically demands significant computational resources, producing high inference latency and considerable hardware costs that limit broad accessibility. The authors build Hunyuan-A13B to reduce that cost: a sparse MoE model that activates only a fraction of its parameters per input while aiming to match models with substantially more activated or total parameters. The paper reports how the model was pre-trained, post-trained, and evaluated across reasoning, coding, long-context, and agent benchmarks.

Key Contributions

1. A sparse MoE architecture with a favorable efficiency ratio. Hunyuan-A13B uses 1 shared expert and 64 fine-grained non-shared experts, activating 8 non-shared experts at a time, giving 13 billion active parameters out of 80 billion total. The authors state that experiments on MoE scaling laws showed at least one shared expert helps, while more than one yields diminishing or fluctuating returns.

2. A curated 20T-token pre-training corpus with enhanced STEM data. The team reused the Hunyuan-TurboS data curation pipeline but optimized STEM acquisition and cleaning, extracting 250 billion tokens of high-quality STEM pre-training corpus, and added a refined knowledge labeling system and a multi-dimensional difficulty grading framework.

3. A structured four-step post-training recipe. Reasoning-oriented fine-tuning (SFT then RL) is followed by all-scenarios fine-tuning (SFT then RL), covering mathematical, code, logical, and scientific reasoning first, then broader capabilities such as creative writing, multilingual tasks, agents, and safety.

4. A dual-mode Chain-of-Thought framework. A fast-thinking mode for routine queries and a slow-thinking mode for multi-step reasoning, selected with control tags (/no_think for fast, /think for slow), defaulting to slow-thinking when no tag is given.

Main Findings

• Pre-trained model beats much larger baselines. Compared with Hunyuan team's earlier open-source MoE model Hunyuan-Large (389B total, 52B activated parameters), Hunyuan-A13B performs better on 12 of 14 evaluation benchmarks with only 1/4 the activated parameters and about 1/5 the total parameters. Against Qwen2.5-72B, which has similar total parameter count, it scores higher on almost all benchmarks. Against Qwen3-A22B (235B total, 22B activated), it outperforms on 7 of 12 benchmarks.

• Selective pre-trained benchmark scores. Hunyuan-A13B records MMLU 88.17, MMLU-Pro 67.23, MMLU-Redux 87.67, BBH 87.56, SuperGPQA 41.32, EvalPlus 78.64, MultiPL-E 69.33, MBPP 83.86, CRUX-I 70.13, CRUX-O 77.00, MATH 72.35, CMATH 91.17, GSM8k 91.83, and GPQA 49.12, with a 256K context length.

• Strong slow-thinking reasoning results. In slow-thinking mode, Hunyuan-A13B scores 87.3 on AIME2024 (the highest in the comparison), 84.7 on ZebraLogic (highest), 89.1 on BBH (highest), 94.3 on MATH, 71.2 on GPQA-Diamond, 82.7 on OlympiadBench, and 91.1 on DROP.

• Agent capability is a standout. In slow-thinking mode, Hunyuan-A13B leads on BFCL v3 (78.3), tau-Bench (54.7), ComplexFuncBench (61.2), and C3-Bench (63.5). In fast-thinking mode, it reaches ComplexFuncBench 74.0 and C3-Bench 65.4. The paper notes C3-Bench simulates changes in task combinations.

• Fast-thinking mode trades accuracy for speed. Fast-thinking scores are consistently lower on hard reasoning (AIME2024 30.6, AIME2025 19.2, MATH 85.4, LiveCodeBench 27.4) while remaining competitive on general tasks (BBH 87, DROP 86.5, IF-Eval 84.4).

• Coding is slightly behind the largest models but competitive. The paper states overall coding performance is slightly lower, yet results are comparable to substantially larger LLMs on FullstackBench (67.8 slow / 58.3 fast) and ArtifactsBench (43 slow).

• Long-context performance holds up with length. On PenguinScrolls, Hunyuan-A13B scores 87.7 (Gemini 2.5 Pro 88.3, DeepSeek R1 87.5, Qwen3-A22B 87.1). On LongBench-v2 it scores 55.0, second-highest behind Gemini 2.5 Pro at 60.9. On FRAMES it scores 81.1, above Gemini 2.5 Pro at 80.1 but below DeepSeek R1 at 85.7 and Qwen3-A22B at 84.0, which the paper attributes to room for improvement in RAG-specific long-context handling.

• RULER degradation is gradual. On the RULER QA sub-task, Hunyuan-A13B averages 76.7 (0-8K 78.7, 8K-32K 75.3, 32K-64K 78.0, 64K-128K 73.9), compared with Gemini 2.5 Pro at 81.7 average, DeepSeek R1 at 72.0, and Qwen3-A22B at 73.0. The authors report its performance decay is second only to Gemini 2.5 Pro.

• New internal benchmarks were introduced. These include ArtifactsBench (frontend code generation), LengthCtrl and InsCtrl (text generation control), and ComplexNLU and Word-Task (natural language understanding). Internal test sets were used alongside public benchmarks to reduce potential data contamination bias. ArtifactsBench is stated to be made publicly available soon.

Methodology in Plain English

The team started by assembling a 20-trillion-token training corpus using the same cleaning pipeline as their earlier Hunyuan-TurboS model, but they put extra effort into collecting and cleaning STEM material, resulting in 250 billion tokens of STEM data. They also built a finer-grained labeling system and a difficulty-grading framework so training data could be selected and filtered along multiple dimensions.

The model itself is a Mixture-of-Experts design: instead of running every parameter for every input, it routes each input through a small subset. One shared expert is always active, and 8 of 64 specialized experts fire per input. It uses SwiGLU activations and Grouped-Query Attention to reduce KV cache memory, and reuses Hunyuan-Large's 128K-vocabulary tokenizer.

Pre-training ran in three stages: a foundation stage on 20T tokens with a warmup to a maximum learning rate of 3×10⁻⁴ and a cosine decay to 3×10⁻⁵ over 13.5 trillion tokens at a fixed 4096 context window; a fast annealing stage decaying from 3×10⁻⁵ to 8×10⁻⁶ over 300B tokens with the context window raised to 8192; and a long-context stage that extended the window first to 32K and then to 256K using NTK-aware positional encoding with alpha values of 50 and 1000.

Post-training came in two halves. The reasoning half used supervised fine-tuning on carefully verified chain-of-thought data in math, code, logic, and science, then reinforcement learning with GRPO using a binary outcome reward model and a multilingual code sandbox supporting 36 programming languages and over 1000 concurrent executions. The RL dataset contained 150K samples in a 2:2:1:1 ratio of mathematics to coding to logic to science, with 10% overlapping the SFT data and 90% novel cases; multiple-choice, true/false, and proof-based problems were excluded. Training progressed from 24K to 32K context lengths, dropped the KL divergence constraint, and used sampling temperatures of 0.6–0.8.

The all-scenarios half broadened the model to writing, knowledge QA, instruction following, multi-turn dialogue, and agent tasks, using a generative reward model alongside domain-specific reward services. The paper states this stage unifies a flexible GRM with domain-specific pipelines across 16 sub-topics and over 30 scoring services. Agent data came from a multi-role synthetic engine with five roles, over 30 types of agent system instructions, and 20,000 format combinations.

Why This Matters

Impact on research. The report offers a concrete data point on how much capability a 13B-activated MoE can reach relative to dense and MoE models with far more parameters, and it documents the full recipe — data curation, architecture choices, RL reward design, and dual-mode CoT — as an open-source release under a CC BY 4.0 license. The internal benchmarks (ArtifactsBench, LengthCtrl, InsCtrl, ComplexNLU, Word-Task) add evaluation targets beyond standard public suites.

Real-world applications (per the capabilities evaluated):

• Latency-sensitive serving, where the authors state superior inference throughput makes the model suitable for real-time, resource-constrained scenarios.
• Tool-using agents and task planning, given the reported BFCL v3, tau-Bench, ComplexFuncBench, and C3-Bench results.
• Long-document analysis such as financial reports, legal documents, and academic papers, covered by PenguinScrolls and LongBench-v2.
• Retrieval-augmented generation over extensive contexts, tested by FRAMES and RULER.

Industry relevance. The efficiency ratio (13B activated out of 80B total, with a 256K context length) targets deployment cost, and the paper positions the model as a scalable alternative to dense models of similar scale. The agent-focused data work, including high-frequency tasks such as Excel processing and deep search, points directly at enterprise automation use cases.

Future Directions

• Closing the RAG gap. Hunyuan-A13B's FRAMES score of 81.1 trails DeepSeek R1 (85.7) and Qwen3-A22B (84.0), which the paper itself flags as room for improvement in RAG-specific long-context processing.
• Raising coding performance. The paper acknowledges overall coding performance is slightly lower than leading models, with LiveCodeBench at 63.9 in slow-thinking mode, matching OpenAI-o1-1217 but below DeepSeek-R1-0120 (65.9) and Qwen3-A22B (70.7).
• Extending long-context stability. The RULER results at 64K–128K (73.9) remain below Gemini 2.5 Pro (80.2), suggesting further work on retaining information at the longest context lengths.
• Improving fast-thinking quality. The drop between slow and fast modes on advanced math (AIME2024 87.3 vs 30.6; AIME2025 76.8 vs 19.2) raises the question of how much of the slow-thinking gain can be retained at lower latency.
• Public release of new benchmarks. ArtifactsBench is stated to be made publicly available soon, which would let other groups reproduce the coding evaluation.

Target Audience

This paper is most useful to machine learning engineers and researchers working on efficient LLM training and serving, especially those designing or deploying Mixture-of-Experts models. It also suits practitioners who need an open-source model for agentic tool use, long-document processing, or latency-constrained deployment, and evaluation researchers interested in the internal benchmarks introduced here. Readers without a background in MoE routing, reinforcement learning reward design, or long-context positional encoding will find the pre-training and RL sections dense.

Authors’ abstract

We present Hunyuan-A13B, an open-source large language model based on a Mixture-of-Experts architecture. It contains 80 billion total parameters but activates only 13 billion during inference, balancing model capability, computational efficiency, and deployment cost. The model is pretrained on a rigorously filtered 20T-token corpus with enhanced STEM data curation, improving factual reliability and reasoning ability. High-quality supervised fine-tuning and large-scale reinforcement learning further enhance its overall performance. Hunyuan-A13B also introduces a dual-mode Chain-of-Thought framework that adapts reasoning depth to task complexity: fast thinking for routine queries and slow thinking for complex, multi-step problems. Evaluations show competitive performance across mathematics, science, programming, general language understanding, and agent tasks, often approaching that of much larger models. Its high inference throughput makes it suitable for latency-sensitive applications. We release Hunyuan-A13B to support open research and practical LLM deployment.