Universal Textual Teaching for LLMs

AI summary

Overview

Research area: Knowledge distillation and prompt engineering for large language models (LLMs). Technical level: Intermediate — the paper is readable without deep math, but it formalizes distillation with optimization notation and assumes familiarity with prompting methods and benchmarks. Scope: The paper proposes Universal Textual Teaching (UTT), a framework that transfers knowledge from a stronger Teacher LLM to a weaker Student LLM purely through a written natural-language document called a Primer, without updating any model weights.

What This Paper Is About

Most knowledge distillation for LLMs works by fine-tuning the Student's parameters on Teacher outputs, which locks the transferred knowledge inside a specific model checkpoint, makes it hard to inspect, and rules out API-only or expensive-to-train models. The authors ask whether the Teacher-Student performance gap can instead be captured as an explicit, readable, reusable piece of text that is simply placed in the Student's context. UTT answers this by identifying problems the Teacher solves and the Student fails, then having several LLM roles collaboratively write and refine a "Primer" that teaches the Student those missing skills.

Key Contributions

1. A language-based reframing of distillation. The paper replaces the usual objective of optimizing Student parameters with an objective of optimizing a discrete text artifact (the Primer), with both Teacher and Student held fixed, making the distilled knowledge auditable and portable across models.
2. A parameter-update-free closed-loop framework. UTT combines paired Teacher-Student evaluation (to find where the gap is) with a four-role interaction loop (Student attempt, Prompter diagnosis, Teacher demonstration, Synthesizer consolidation) plus a validation gate that accepts a new Primer only if it does not reduce accuracy on the current batch.
3. Empirical validation on hard math and code tasks. On Omni-MATH-2 and KernelBench, UTT raises the Student's accuracy from 9.4% to 48.6% (KernelBench), Fast 1 from 9% to 35%, and math accuracy from 27.6% to 51.7%, beating representative prompt-engineering and parameter-updating distillation baselines.
4. Evidence of cross-model transferability. A Primer synthesized for one Teacher-Student pair improves other Students that took no part in its synthesis; in the Figure 1 setting, Qwen3.6-27B gains 41.8 and 31.5 absolute points on KernelBench and Omni-MATH-2 respectively from Primers it did not help create.

Main Findings

• Large gains on the source Student: In the Pro→Flash configuration, the Primer lifts KernelBench accuracy from 9.4% to 48.6% (+39.2 points), Fast 1 from 9% to 35% (+26), and Omni-MATH-2 accuracy from 27.6% to 51.7% (+24.1).
• Effectiveness across model pairs: Synthesizing a Primer for Qwen with Pro as Teacher yields 50.0% KernelBench accuracy, 32% Fast 1, and 62.4% math accuracy. With Opus as Teacher, Flash reaches 48.2% KernelBench accuracy, 39% Fast 1, and 46.6% math accuracy.
• Primers transfer without resynthesis: The Flash-derived Primer improves Qwen by 32.0, 25, and 25.8 points (KernelBench accuracy, Fast 1, math); the Qwen-derived Primer improves Flash by 34.4, 21, and 16.1 points.
• Both Teacher and Student can be swapped: An Opus-derived Primer applied unchanged to Qwen, which never participated in synthesis, yields 51.0% KernelBench accuracy, 22% Fast 1, and 65.8% math accuracy.
• Generalization to unseen problems: On the distillation training set, Pro→Flash KernelBench accuracy rises from 6.7% to 65.2%; on the held-out distillation test set, Pro→Qwen math accuracy rises from 9.7% to 64.8%.
• Existing abilities are largely preserved: On the retention subset, KernelBench accuracy improves from 34.0% to 74.0%, 27.0% to 81.0%, and 37.1% to 78.6% across the three configurations; on Omni-MATH-2, Pro→Flash improves from 58.0% to 69.0%, while Pro→Qwen and Opus→Flash drop by 9.2 and 14.2 points but keep most of their baseline performance.
• Some frontier extension: On problems neither model solved at baseline, the primed Student is nonzero in five of six configurations, with KernelBench reaching 11.7% to 18.6% and Omni-MATH-2 at 5.3% (Pro→Flash) and 30.0% (Pro→Qwen).
• Outperforms prompt engineering: UTT is best on all six task/model combinations in Table 2. Strong baselines include GEPA (41.1% math on Flash, 34.8% KernelBench accuracy on Flash) and MIPROv2 (45.0% KernelBench accuracy on Flash, 44.6% on Qwen).
• Outperforms parameter-updating distillation: Under Pro→Qwen, UTT (62.4% math, 50.0% KernelBench accuracy, 32% Fast 1) exceeds the strongest baseline by 7.2, 8.2, and 4 points; the best competing methods are LUFFY (55.2% math), Fine-tune-CoT (41.8% KernelBench accuracy, 28% Fast 1), and SeqKD (53.5% math).
• Naive Teacher summarization is insufficient: A "Teacher summary" prompt helps Flash (36.7% math, 40.0% KernelBench accuracy) but is inconsistent on Qwen (32.8% math, 10.2% KernelBench accuracy), which the authors read as evidence that interactive Teacher-Student exchange matters.

Methodology in Plain English

The pipeline has three stages, and no model weights ever change.

1. Find the gap. Both the Teacher and the Student answer every problem several times, and each answer is checked for correctness. Problems are sorted into three buckets: distillation (Teacher right, Student wrong — the teaching material), retention (Student already right — used to check nothing is broken), and frontier (both wrong — used to check for unexpected gains). The distillation bucket is split 7:3 into a training portion used for writing the Primer and a test portion held back for evaluation.
2. Write the Primer in rounds. The training problems are processed in batches (default batch size 16). For each problem the Student tries it with the current Primer in context and receives evaluator feedback. If the Student is right, the loop moves on. If wrong, the Prompter reads the problem and feedback and writes a short teaching instruction describing the underlying skill gap; the Teacher then produces a worked solution guided by that instruction. A record is kept only when the Student's attempt failed and the Teacher's demonstration passed the checker, so bad demonstrations are filtered out. The Synthesizer merges these records into the existing Primer, updating sections rather than appending a log, and is told to keep it a compact study guide within a word limit.
3. Validate before accepting. The candidate Primer and the current Primer are both scored on the same batch; the candidate is kept only if it gets at least as many correct Student outputs. The final Primer is then placed in the Student's context at inference time.

Setup details: KernelBench uses all 100 Level 1 problems; Omni-MATH-2 uses 200 problems from difficulty levels 5–10. Each model generates five responses per problem (pools of 500 and 1,000 samples). KernelBench kernels and PyTorch references are run on an NVIDIA RTX A6000. Max output length is 16,384 tokens for KernelBench and 32,768 for Omni-MATH-2; temperature is 0 for DeepSeek and Qwen and unspecified for Opus. Models are DeepSeek V4 Pro, DeepSeek V4 Flash, Qwen3.6-27B, and Claude Opus 5, in three configurations: Pro→Flash, Pro→Qwen, and Opus→Flash. The Teacher model also fills the Prompter and Synthesizer roles.

Why This Matters

Impact on research: The paper offers an alternative axis for distillation research — instead of asking how to better fit a Student to Teacher outputs, it asks how to verbalize the Teacher's advantage. Because the artifact is text, results are inspectable and editable, and the same artifact can be evaluated, shared, and re-applied without retraining. It also draws a clear line between UTT and automatic prompt optimization: UTT selects teaching content from an observed Teacher-Student gap rather than searching directly for task accuracy.

Real-world applications:

• Improving models behind commercial APIs where fine-tuning is impossible or contractually restricted.
• Upgrading a small self-hosted model for a specialized task (e.g., GPU kernel writing) by borrowing a frontier model's expertise once, then deploying offline.
• Sharing one audited teaching document across a fleet of heterogeneous models rather than maintaining per-model checkpoints.
• Cheap domain adaptation where the distillation artifact can be reviewed by humans before deployment.

Industry relevance: Eliminating training runs removes GPU cost and MLOps complexity, and a text artifact can be version-controlled and inspected for compliance in a way that weight updates cannot. The KernelBench results are directly relevant to teams generating GPU kernels, where correctness plus speedup against a PyTorch reference is the practical bar.

Future Directions

• Stress-testing transfer more broadly: the paper tests three Teacher-Student configurations; whether a single Primer can serve a large, heterogeneous model pool — and how it degrades — remains open.
• Preventing retention regressions: Omni-MATH-2 retention dropped by 9.2 and 14.2 points in two configurations, so the validation gate (which checks only the current batch) may need stronger or broader safeguards.
• Scaling and cost analysis: the number of batches, primer word limit, and the compute/API resources consumed are referenced to appendices that are truncated here, so the practical cost of synthesis is not reported in the visible text.
• Extending beyond text-only tasks: the authors report only a preliminary confirmation of transfer to multimodal LLMs (Appendix B.1), leaving visual and other modalities largely unexplored.

Target Audience

Researchers and engineers working on LLM distillation, prompting, and inference-time adaptation; practitioners who need to strengthen a model they cannot fine-tune (API-only or budget-constrained settings); and readers interested in interpretable, portable alternatives to parameter-based knowledge transfer. The paper assumes comfort with benchmark tables and standard prompting terminology but does not require deep mathematical background.

Authors’ abstract

Knowledge distillation (KD) transfers knowledge from stronger Teacher models to weaker Student models, but most methods require training the Student parameters, thereby binding the distilled knowledge to a specific architecture and checkpoint. This implicit representation is difficult to interpret or reuse across models and limits KD for API-only or costly-to-train models. This paper studies knowledge transfer for large language models (LLMs). We introduce Universal Textual Teaching (UTT), a parameter-update-free framework that distills observed Teacher-Student knowledge gaps into a textual, interpretable, and reusable natural-language artifact called Primer. Specifically, UTT first identifies representative gap cases through paired evaluations, and iteratively updates the Primer via multi-role interactions: the Student attempts each task, the Prompter turns evaluation feedback into a teaching instruction, the Teacher provides a targeted demonstration, and the Synthesizer consolidates validated lessons. Empirically, on the challenging math (Omni-MATH-2) and code generation (KernelBench) tasks, extensive results confirm the effectiveness of the method: UTT remarkably raises the Student's accuracy from 9.4% to 48.6% and Fast1 accuracy from 9% to 35% on KernelBench, while increasing mathematical reasoning accuracy from 27.6% to 51.7%. UTT also performs better than representative prompt engineering and parameter-based KD methods. Of note, UTT is shown to be generalizable across different Teachers and Students: a Primer synthesized for one Teacher-Student pair can generalize to other Students that do not participate in the synthesis.