MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code

AI summary

Overview

Research area: Machine learning systems — GPU kernel optimization agents, domain-specific languages (DSLs), and evaluation methodology for agent knowledge reuse.

Technical level: Intermediate. Readers should know roughly what a GPU kernel is and what an LLM agent loop looks like; the paper's statistics are explained in plain terms.

Scope: The paper introduces MetaKernelBench, a 74-problem benchmark that measures whether a natural-language "skill" distilled from one kernel-optimization attempt in one DSL (CuTe DSL) helps a fresh attempt at the same problem in another DSL (TIRx), across six models and both transfer directions.

What This Paper Is About

GPU kernel optimization agents now save what they learn — in knowledge bases or as distilled "skills" — but existing kernel benchmarks only score each attempt's correctness and speed, never whether the retained experience is actually worth reusing. This paper asks a narrower, cleaner question: if an agent solves (or fails) a problem in one kernel DSL and writes down what it learned in natural language, does that note improve a new attempt at the same problem in a different DSL? MetaKernelBench answers this by pairing every problem in two DSLs and comparing skill-conditioned attempts against solo attempts under matched budgets.

Key Contributions

1. A new evaluation dimension. The authors formulate same-problem, cross-DSL kernel knowledge transfer as a metric that complements per-implementation correctness and speed, rather than replacing them.
2. The benchmark itself. MetaKernelBench pairs 74 fused-subgraph problems as CuTe DSL and TIRx variants that differ only in the DSL, drawn from SOL-ExecBench (42 problems), FlashInfer-Trace and FlashInfer-Bench (20), vLLM tests and kernels (8 cache-maintenance problems), and flash-linear-attention (4 linear-attention problems).
3. A transfer protocol with no success filter. The source skill is transferred verbatim whether or not the source attempt passes verification, the source implementation and trajectory are withheld, and the skill-conditioned attempt is compared to the target variant's solo attempt under matched agent configuration and per-attempt budget.
4. An empirical characterization plus diagnostics. Transfer is measured across six models in both directions, then explained through content labeling of skills (an 11-category rubric over 841 labeled skills), two worked case traces, and a six-class failure taxonomy.

Main Findings

• Transfer is real but small and unstable. Across six models and both directions, paired lift over solo attempts ranges from −19.1% to +29.3%. Four models gain in both directions, by 2.3% to 29.3%; GLM-5.3-Flash is negative in both directions; Qwen3.8-Flash gains 4.7% from CuTe DSL skills and loses 2.6% from TIRx skills.

• Regressions never disappear. Every model in every direction harms some problems, at rates of 16.2% to 44.6%, alongside help rates of 21.6% to 40.5%.
• Most effects are runtime, not pass/fail. Verdict changes account for 76 of the 277 helped comparisons and 75 of the 223 harmed ones; the rest are runtime differences between attempts that both pass.
• Aggregate numbers hide opposing families. Pooled lifts mix families with opposite signs in 11 of the 12 model-and-direction cells. GLM-5.3 gains 67.9% on KV Cache from TIRx skills but loses 40.9% from CuTe DSL skills, with the signs reversed on Linear Attention. GPT-5.6 Sol's 8.7% CuTe→TIRx lift comes from the 10 Quantization problems, where it reaches 99.1% against −1.1% on the other 64. Removing each cell's best family turns that lift and three of the other four positive CuTe-skill lifts negative, while all four positive TIRx-skill lifts stay positive.
• The source attempt's standing matters more than its verdict. When the source attempt stands above the target's solo attempt, 71.2% of comparisons are helped and 8.1% harmed; at the same tier, 25.7% and 21.8%; when the source stands below, 10.6% helped and 53.8% harmed (pooled over models and directions). Within each model, 55.0% to 94.1% are helped when the source stands above, and 35.3% to 72.4% are harmed when it stands below. At the same tier, the three larger models still retain lifts of 7.2% to 16.4%.
• Skills are not interchangeable across directions. Among the 277 helped comparisons, the reverse direction on the same model and problem is harmed 41.5% of the time and helped 23.1%, with harm exceeding help for all six models — an asymmetry the authors argue reflects the transferred skill, not chance.
• Solo quality and transfer differ by model. DeepSeek-V4-Pro passes every variant in both DSLs but has the lowest passing-only geometric mean speedups, 1.8× for CuTe DSL and 2.4× for TIRx. Qwen3.8-Flash has the lowest pass rates (64.9% CuTe DSL, 62.2% TIRx) yet the highest TIRx speedup, 8.7×. Every model's TIRx speedup exceeds its CuTe DSL speedup, while pass rates favor neither DSL. The four models with the highest pooled solo pass rates are the four with positive lift in both directions, but the authors caution that six models are too few to conclude that solo pass rate predicts lift. Gemini 3.7 Flash's scored TIRx pass rate excludes 21 verifier-passing implementations that used inline CUDA.
• Certain skill content travels badly. "Unsupported closure" — conclusions that rule out alternatives beyond the stated evidence — is 7 to 17 percentage points more prevalent in harmed than helped comparisons for five of six models. "Agenda handoff," which leaves suggestions open for later work, is 11 to 27 points more prevalent in harmed comparisons for the three larger models. "Verified reformulation" is 7 to 9 points more prevalent in helped comparisons for Gemini 3.7 Flash, DeepSeek-V4-Pro, and GLM-5.3, but 12 points more prevalent in harmed comparisons for GPT-5.6 Sol. "Ambition capping" is 21 points more prevalent in helped comparisons for GLM-5.3-Flash but 11 points more in harmed comparisons for DeepSeek-V4-Pro.
• Big models are misled; small models stumble on their own. The two skill-side failure classes (search truncation and unverified adoption) account for 65% to 86% of assigned causes for the three larger models and 39% for Gemini 3.7 Flash, against 33% for GLM-5.3-Flash and 6% for Qwen3.8-Flash. For those two, faulty execution, toolchain friction, and budget or artifact mismanagement account for 63% and 94%. The single largest cause is search truncation for DeepSeek-V4-Pro (8.8%) and GPT-5.6 Sol (10.1%), unverified adoption for GLM-5.3 (11.5%), faulty execution for GLM-5.3-Flash (14.2%) and Qwen3.8-Flash (8.8%), and rule and precision failure for Gemini 3.7 Flash (10.1%).
• Harm happens while the skill is being used. The skill is followed fully or partially in over 80% of harmed comparisons and fully in 39% to 76% of them, versus 72% to 86% of helped comparisons. The three larger models follow it fully 6 to 16 points less often when harmed than when helped; the three Flash-tier models follow it 24 to 39 points less often. Full-follow share ranges from 77% to 85% on Attention and Linear Attention against 56% to 75% on KV Cache and MoE for the larger models, and has a rank correlation of 0.49 with paired lift across the 36 model-and-family cells.
• Bad claims propagate. Unverified source claims reappear in the attempt's own skill in 17% to 68% of harmed comparisons versus 5% to 32% of helped ones, for every model.
• Two illustrative traces. On problem p68, a GLM-5.3 CuTe→TIRx transfer is helped: the TIRx attempt reuses a recorded thread mapping, reruns a recorded diagnostic, fixes a suspected indexing fault, and fuses a recurrence the source could not. On p71, a TIRx→CuTe transfer is harmed: the attempt cites the source's unsupported claim ruling out folding the recurrence, keeps eager execution, and never implements the native recurrence its solo attempt did.

Methodology in Plain English

The design is a paired A/B experiment built so that the only thing that changes between the two arms is the transferred note.

Problems. 74 fused subgraphs from production models, serving stacks, and reference implementations are assigned to six families by their main implementation constraint: Attention (18), KV Cache (8), Linear Attention (13), MLP & Norm (17), MoE (8), and Quantization (10). Each has a DSL-neutral specification: a task description, a PyTorch reference function, and a generator of fresh inputs for a fixed workload. A candidate was kept only if its defining behavior survives the grading contract, it has real optimization headroom, and it can be written natively in both DSLs.

Two variants. Each problem becomes two task variants that differ only in the DSL — CuTe DSL, which uses layout algebra to partition data and computation across threads, and TIRx, which uses storage layouts to guide compiler dispatch of tile primitives. Both are embedded in Python, so implementations may mix eager PyTorch, plain Python, and DSL kernels; the eager/DSL split is itself an optimization choice. Other kernel languages and compilers, CUDA Graphs, calls to the reference, and kernels imported from DSL kernel libraries are forbidden, and a "bypass test" rejects a DSL step if skipping it still yields valid outputs. Timing covers the whole invocation, including Python dispatch.

The protocol. Every attempt runs in a CPU-only sandbox with two cores, 8 GB of memory, and no network, using Terminal-Bench's Terminus-2 scaffold through Harbor 0.20.0, with a 40-minute wall-clock budget and at most 10 feedback submissions. Each attempt is told to maintain a natural-language skill recording evidence-backed findings, work partitioning, failure reasons, applicability conditions, and open questions, with code, commands, and API instructions forbidden. The agent attempts each variant solo and writes such a skill. That skill — and only that skill — is attached verbatim, under a neutral header, to a fresh attempt by the same model on the other DSL, with the source code and trajectory withheld. Attempts are scored by a verifier using float64-anchored correctness (five correctness draws, 10 warmup calls, 50 timed calls, correctness rechecked on three timed outputs) and end-to-end runtime against the eager reference, with speedups capped at 100×.

The metric. For each resolved comparison, the ratio of baseline to treatment runtime is clipped to [1/4, 4] when both arms pass; otherwise the comparison scores 4 for a failure-to-pass rescue, 1/4 for a pass-to-failure break, and 1 for joint failure or a skipped treatment. This makes a verdict flip weigh as much as the largest retained runtime change. Aggregates are paired lift (a geometric mean of those ratios, minus one), help rate (share with ratio ≥ 5/4), and harm rate (share ≤ 4/5), reported by model, direction, and family. Skipped treatments — Qwen3.8-Flash has 43 missing-skill treatments — inherit the baseline outcome and score as neutral.

Analysis layers. Beyond the numbers, LLM annotators label each of 841 skills against a fixed 11-category rubric, record whether the skill-conditioned attempt followed the source skill fully or repeated its unverified claims, and assign one primary cause from a six-class taxonomy to each harmed comparison with a target-side failure.

Why This Matters

Kernel benchmarks assume that better agents come from better single-shot implementations. This paper argues that retained experience has its own, separately measurable value — and that naive reuse can actively destroy performance. That reframes how agent memory and skill libraries should be evaluated: a skill is not free knowledge, it is an intervention with a help rate and a harm rate.

Real-world applications:

• LLM serving stacks. The benchmark's families — Attention, KV Cache, MoE, Quantization, MLP & Norm — are exactly the kernels that dominate inference cost, and the paper's source material comes from vLLM, FlashInfer, and production model subgraphs.
• DSL migration and portability. Teams moving kernels between CuTe DSL and TIRx (or evaluating whether to) get evidence that strategy-level knowledge transfers partially while syntax-level recipes do not.
• Agent memory and skill-library design. The finding that source standing (pass tier plus speed) predicts transfer better than source success alone gives a concrete filter for what to store or retrieve.
• Autotuning and compiler pipelines. The distinction between eager-PyTorch/DSL work partitioning and kernel strategy shows which decisions can be carried over and which must be re-derived per DSL.

Industry relevance: GPU kernel engineering is expensive and increasingly automated. If distilled skills can be filtered for the harmful patterns the paper identifies — unsupported closure and open-ended agenda handoff — agent pipelines could reuse optimization knowledge across DSLs without paying for regressions.

Future Directions

• Filtering and validating skills before transfer. The content analysis points at specific harmful patterns; whether suppressing unsupported closure or agenda handoff during skill writing raises paired lift is untested here.
• Generalization beyond the paired setting. The authors state explicitly that this design does not test transfer to new problems, new workloads, or new hardware — only the same problem in another DSL.
• Isolating the contribution of each guidance type. Paired lift measures the combined outcome and does not separate the effect of eager-PyTorch advice, work partitioning, and kernel strategy, nor does it equalize total compute, since producing the source skill costs a separate 40-minute attempt.
• Scaling the model panel and closing the skill loop. Six models are described as too few to establish that solo pass rate predicts lift, and the observed propagation of unverified claims into the next generation of skills raises the question of whether multi-hop skill chains degrade further.

Target Audience

Researchers and engineers working on LLM agents for code and kernel generation, benchmark designers interested in evaluating retained experience rather than single attempts, GPU performance engineers comparing DSLs, and ML systems teams building agent memory or skill libraries. Readers focused on raw kernel speed records will find the paper's contribution elsewhere; its subject is the reusability of what an agent writes down.

Authors’ abstract

Recent GPU kernel optimization agents retain what they learn in knowledge bases or as distilled skills. Kernel benchmarks score each attempt's implementation for correctness and speed but leave the reuse value of retained experience unmeasured. We introduce MetaKernelBench, which measures whether experience distilled from an attempt in one kernel domain-specific language (DSL) improves a fresh attempt at the same problem in another. Its 74 problems are fused subgraphs in six families, each posed as a pair of CuTe DSL and TIRx variants that differ only in the DSL. The agent first attempts each variant solo and is instructed to distill what it learns into a natural-language skill, which is transferred whether or not the source attempt passes verification. The skill is the only extra input to a skill-conditioned attempt by the same model in the other DSL. We compare each skill-conditioned attempt with the solo attempt on the same variant under matched per-attempt budgets, scoring correctness and end-to-end runtime. Across six models and both directions, paired lift over solo attempts ranges from -19% to +29%. Four models gain in both directions, yet regressions occur on 16% to 45% of problems in every model and direction. Outcomes follow the source attempt's result relative to the target's solo attempt rather than source success alone, improving in 71% of comparisons when the source stands above and regressing in 54% when it stands below. MetaKernelBench complements implementation-quality metrics by measuring same-problem cross-DSL kernel knowledge transfer.