Towards Automated Kernel Generation in the Era of LLMs

AI summary

Overview

Research area: Automated GPU/NPU kernel generation and optimization using large language models (LLMs) and LLM-based agents.

Technical level: Intermediate. The paper is a survey rather than a methods paper, so no prior background in kernel programming is strictly required, but familiarity with LLM training concepts (supervised fine-tuning, reinforcement learning) and basic GPU programming concepts (CUDA, Triton) helps.

Scope in one sentence: This survey consolidates the fragmented literature on LLM-driven kernel generation into a single structured reference, organizing post-training methods, agentic optimization workflows, training datasets, and evaluation benchmarks, and outlining open challenges.

What This Paper Is About

Modern AI systems are limited not just by hardware peak performance but by the quality of the GPU kernels that translate high-level algorithms into low-level hardware operations. Writing near-optimal kernels requires deep expertise in hardware architecture and programming models, and it does not scale because implementations are tightly coupled to specific hardware generations and vendors. The paper surveys how LLMs and LLM-based agents are being used to automate this kernel generation and optimization process, and it systematically organizes the datasets and benchmarks that support learning and evaluation in this area.

Key Contributions

1. A structured survey of LLM-based kernel generation methods, split into two post-training families — supervised fine-tuning (SFT) and reinforcement learning (RL) — covering named systems such as KernelLLM, ConCuR, Kevin, CUDA-L1/L2, SparseRL, CUDA Agent, AutoTriton, TritonRL, QiMeng-Kernel, Dr. Kernel, Kernel-Smith, and AscendKernelGen.

2. A taxonomy of agentic kernel generation organized along four structural dimensions: learning mechanisms, external memory management, hardware profiling integration, and multi-agent orchestration.

3. A consolidated resource infrastructure, including a structured overview of training corpora and kernel knowledge bases (Table 1) and a structured overview of evaluation benchmarks (Table 2), plus a literature collection tailored for retrieval-augmented generation (RAG).

4. An agenda of open challenges and future directions, spanning evaluation reliability and generalization, data scarcity and synthetic scaling, agentic training and harness engineering, scalable infrastructure, and human-AI collaboration. The authors also maintain an open-source GitHub repository at https://github.com/flagos-ai/awesome-LLM-driven-kernel-generation.

Main Findings

• Two dominant post-training families. The survey identifies supervised fine-tuning and reinforcement learning as the main ways LLMs are specialized for kernel generation. KernelLLM collects samples and uses the Triton compiler to produce aligned PyTorch–Triton examples; ConCuR curates kernel datasets with reasoning traces, producing KernelCoder, which achieves 17% on the fast_1 metric of KernelBench level 1.

• Industrial-scale SFT results. InCoder-32B introduces a three-stage data curation pipeline consisting of pre-training, mid-training, and post-training, achieving a fast_1 score of 22.2% on KernelBench Level 1.

• RL systems report large gains. CUDA-L1 introduces contrastive RL with an LLM-as-a-judge for dense feedback and is refined by CUDA-L2, which reports performance improvements over cuBLAS on its evaluated workloads. CUDA Agent introduces a large-scale agentic reinforcement learning system with a skill-augmented CUDA development environment, achieving state-of-the-art results on KernelBench and delivering a 99% faster rate over PyTorch Eager on KernelBench Level-1. Kernel-Smith generates Triton kernels that achieve a 70% fast_1 score on KernelBench level 1.

• Agents turn one-pass generation into closed loops. Relying on foundation LLMs alone reduces development to static one-pass inference. Agent-based approaches add planning, tool use, and evaluation of intermediate results, enabling exploration across workloads and hardware. The survey groups them into learning mechanisms (e.g., Caesar in KernelBench, Inference-Time Scaling, PEAK, AutoKernel, MaxCode, K-Search), external memory (KernelEvolve, ReGraphT, KernelBlaster, EvoKernel, KernelSkill), hardware profiling integration (QiMeng-TensorOp, QiMeng-GEMM, QiMeng-Attention, SwizzlePerf, CUDA-LLM, TritonForge, PRAGMA, KernelBand), and multi-agent orchestration (STARK, AKG, Astra, CudaForge, KForge, KernelFalcon, GEAK).

• Population-based evolution to escape local optima. Frameworks such as Lange et al., FM Agent, EvoEngineer, GPU Kernel Scientist, and cuPilot use mutation, crossover, diversity preservation, and multi-population dynamics rather than single-trajectory refinement.

• Data resources are split into corpora and knowledge bases. Training corpora include structured datasets (the Stack v2, HPC-Instruct, KernelBook, KernelBench samples) and code-centric repositories organized into three layers: high-performance operator libraries (e.g., CUTLASS, FlashAttention, FlagAttention, AoTriton, xFormers, Liger-Kernel, FlagGems, Bitsandbytes, Gemlite, FlashInfer, FBGEMM, Transformer Engine, DeepGEMM, Tile Kernels), framework and system integration (PyTorch/ATen, vLLM, SGLang, llama.cpp, TensorRT-LLM, DeepSpeed), and domain-specific languages (Triton, TileLang, cuTile). Knowledge bases supply documentation, guides, and community indices such as the CUDA Guide, PTX ISA, tuning guides, Nsight Compute, GPU-MODE, Triton Index, Awesome-CUDA, Awesome-GPU, LeetCUDA, and Triton-Puzzles.

• Benchmarks have grown in three directions. Metrics expanded from correctness and runtime speedup (ParEval) to efficiency, robustness, and composite measures such as fast_p; hardware coverage extended beyond NVIDIA GPUs to AMD GPUs, Huawei NPUs, and Google TPUs (e.g., MultiKernelBench); and workloads shifted toward production-grade kernels (FlashInfer-Bench, SOL-ExecBench, BackendBench). Concrete benchmarks include ParEval (420 expert-selected tasks across 12 algorithmic domains), KernelBench (250 PyTorch-to-CUDA tasks), TritonBench (184 high-level kernels in TritonBench-G and 166 fusion tasks in TritonBench-T), MultiKernel-Bench (285 tasks across 14 operator categories), a ROCm benchmark with 30 expert-verified kernels, Robust-kbench (9 deep learning task categories), CUDAEval (313 curated tasks from the Stack v2), FlashInfer-Bench (eight representative kernel types used in LLM inference), and SOL-ExecBench.

• The fast_p metric. The survey highlights fast_p, which reports the proportion of generated kernels that are both correct and achieve a speedup greater than p, combining correctness and performance into one composite number.

• The field remains nascent. The authors state that LLM-driven kernel generation is still an early-stage research area, with proof-of-concept systems not yet converted into robust, scalable deployment.

Methodology in Plain English

The authors did not run new experiments; this is a survey. Their approach was to read across a fast-growing but scattered literature and organize it. They first laid out background on LLMs, agents, and kernel programming paradigms. They then grouped existing methods into two broad buckets: approaches that specialize an LLM through supervised fine-tuning or reinforcement learning, and agent-based approaches that put the model inside an iterative loop with execution feedback. For agents, they proposed four dimensions to classify work: how the agent learns, how it stores external knowledge, how it uses hardware profiling data, and how multiple agents split responsibilities. Finally, they catalogued the data and benchmarks the field relies on into structured tables, and distilled the unsolved problems into a set of open challenges and research directions.

Why This Matters

Impact on research. The paper gives the field a common vocabulary and a shared map of methods, datasets, and benchmarks, which the authors argue is currently missing. It also flags evaluation reliability as a first-order problem, noting vulnerability to reward hacking, where kernels score well on benchmarks without delivering benefits in real deployments.

Real-world applications.

• Faster and cheaper large-scale LLM training and inference, since kernels implementing operations such as matrix multiplication and attention dominate execution time.
• Production inference serving systems, where benchmarks such as FlashInfer-Bench target kernels used in LLM inference and multi-agent systems like Astra target production-grade SGLang kernels.
• Cross-vendor hardware portability, spanning NVIDIA GPUs, AMD GPUs, Huawei NPUs, and Google TPUs, which the datasets and multi-platform benchmarks such as MultiKernelBench and MultiKernelBench's 14 operator categories are designed to measure.
• Specialized workloads such as diffusion model acceleration (DiffAgent) and sparse matrix operations (SparseRL), plus quantized and low-precision kernels (Bitsandbytes, Gemlite, DeepGEMM).

Industry relevance. Kernel engineering is described as critical but time-consuming and non-scalable, with implementations tightly coupled to particular hardware architectures and vendors. Reducing that burden matters directly to the cost and efficiency of AI infrastructure. The paper also notes that general-purpose coding agents such as Claude Code and OpenCode may allow organizations to specialize existing agents through harness engineering rather than building task-specific agents from scratch, which lowers the barrier to adoption.

Future Directions

• More reliable evaluation and generalization. Build evaluation frameworks robust to reward hacking, broad across workloads and platforms, and capable of measuring system-level impact rather than isolated kernel scores. The authors note that kernel-level improvements' effect on end-to-end AI systems remains insufficiently understood.

• Solving data scarcity through synthetic scaling. Construct large-scale kernel datasets, generate synthetic data, and collect execution-driven optimization traces. High-performance kernels are sparsely represented in existing corpora, and those corpora mostly contain final implementations while omitting optimization trajectories and hardware-aware expertise.

• Long-horizon agentic training and harness engineering. Train models explicitly for iterative cycles of generation, execution, profiling, and refinement, since today's foundation models are not trained for such trajectories and existing systems rely on handcrafted workflows that struggle with exploration efficiency, context management, and long-term credit assignment.

• Scalable synthesis and training infrastructure. Develop distributed, securely isolated sandboxes, address latency mismatches between agent rollout and kernel compilation and verification, and design fault-tolerant multi-device services.

• Human-AI collaboration. Establish effective bidirectional feedback loops: human-in-the-loop guidance to steer optimization with high-level objectives and constraints, and human-from-the-loop learning to transfer agent-discovered knowledge back to developers.

Target Audience

This survey is most useful to researchers and engineers entering or already working in automated kernel generation, LLM-based code generation, compilers, and performance engineering. It is also valuable to practitioners at AI infrastructure and hardware companies who need a single reference for which methods, datasets, and benchmarks exist before choosing an approach, and to graduate students looking for open problems — particularly in evaluation reliability, data curation, long-horizon agent training, and scalable sandbox infrastructure. Because the paper includes a RAG-oriented literature collection and a GitHub repository, it is additionally suited to people building retrieval or tooling around this topic.

Authors’ abstract

The performance of modern AI systems is fundamentally constrained by the quality of their underlying GPU kernels, which translate high-level algorithmic semantics into low-level hardware operations. Achieving near-optimal kernels requires expert-level understanding of hardware architectures and programming models, making kernel engineering a critical but notoriously time-consuming and non-scalable process. Recent advances in large language models and LLM-based agents have opened new possibilities for automating kernel generation and optimization. LLMs are well-suited to compress expert-level kernel knowledge that is difficult to formalize, while agentic systems further enable scalable optimization by casting kernel development as an iterative, feedback-driven loop. Rapid progress has been made in this area. However, the field remains fragmented and lacks a systematic perspective for LLM-driven kernel generation. This survey addresses this gap by providing a structured overview of existing approaches, spanning LLM-based approaches and agentic optimization workflows, and systematically organizing the datasets and benchmarks that underpin learning and evaluation in this domain. Moreover, key open challenges and future research directions are further outlined, aiming to establish a comprehensive reference for the next generation of automated kernel optimization. To keep track of this field, we maintain an open-source GitHub repository at https://github.com/flagos-ai/awesome-LLM-driven-kernel-generation.