EdgeKernelBench: Unified LLM-Driven Kernel Generation for Edge Backends

AI summary

Overview

Research area: Machine learning systems — LLM-driven code generation for high-performance kernels, compiler infrastructure, and edge deployment.

Technical level: Intermediate (the core ideas are accessible; the evaluation protocol, IR terminology, and agent architecture assume some familiarity with tensor compilers and kernel optimization).

Scope: This paper introduces EdgeKernelBench, a benchmark plus unified intermediate-representation (IR) framework for LLM-generated kernels across heterogeneous edge backends, and LEKA, a layered agent that optimizes kernels using device feedback.

What This Paper Is About

Most work on LLM-generated GPU kernels targets server accelerators with uniform software stacks and generous resources. Edge deployment is different: hardware and runtimes are fragmented, and memory as well as speed determines whether a model is deployable. The authors build a benchmark that evaluates LLM kernel generation across three edge backends (CPU/LLVM, GPU/CUDA, GPU/OpenCL) with tasks ranging from single operators to compact model graphs, and show that one-shot generation produces functionally correct code far more often than it produces fast code. They then propose LEKA, an agent that splits optimization across IR levels so each agent handles one representation layer, closing much of that gap.

Key Contributions

1. EdgeKernelBench — a benchmark combining a unified IR deployment framework with a hierarchical task set: 331 canonical tasks spanning three complexity levels (128 Basic Compute Patterns, 162 Composed Operator Graphs, 41 Compact Model Graphs) and three edge application domains (Mobile and Personal Intelligence, Embodied Robotics, Smart AIoT).
2. A unified IR generation interface — instead of emitting native code per backend, the LLM expresses computation, scheduling, and memory decisions in a shared IR that a deterministic backend adapter lowers to each target, so optimization knowledge is reusable across devices.
3. LEKA (Layered Edge Kernel Agent) — an agent architecture that assigns one agent per IR level (L0, L1, L2 plus a hardware-sensing agent), linked through a shared candidate tree and guided by on-device profiling reports.
4. Systematic evaluation — five LLMs evaluated on 650 task instances across three real edge devices, including comparisons against production inference engines (MNN 3.3.0, TensorRT 10.3.0) and a pooled failure analysis.

Main Findings

• Correctness does not translate into speed. With a single generation attempt, the best model reaches 73.5% Compile@1 and 64.5% Correct@1, but only 25.5% Fast@1.0 — a large gap between writing working code and writing fast code.
• Performance collapses as program scope grows. Across backends, the highest Fast@1.0 falls from 51.0% at Level 1 to 16.2% at Level 2 and 4.0% at Level 3. At Level 3, the best Fast@1.0 on the three backends lies between 0.0% and 4.0%.
• CPU is easier than GPU for correctness. At Level 1, the highest Correct@1 is 94.5% on CPU, versus 79.6% on CUDA and 73.5% on OpenCL. CUDA generally yields higher Fast@1.0 than OpenCL.
• Model strengths differ by dimension. Gemini-3.7-Flash achieves the highest Level 3 Correct@1 on all three backends. DeepSeek-V4-Pro reaches 75.3% Correct@1 at Level 2 on CPU but only 3.2% Fast@1.0, compared with 15.1% for GPT-5.6-sol — high correctness does not imply strong optimization.
• LEKA substantially improves results on the 69-instance agent subset. Relative to one-shot generation with GLM-5.3-Flash, LEKA raises Compile@1 from 60.9% to 100.0%, Correct@1 from 42.0% to 100.0%, and Fast@1.0 over TVM from 13.0% to 43.5%.
• LEKA beats a budget-matched general coding agent. Against a pure Claude Code baseline using the same base model, task contracts, evaluator, and time budgets, LEKA raises Fast@1.0 from 26.1% to 43.5%, with an overall geometric-mean speedup of 3.8x relative to Claude Code.
• Layered gains are largest at medium complexity. Pooled geometric mean speedup rises from 3.4x at Level 1 to 4.7x at Level 2, then declines to 2.9x at Level 3, which the authors attribute to the base model's limits on larger graphs.
• Generated programs can beat production engines, but mostly on simple tasks. Against MNN, Fast@1 peaks at 50.7% on CPU and 31.6% on GPU, with maxima of 14.9% at Level 2 and 5.3% at Level 3. Against TensorRT, the highest Fast@1 is 43.2% at Level 1 and 5.3% at Level 2, no Level 3 candidate wins, and at a speedup threshold of 2 only 7.4% of tasks remain.
• Failure modes are dominated by memory and API issues. Pooled across all models and backends, memory errors account for 34.0% and syntax/API errors 25.4% of failures, followed by correctness errors at 18.9% and parallel errors at 12.0%. Most memory errors concern allocation legality rather than insufficient capacity.
• Case study shows cumulative cross-layer gains. On SqueezeNet1.1 on RK3588, the Relax agent adopts NHWC layout and rewrites 1x1 convolutions as matrix multiplications (1.3x), the TIR agent adds shared-memory tiling (1.7x), and the OpenCL agent applies register blocking (3.5x over the initial implementation).

Methodology in Plain English

The authors start from standard PyTorch reference programs, normalized into static, self-contained form with traceable provenance, and organize them along two independent axes: how complex the program is (single operator, fused operator graph, compact model graph) and which edge application it comes from. Each task in the evaluation is paired with a specific backend and domain: CPU/LLVM uses Mobile and Personal Intelligence tasks, GPU/CUDA uses Embodied Robotics, and GPU/OpenCL uses Smart AIoT, giving 650 instances in total across the three devices.

Rather than asking the model to write backend-native code, the framework asks it to express graph-level, scheduling, and memory decisions in a shared IR. A deterministic compiler adapter then lowers that IR to the target — LLVM IR for CPU, CUDA C for the Jetson GPU, OpenCL C for the RK3588 GPU — using TVM 0.23.0 as the reference implementation. Every candidate program is compiled, checked against the PyTorch reference across three random seeds, and then timed in a paired protocol against a baseline (TVM tuned with MetaSchedule, or MNN / TensorRT on the subsets those engines support), with a stability criterion on repeated measurements.

For LEKA, the optimization loop is split by IR level. Each layer agent edits only within its own contract and scope, compiler conversions carry earlier optimizations forward, and a shared candidate tree records every attempt and its measurements so branches can be compared or rolled back. A hardware-sensing agent profiles candidates on the target device and reports bottlenecks back to the level that owns the relevant decision, so feedback is attributed to a specific layer rather than to the whole program.

Why This Matters

Impact on research. The paper reframes LLM kernel generation as a problem of representation choice, not just code emission. By showing that one-shot correct code rarely becomes fast code (64.5% Correct@1 versus 25.5% Fast@1.0), and that splitting decisions by IR level recovers much of the gap, it gives the field a measurable baseline and a concrete architectural alternative. The hierarchical benchmark also makes scope-dependent degradation visible instead of averaging it away.

Real-world applications:

• Deploying updated models on phones and IoT devices where vendor runtimes differ and memory is scarce.
• Bringing up new or rarely used accelerators without hand-writing a kernel library per operator and backend.
• Optimizing compact vision and transformer graphs (MobileNetV2/V3, ResNet-18, ConvNeXt, NanoDet, U-Net, PointPillars backbones, Llama/Qwen2/Mistral/Phi-3 decoder blocks) for robotics and smart-camera products.
• Reducing the manual engineering cost of implementing, validating, and maintaining custom operators in edge inference stacks.

Industry relevance. Hardware vendors and inference-engine teams face exactly the fragmentation described: each new chip or runtime needs its own kernels. A shared IR plus adapter design lets a single generation workflow cover multiple targets, and the MNN/TensorRT comparisons give practitioners a realistic sense of when generated kernels are competitive — mainly on simple operators, rarely on multi-operator graphs.

Future Directions

• Pushing optimization responsibility to lower IR levels. The authors explicitly ask whether, as models become more capable, the boundary between LLM decisions and compiler decisions can shift toward lower-level representations — and at what cost in constraint complexity.
• Better layer boundaries and IR design. How to define the split between levels, and how to design IRs that help agents reason about optimization choices and constraints, remains open.
• Closing the Level 3 gap. Geometric-mean speedup declines to 2.9x at the compact-model-graph level, which the authors attribute to base-model limits on long dependency chains; stronger models or better cross-level coordination are natural next steps.
• Extending beyond the current backends. The appendix lists extensions to NPU targets and heterogeneous execution, plus tighter compiler integration and improved coordination between IR layers and agents, as future work.

Target Audience

Researchers and engineers working on LLM-based code generation, tensor compilers, and ML systems; performance engineers who deploy models to edge devices and want to know where generated kernels are competitive with hand-tuned or engine-provided implementations; and benchmark designers interested in evaluation protocols that separate compilation, correctness, and paired speedup across heterogeneous hardware. Readers without systems background can follow the problem framing and headline results, but the LEKA design and evaluation protocol are best appreciated with some familiarity with TVM-style IR levels.

Authors’ abstract

Recent works show that LLMs can generate kernels with strong performance on server GPUs. Edge deployment presents distinct challenges because the hardware is heterogeneous, the software stacks are fragmented, and the resource budgets are tight. LLM kernel generation across diverse edge backends has yet to be studied systematically. We introduce EdgeKernelBench, which combines a unified IR deployment framework with a hierarchical benchmark for edge workloads. The unified IR captures reusable computation, scheduling, and memory decisions and connects to each target backend through a compiler interface. The benchmark spans progressively larger program scopes, from individual operators to compact model graphs, across representative edge application domains. We instantiate the design with TVM as a reference compiler and validate it across heterogeneous edge backends. With a single generation attempt, the best model achieves 64.5% Correct@1 but only 25.5% Fast@1.0. This gap shows that functional correctness does not consistently translate into performance gains. We therefore introduce the Layered Edge Kernel Agent (LEKA), which organizes optimization through a layered IR. On the agent evaluation subset, LEKA achieves 43.5% Fast@1.0, compared with 13.0% for direct generation and 26.1% for Claude Code. LEKA and Claude Code use the same base model and optimization budget. Together, EdgeKernelBench and LEKA support unified evaluation and layered optimization of kernels generated by LLMs across edge backends.