KernelBench: Can LLMs Write Efficient GPU Kernels?

Updated Paper: https://storage.googleapis.com/anonymous-files/kernelbench.pdf

We thank you for appreciating KernelBench design and suggesting further improvements. As you noted, automatic GPU code generation is an underexplored area with many interesting research questions; KernelBench facilitates research in this direction as the first benchmark and environment for kernel development, with “simple-yet-effective” task definition and “well-designed evaluation”. In fact, we have already seen enthusiasm from the community, with multiple projects tackling KernelBench through agentic optimization and post-training.

Providing Model with Hardware Information
We totally agree that specifying hardware information is important as GPU kernels are inherently platform-dependent. In fact, this was already studied in our original submission. In Section 5.2.2, we provided the model with the exact kind of GPU hardware specifications (see Appendix C.5) that the reviewer described, and found that current models rarely conduct optimization correctly for underlying hardware when provided with such information.

Clarifying Design Choice to Test Kernel Fusion
In KernelBench the model has full flexibility to decide what subset of operators in the PyTorch reference to optimize and fuse. We believe this is one of the crucial abilities when a model is given distinct or new architectures in the real-world setting.

KernelBench’s 3-level categorization helps disentangle fusion decisions and kernel generation. Level 1 problems (single operators) only test the model's ability to write optimized kernels; Level 2 and 3 problems are designed to additionally evaluate the model's ability to identify and leverage fusion opportunities; Appendix K provides a detailed task breakdown.

Fusion Patterns in Model Generated Code
To answer your questions about fusion patterns in generated kernels, we manually inspected the kernels generated by the best performing model, DeepSeek R1, and provided new analysis in Appendix L. We focus on level 2 problems, which are composed with one mainloop (e.g. conv, matmul) and 2-5 epilogue operations (non-linearities, reductions, etc). We observe model-generated code always attempts to generate 1-2 fused kernels per problem. As shown in Figure 19, the fused kernels tend to contain more than half the operators in the program. To explicitly answer your question, only 18% of programs fuse all operators into a single kernel.

Regarding your question on the quality of fusion decisions and whether they cause low performance, we analyzed the generated kernels that were slower than PyTorch Eager (as shown in Table 17) and draw two observations: 1) main loop operators (eg. Conv) were not fused with epilogue operators 2) the model’s attempt to fuse main loop operators (e.g. GEMM + other ops) was not faster than launching highly-optimized cuBLAS kernels. Also refer to “Analysis of Performance Degradation Cases” in response to Reviewer ufH2 for a related study.

Comparison with SOTA Compiler-Based Approach
Thank you for bringing up relevant compiler-based approaches (AStitch, Welder, ROLLER, TVM), which we have added and elaborate on in our updated related works.

To directly address your concern, we compare fusion decisions in model-generated kernels with auto-fusion compilers. Since AStitch, Welder, ROLLER could not be run on KernelBench due to format incompatibilities or outdated support for KernelBench PyTorch 2.5 /CUDA 12.4 (Appendix B), we focus comparison on widely-adopted torch.compile with SoTA performance (better than TVM, see Table 3 in PyTorch 2 $1$ , ASPLOS ‘24) and employs an auto-fusion policy over TorchInductor’s define-by-run IR. We show both fusion decisions of R1 and torch.compile in Table 17. Torch.compile often creates sophisticated fusion patterns by breaking Convolutions or GroupNorm into smaller multi-pass kernels that compute partial results and statistics in parallel — behavior that R1-kernels rarely exhibit.

DAG Representation
Per your suggestion, we conduct experiments on using a DAG representation (ONNX, torch.fx graph) “instead” of a PyTorch Reference, which might help highlight fusion opportunities. We explored this in Appendix M.2 and found that DAG representations cause output mismatch issues on problems that succeed with PyTorch representations – see response to reviewer ufH2 for details.

We appreciate the comprehensive comments and hope that our additional experiments, analysis of results, and discussion addressed your concerns. We hope you find the paper significantly improved and consider reflecting this in your final score.

$1$ PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. ASPLOS '24. https://doi.org/10.1145/3620665.3640366

Updated Paper: https://storage.googleapis.com/anonymous-files/kernelbench.pdf

We sincerely thank you for the detailed and insightful review! We are truly encouraged by the positive feedback, particularly that the work is seen as "well-supported by strong evidence" and a "comprehensive benchmark." We especially appreciate the recognition of our "novel metric, fast_p," as offering a "more holistic and nuanced assessment," and that you found value in our "wide range of experiments across multiple dimensions." We were glad to read that you felt our work "provides valuable insights into $LLMs'$ capabilities in generating CUDA kernels", which are valuable for AI and HPC communities as we explore automating kernel optimization. These strengths highlight core goals we aimed for, and we're glad they resonated.

Analysis of Performance Degradation Cases
Per your suggestion, in addition to our existing error analysis (4.2) and case study of the fastest kernels (Appendix D), we added a new “Case study: Performance Degradation” (Appendix N), which specifically examines instances where generated kernels underperformed compared to the baseline. Here are our findings:

1. LLM implementations of core ops (matmul, conv) underperform highly-optimized proprietary (e.g. cuDNN) kernels in PyTorch.
2. LLM correctly identifies fusion patterns, but fused operations (often matmuls) are not efficiently implemented, outweighing benefits from reduced memory access
3. LLM blocks better PyTorch native fusion capabilities by generating a custom kernel for a minor task that prevents optimizing across a larger sequence of operations.

Alternative Input Specification
Per your suggestion, we explored in Appendix M using 1) Natural Language 2) DAGs of program operators as input specification. On a representative Level 2 problem that model succeeded with PyTorch representations, it failed with compilation and logical issues on natural language representation due to ambiguity on exact behaviors, even when provided with verbose dimension details. DAG representations capture the program execution much better and hint the model to conduct fusions, but going from DAG to kernel directly can lead to logical errors resulting in output mismatch.

Using Libraries & DSLs
Reviewer raises the point of whether generating code using frameworks like Triton/CUTLASS would be helpful. To address your feedback, we extended KernelBench with a Triton task specification and evaluation backend. As shown in Appendix O Table 20, we found models perform worse when using Triton, both in terms of correctness and performance: fast_1 for DeepSeek R1 drops from 12%, 36%, 2% to 6%, 13%, 2% across 3 levels respectively. Qualitatively, we found models generate many Triton-related errors, likely due to Triton being a rarer source of training data than CUDA, highlighting potential challenges for using domain-specific libraries.
We reiterate KernelBench’s goal is to propose a new benchmark with thorough baseline evaluation as a first step, rather than solving kernel generations.

Level-Specific Prompting and Scoring

We made a deliberate choice to not explicitly weight fast_p by task complexity (e.g., Level 1 vs. 3). We report the levels separately, and harder tasks are expected to yield lower scores naturally, reflecting greater challenges. This approach is common in other coding benchmarks (LiveCodeBench) that include easy/medium/hard problems without score normalization by difficulty. Regarding level-specific prompting, our baseline evaluations intentionally used general prompts, rather than the suggested task-specific ones, to evaluate each model's fundamental ability to independently discover and select optimization strategies across task complexities without explicit steering.

Architecture-Specific Optimizations
Based on your suggestions, we added experiments for eliciting architecture-specific optimizations: Tensor Cores and asynchronous memory transfers (Appendix G.3) on Ampere GPUs, in addition to existing experiments in Section 5.2. We provided DeepSeek-R1 with examples using wmma and memcpy_async instructions, on simple KernelBench matrix multiply problems in FP16 (compatible with Tensor Cores). We observed that the model attempted to apply yet struggled to utilize those advanced instructions. Among the 5/17 correct kernels that use WMMAs, successfully leveraging Tensor Cores did not lead to better performance over PyTorch. No kernels used memcpy_async correctly. This highlights that successfully utilizing hardware features remains challenging for models and KernelBench provides a playground for the community to develop methods that address this limitation.

Once again, we thank you for your valuable time and feedback. We hope that our additional experiments, analysis of results, and discussion addressed your concerns. We hope you find the paper significantly improved and consider reflecting this in your final score.

Updated Paper: https://storage.googleapis.com/anonymous-files/kernelbench.pdf

We thank reviewer AoMw for your review! We are glad the reviewer appreciates our "fast_p evaluation metric for KernelBench" and finds that our "claims are supported by clear and convincing evidence." Below, we address your comments regarding the evaluation metrics, platform dependency, and comparison to other benchmarks.

Clarity of fast_p vs. Separate Correctness and Speedup Metrics

“Adding two separate metrics for correctness and speedup” is a great suggestion for added clarity. In our revision (see link above), we included an extended table presenting correctness (equivalent to fast_{p=0}) and geo-mean speedup as separate metrics (Appendix I, Table 16), providing a disaggregated view for these specific aspects. The geometric mean of speedups only includes the correct generations, as fast but incorrect code is not helpful.

Thanks for acknowledging that the “evaluation metric fast_p makes sense for KernelBench!” We also reiterate that for kernel generation, speedup and correctness are tightly coupled, motivating our choice to design fast_p. To provide more context on this, we've also added a section discussing our metric design explorations (Appendix I). We hope the combination of the original fast_p and these new separate metrics offers more clarity.

Platform Dependency and Reproducibility

We agree entirely about the platform-specific nature of hardware performance tuning. Thus, it is very important to compare results when controlling for both the input program and the underlying hardware. For instance, our evaluations (as noted in paper Section 4.4 and Appendix G.1) across several hardware platforms, including L40S, A100, and H100 GPUs, revealed reasonable consistency in the kernel generations at Level 1 but more pronounced variation in Level 2. Other than this hardware evaluation study, most of our experiments in the paper are done on an Nvidia L40S, and we expect the results to be reproducible on this type of GPU.

Comparison to Existing Programming Benchmarks

To address your comment, we've added a new experimental section (see Appendix J) where we compare model performance on KernelBench (KB) with LiveCodeBench (LCB). (We chose LCB as HumanEval performance is quite saturated for current models). We present the relative rankings of models across these benchmarks.

Our results show, perhaps unsurprisingly, that models performing well on general coding benchmarks tend to also perform better on KernelBench, but variability in rankings (e.g. o1 ranks 1st in LCB and 2nd in KB Level 1, and R1 ranks 2nd in LCB but 1st in KB Level1) across different levels of KernelBench suggests that additional skills are required for high performance in kernel-specific tasks – intuitively, this aligns with the major differences between GPU programming and standard programming problems found in popular coding benchmarks. We would also like to highlight that KernelBench is not merely another code generation benchmark; it adds the critical dimensions of performance optimization and hardware awareness, testing a model's ability to generate not only correct code, but also efficient code, which presents distinct challenges.

Relation to Broader Literature of Leveraging Feedback

We definitely see the connection with existing works on leveraging feedback too (we also added citations of works listed by you here)! We believe KernelBench takes this concept into a particularly challenging and impactful domain. Optimizing hardware kernels (notoriously difficult even for human experts) offers tangible real-world benefits (cost and energy savings for AI!), making it a high-stakes environment given the ubiquity and importance of AI systems today.

We intentionally designed KernelBench to facilitate precisely iterative, feedback-driven improvement. By providing rich, actionable feedback signals—clear correctness checks (pass/fail), compilation status, precise runtime measurements, and speedup relative to a baseline—KernelBench creates an environment where AI systems can directly learn from their attempts and refine their solutions. The goal of our baseline results using feedback is to thoroughly characterize the degree to which we can solve KernelBench. We find that despite using these techniques, the best model gets fast_p=1 of only 18% on level 3, showing there’s a lot more progress to be made on this benchmark. In this sense, we see KernelBench as a stepping stone for pushing forward research in automated kernel engineering, providing a crucial contribution to the community as a standard evaluation environment.

In light of these clarifications as well as new experimental results and modifications to the paper to address your comments, we would really appreciate it if you would re-examine our paper and consider raising your score.