AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis

Problem
The paper addresses the challenge of efficiently compiling large language models into CUDA kernels without manual intervention, a gap in the literature regarding automated kernel synthesis. The authors highlight the need for a system that guarantees safety properties such as deadlock-freedom and race-freedom, which is particularly relevant given the increasing complexity of model architectures. This work is presented as a preprint, indicating it has not yet undergone peer review.

Method
AutoMegaKernel (AMK) compiles Llama-family models into a single persistent cooperative CUDA kernel, executing the entire forward pass in one launch. The system employs a frozen schedule-IR validator that performs static checks to certify the absence of deadlocks and race conditions, rejecting unsafe schedules before execution. The authors tested AMK against 7,160 adversarial schedules, with 6,091 deemed unsafe, achieving zero false-accepts while successfully accepting all 360 real lowerings. The framework supports retargeting across multiple architectures (sm_80, sm_90, sm_120) from a single codebase and auto-generates correct megakernels for all ten supported models. The system includes an unattended autoresearch loop that iteratively improves the megakernel’s performance, achieving a performance increase of 1.25 to 1.72 times over its baseline.

Results
AMK’s int8 (W8A16) megakernel outperforms CUDA-graphed cuBLAS in batch-1 decoding across various NVIDIA datacenter inference hardware. Specifically, it achieves up to 1.33x speedup on the L4, 1.25-1.27x on the L40S, and 1.08x on the A10G, while maintaining a 1.19-1.23x advantage on the consumer-grade RTX 5090. Notably, the performance does not correlate cleanly with bandwidth, as the L40S (864 GB/s) outperforms the A10G (600 GB/s). However, AMK does not match cuBLAS performance on high-bandwidth training-class GPUs like the A100/H100, where it encounters a cross-SM synchronization bottleneck.

Limitations
The authors acknowledge that while AMK successfully certifies safety properties, it does not provide a mechanized proof of correctness. Additionally, the performance gap on high-bandwidth training-class GPUs suggests that further optimization is needed for those scenarios. The paper does not explore the implications of using different precision formats beyond the W8A16 and bf16 comparisons.

Why it matters
The development of AutoMegaKernel has significant implications for the deployment of large language models in production environments, particularly in optimizing inference performance while ensuring safety. The ability to automate kernel synthesis could streamline workflows in machine learning engineering, making it easier to leverage advanced models without extensive manual coding. This work contributes to the ongoing discourse on efficient model deployment strategies, as discussed in related literature, and is available on arXiv.