Kernel Forge: LLM agents that optimize CUDA kernels in PyTorch models

Most of a machine learning model's runtime is concentrated in a handful of compute kernels: matrix multiplication, convolution, normalization. Optimizing those kernels is one of the most direct ways to cut inference latency and cost, but it has traditionally required expert engineers hand-writing low-level GPU code, a scarce and expensive profile to hire. Kernel Forge, presented today, July 30, on arXiv as arXiv:2607.24762, aims to close that gap with an open source, end-to-end agentic harness that accepts any unmodified PyTorch model and returns that same model with its kernels already optimized and integrated.

Agentic systems that generate kernels are not new, but the authors point to four recurring gaps in existing tools. They are evaluated on randomly generated tensors and isolated kernels rather than real models. They emit standalone CUDA code the developer must reintegrate into their project by hand. They focus almost exclusively on PyTorch LLM models, leaving out other workloads. And they offer little support for inspecting and debugging results. Kernel Forge presents itself as a direct answer to that list.

Tree search instead of linear refinement

The most striking technical decision is the use of Monte Carlo Tree Search (MCTS) to explore several optimization paths in parallel, instead of a single linear chain of refinements. The difference matters because optimizing a kernel is a search problem full of dead ends: a promising transformation can block later improvements, and an agent that only iterates on its last attempt tends to get stuck in local optima. With MCTS, the system can back up, open alternative branches and compare complete paths before committing to one. It is the same logic that has worked in other search domains with enormous spaces, applied here to a problem where every candidate can be compiled and measured.

Complete models, not lab kernels

The second differentiator is scope. Kernel Forge works on the PyTorch model as it stands, with no prior adaptation, and covers vision, diffusion and LLM workloads, not just language models. The result is not a loose CUDA file someone has to stitch back into the project, but the original model with its kernels replaced in place.

The system also ships a graphical interface for monitoring the optimization's progress and inspecting candidate kernels, something the authors find missing in previous tools. For a workflow where the agent generates dozens of variants, being able to see what it is trying and why it discards paths is the difference between an auditable process and a black box.

Who it matters for and what remains to be checked

The obvious beneficiaries are teams serving inference at scale that pay for every millisecond, and ML teams without a CUDA specialist on staff. It is also a relevant piece for anyone following the evolution of agentic harnesses: much of the value is not in the model writing the code, but in the search structure, the evaluation on real workloads and the integration of the result into the project.

The open questions are the usual ones in this domain. The abstract does not detail speedup figures or how the numerical equivalence of the generated kernels is verified, and that verification is the critical point: a kernel that is 20 percent faster but degrades the model's accuracy is not an optimization, it is an expensive bug. The full evaluation will be worth reading and, better yet, the code worth trying, since the authors release it as open source.

From ElephantPink, what interests us most about Kernel Forge is not the CUDA generation itself but the harness design: search with backtracking, evaluation on complete models and output integrated into the project. It is the direction serious agentic systems are converging on, and applying it to a domain as measurable as kernels will help separate what works from what merely looks like it does.