Vibecoding GPU Kernels with LLMs

# vibecoding gpu kernels

Handwriting GPU kernels takes patience, effort, and also your sanity. I have been experimenting with modern LLMs (Claude Opus 5, GPT 5.6 Sol) to literally print kernels for me. I'm gonna talk about what I learnt doing that.

Honestly, writing GPU kernels is an embarrassingly verifiable task. 

First, you decide what operation(s) you want to write the kernel for. Second, you write the first version and make sure it compiles without any obvious errors. Third, you check correctness with a slow reference implementation. If it doesn't match, you try to fix the correctness issue. Once that is done, you benchmark the execution time of the kernel. The subsequent versions then build on top of this, and you keep optimizing until either you reach the roofline metrics or you are satisfied.

The image above shows it as a verifiable loop:

- The compile check is a tight local loop (B <-> C) before you even think of correctness.

- The correctness check (D <-> E <-> F) is the core verifiable-reward loop. This is the part that makes kernel writing a good problem for automated verification, since you have a ground-truth reference to check against.

- Optimization (G -> H -> back to D) reuses the same correctness loop for every new version, since a fast but wrong kernel is worthless, correctness has to be verified for each version.

- The roofline/satisfaction check (I) is the outer loop that decides whether to keep optimizing or stop.

## First version

The image above is still a higher-level view and the devil lies in the details. We need to make sure that our LLM agent has all the required context to even start writing a good first version.

The CUDA DSLs come into action here. Triton, CuTeDSL, and Tilelang are the ones which are very easy to get started with, in Python. The learning curve is less steeper compared to CUDA C++, however, the abstractions in those DSLs may confuse our agent even more. We need a way to pass context of those abstractions to the agent.

Moderns LLMs already know how to write "good" Triton. They can work nicely with Triton abstractions even without any context. However, for other DSLs like CuTeDSL (which provides much more control than Triton), I have found that having a context directory where the agent can look for understanding the DSL abstractions helps a lot.

> For example, cloning the NVIDIA cutlass repository in the context directory is a good way to let the agent look for abstractions related to Layout Algebra, Copy/GEMM atoms, memory hierarchy, example kernels, and so on while writing kernels in CuTeDSL.

In my experience, a good first version of the kernel compiles without any obvious errors and passes the correctness test which I'll talk about below.

## Test, Benchmark, and Profile

Given enough context to the agent, the real bottleneck now shifts to validation. The reference implementation itself, and validating against it becomes more and more important. I call this phase correctness testing or simply testing. The speed of the reference implementation does not matter as much as its intention. What you intend to measure and verify is what your agent will optimize for. 

Usually, when the computation is not supposed to take place in lower precision I measure the Max Absolute/Relative Error (MAE), Mean Squared Error (MSE/RMSE), and PSNR (Peak Signal to Noise Ratio). When lower precisions are involved, I tend to measure PSNR, and Cosine similarity (cossim).

The way you make the kernel versions actually run on a GPU depends on whether the GPU is available locally or via cloud. Regardless, our agent should have the ability to access its outputs in some way or the other. 

I find the below rung methodology a good way to have N number of testing functions:

which you can call like:

For the benchmark rung, you can do multiple things:

- Benchmark End-to-end kernel execution time for the overall time spent

- Use intra-kernel tracing to benchmark sections within a kernel and dump them in the output (using a custom tracer or CUPTI)

- Dump the generated IR, PTX, SASS, and CUBIN to a dumps directory and let the agent go through it

The last point can be expanded a bit more. At times, It might happen that the DSL may lower to generate suboptimal PTX (eventually SASS) and you find better instruction(s) or shape(s) that can be used instead. Our agent can read through the PTX/SASS text files and inline lower-level code instead of letting the DSL handle the suboptimal part. Again, passing the "searchable" PTX documentation as context is very helpful here.

The last thing to bring it all together is Profiling. If your agent can access the NCU (Nsight Compute Systems) CLI, you can ask it to profile and generate a report for your kernel as a part of the verifiable feedback loop above.

## Closing thoughts

"Is GPU kernel development dead then?" 
"Well yes but actually no"

Yes, because the difficult part of layouts, indexing, abstractions, and the overall structure can be largely solved by agents with enough context. You can easily reduce the work of 2-3 weeks to 1-2 days. No, because the real bottleneck has shifted from kernels to validations. There's no one way of doing things now: the better your context and harness, the faster the process. Specialized cases will benefit even more, and a good harness is all you will need.

Finally, instead of treating the agents as autonomous, treating them as really smart assistants which you can guide is still needed here. This is where your fundamental understanding of GPUs and kernels come handy. The human part (you) is still needed here. 

It's a bittersweet feeling, I know :)