Cohere Megakernel Beats vLLM by 1.58x on H100 GPUs
Cohere open-sourced a megakernel serving engine that runs entire LLM decode steps as a single CUDA kernel, boosting inference speeds on Nvidia H100 GPUs by up to 1.58x over vLLM.
Cohere has released cohere-megakernel under the Apache 2.0 license, an open-source serving engine designed to consolidate the entire decoding process of a large language model into one persistent CUDA kernel. By keeping active computations on the graphics hardware throughout execution, the system avoids the overhead of launching multiple small kernels per layer and paying synchronization penalties. Currently optimized for Cohere's 30-billion-parameter North Mini Code model, the engine achieves up to 1.58x faster throughput than vLLM on Nvidia H100 GPUs.
Running on a single H100 in BF16 format, the megakernel reaches 292 tokens per second at batch size 1, capturing 62 percent of the theoretical memory bandwidth Speed-of-Light limit, compared to vLLM's 185 tokens per second at 39 percent. On real-world benchmarks, the system delivers 1.25x to 1.41x end-to-end speedups. Specifically, on LiveCodeBench v6, it achieved 803 tokens per second compared to vLLM's 625 tokens per second, representing a 1.28x speedup, while maintaining accuracy with a 70.3 percent score. On SciCode, the megakernel scored 38.9 percent, slightly ahead of vLLM's 38.2 percent.
The release functions as an OpenAI-compatible server supporting streaming, tool calling, continuous batching, and a paged KV cache. However, it remains a narrow research release. It is currently restricted to H100 GPUs, BF16 precision, a maximum batch size of 8, and decode-only operations, with prefill tasks still running on standard PyTorch kernels. The task schedule is also hardcoded specifically for North Mini Code, which features 3.3 billion active parameters per token.
For AI practitioners, this release challenges the assumption that megakernels require complex compilers or exotic programming models. Cohere built the system using a shared 12-warp Application Binary Interface and global-memory counter barriers, demonstrating that developers can assemble megakernels by hand using existing, highly optimized GEMM and attention kernels. This approach allows teams serving Mixture-of-Experts models at low batch sizes to reclaim idle GPU memory bandwidth without rewriting their serving infrastructure from scratch.
This is our own summary of reporting by AlphaSignal
