Cohere speeds up North Mini Code with new megakernel

Cohere has introduced a highly optimized serving engine for its North Mini Code model, a 30-billion parameter model with 3.3 billion active parameters per token. Built around a decode megakernel running in BF16 precision on a single NVIDIA H100 GPU, the engine achieves end-to-end speeds that are 1.25 to 1.41 times faster than the popular vLLM framework. At a batch size of one, the system reaches 292 tokens per second, representing 62 percent of the hardware's theoretical memory bandwidth limit, compared to vLLM's 185 tokens per second.

The performance leap addresses the memory-bound nature of autoregressive decoding. For North Mini Code, each decode step requires streaming 6.6 gigabytes of weights and roughly 0.5 gigabytes of key-value cache at an 8K context window. While an H100 GPU provides 3.35 terabytes per second of bandwidth—yielding a theoretical limit of 470 tokens per second—traditional engines like vLLM utilize only 39 percent of this capacity. Cohere's megakernel consolidates the entire forward pass into a single persistent kernel, eliminating the launch and synchronization overhead of running dozens of small, sequential kernels.

Unlike experimental compilers, Cohere's engine is a fully fledged serving system. It supports continuous batching, paged attention, and ragged sequence lengths behind an OpenAI-compatible endpoint with tool calling. The architecture relies on a single CUDA file that restructures ordinary tiled matrix multiplications and paged attention into a unified calling convention. Each of the H100's 132 streaming multiprocessors runs a single threadblock of 12 warps, executing tasks from a host-built static schedule while using global memory counters for synchronization.

For AI practitioners, this development proves that megakernels can transition from academic research into production-ready infrastructure. The engine maintains its performance advantages across various batch sizes and up to a 256K context length without any measurable loss in model accuracy. Furthermore, Cohere's simplified design demonstrates that developers can write and integrate these high-performance kernels manually without relying on complex, bug-prone compilation frameworks.