Kog targets 30x faster LLM inference on standard GPUs

French startup Kog is developing the Kog Inference Engine (KIE) to drastically accelerate AI inference on standard datacenter graphics processing units, such as the Nvidia H200 and AMD MI300X. Rather than relying on specialized hardware, the company aims to deliver up to 30x faster large language model inference purely through low-level software optimization. In a recent technical preview, Kog demonstrated a processing speed of 3,000 tokens per second per request. This benchmark was achieved using Laneformer 2B, an open-source, purpose-built model with 2 billion parameters.

The startup's deep-level optimization strategy stems from the unique background of its solo founder, Gaël Delalleau. A former white-hat hacker and solid-state physics graduate, Delalleau applies reverse-engineering techniques down to assembly language and binary code to maximize GPU performance. However, this hands-on approach is highly time-consuming. Kog's 11-person team must spend weeks or months conducting hardware-specific research for every new GPU they support.

For AI practitioners and developers, this software-centric approach could significantly lower operational costs and reduce latency bottlenecks. Currently, workflows like software engineering suffer from long wait times, prompting providers like Anthropic to charge premium prices for features like Claude's Fast Mode. By unlocking the untapped memory bandwidth of existing enterprise GPUs, Kog hopes to make rapid single-request decoding accessible without requiring expensive hardware migrations or model fine-tuning.

Backed by Varsity VC, Scaleway, Bpifrance, and the French Tech 2030 program, Kog is currently scaling its techniques from small models to larger LLMs. The startup plans to implement its first major model at a 10x speed improvement by September. Success on this front will be critical as the company seeks to demonstrate customer traction and raise a Series A funding round.