LMSYS Rebuilds SGLang Cache to Support Hybrid AI Models

LMSYS has redesigned the caching architecture of its SGLang model-serving engine to support hybrid AI architectures. Previously, prefix caching—which skips recomputing key-value states for shared token sequences—only worked reliably for pure transformer models. Hybrid models combining full attention, sliding window attention, and Mamba or state-space layers broke these caching rules, forcing slow recomputations. The new Unified Radix Cache replaces separate, duplicated cache classes with a single composable radix tree.

To handle the conflicting reuse rules of different layers, the Unified Radix Cache uses pluggable components for full attention, sliding window attention, and Mamba configurations. These components use a voting mechanism to determine the deepest prefix boundary that all active layers can safely accept. This prevents invalid cache reuse while maximizing efficiency. Developers can activate this feature by setting the environment variable SGLANG_ENABLE_UNIFIED_RADIX_TREE=1.

The performance improvements are substantial. When using the L3 Mooncake cache, the DeepSeek-V4-Flash model achieved 145.5K effective input tokens per second, compared to just 9.4K tokens per second using a GPU-only cache at a 98 percent hit rate. Additionally, a new session-aware eviction policy, enabled via the --enable-session-radix-cache flag, attaches a session identifier to requests. This optimization improves Time-to-First-Token by 2.9 percent to 16.6 percent on SWE-bench agent workloads compared to standard least-recently-used eviction.

For developers running long sliding-window attention workloads, LMSYS has also introduced an experimental, opt-in Rust tree core. This prototype reduces Time-to-First-Token by up to 42 percent. For practitioners, these updates mean that modern open-weight hybrid models can finally run multi-turn agentic workflows and long-context conversations without suffering massive latency penalties from full prompt recomputation.