NVIDIA Accelerates Hybrid Model Decoding with ReplaySSM

Engineers from NVIDIA and Princeton University's Dao AI Lab have developed ReplaySSM, a memory-management approach for state-space model (SSM) layers in hybrid AI architectures. Instead of updating and saving full state matrices on every token step, ReplaySSM maintains a baseline checkpoint alongside a small buffer of recent inputs using a window size of L = 16. On NVIDIA Blackwell Ultra B300 GPUs operating at batch size 256, the method cuts state memory bandwidth traffic roughly in half, accelerating SSM decoding kernels by 1.46x to 1.80x and boosting overall decode step speed by 1.20x to 1.47x.

The system simplifies rollback handling in speculative decoding by appending draft inputs straight into the buffer. During verification with multi-token prediction heads using draft count T = 4 on GSM8K prompts, ReplaySSM yielded 1.85x and 2.10x higher speculative throughput at batch size 512 on 120B-class models compared to tuned vLLM baselines. Evaluation spanned four target architectures: Nemotron-3-Nano-4B on one H100 GPU (21 SSM and 4 attention layers), Nemotron-3-Super-120B-A12B on a B300 GPU (40 SSM and 8 attention layers), Qwen3.5-4B on an H100 (24 SSM and 8 attention layers), and Qwen3.5-122B-A10B on a B300 (36 SSM and 12 attention layers).

ReplaySSM also separates output calculation from recurrent updates, allowing each task to execute at different mathematical precisions. Reading low-precision NVFP4 state copies speeds up non-flush kernels by 3.8x on Mamba-2 layers. By maintaining recurrence checkpoints in FP16 or FP8 while reading NVFP4 for outputs, ReplaySSM prevents compounding quantization errors over extended contexts. On GPQA-Diamond benchmarks, three of four models remained within about 2 percentage points of FP32 accuracy. Nemotron-3-Super-NVFP4 landed within 1.4 points of its 79.3% FP32 score, Qwen3.5-122B-A10B stayed within 1.7 to 2.0 points of its 82.7% reference, and Qwen3.5-4B gained 0.7 to 1.5 points over its 76.3% base, whereas Nemotron-3-Nano-4B dropped 5.6 to 6.7 points from 56.7%.

On PG-19 runs out to 128K generated tokens using Qwen3.5-4B, this dual-precision split suppressed KL divergence from 0.136 down to 0.0038. ReplaySSM removes major memory bandwidth bottlenecks for operators serving hybrid models built with Mamba-2 or Gated DeltaNet layers. The technique is already deployed in production models like GLM-5.3-Flash and Kimi-K3, with open-source integrations available across TensorRT-LLM, vLLM, SGLang, and FlashInfer.