Perplexity Debuts ROSE Serving Stack to Outrun vLLM

Perplexity has detailed its new custom embedding serving stack, which achieves significant latency and throughput improvements over vLLM. The architecture consists of three specialized services working in tandem. Ivy, a Rust-based HTTP gateway, handles CPU-side tasks such as JSON parsing, tokenization, input templating, and batch splitting. Tulip, a gRPC inference server built with Rust, tokio, and tonic, manages scheduling and batching. Finally, the Runtime-Optimized Serving Engine, or ROSE, executes model forward passes in Python and manages CUDA graphs. By dropping the KV cache and swapping paged attention for ragged attention, ROSE successfully reuses optimized LLM kernels for embedding workloads.

To overcome CPU-side bottlenecks on small embedding batches, the stack employs CUDA graphs and a custom LazyTensor abstraction. Because embedding shapes vary, Tulip captures CUDA graphs lazily as live traffic arrives, padding token counts to buckets of 64 or 256 to avoid capturing thousands of graphs upfront. Meanwhile, LazyTensors track host buffers in page-locked memory alongside a cudaMemcpyAsync event, allowing the CPU to prepare the next batch while the GPU processes the current one. Tulip's scheduler operates on the principle that latency scales with token count rather than sequence count, pulling sequences first-come, first-served until hitting a budget of approximately 512 tokens, which saturates a sub-1B model.

In benchmark tests against vLLM v0.22.0 in BF16, Perplexity's stack achieved p50 and p99 wins across low-latency single-query, batch scoring, high-throughput indexing, and concurrent request scenarios. These tests utilized real model weights for BGE-M3 and pplx-embed-1-0.6b at sequence lengths of 128, 512, and 4096 tokens. ROSE integrates multiple attention backends, including FlashInfer 2, FlashInfer 3, and FlashAttention 4, to optimize performance based on head count and dimension. For AI practitioners, this development provides a clear blueprint for optimizing search pipelines. It demonstrates that for short queries, overcoming kernel launch overhead via CUDA graphs and lazy tracking is far more critical than optimizing the attention kernels themselves.