SpiralDB's Vortex Streams S3 Data to GPUs at 60 Gbps

Onur Satici, a staff engineer at SpiralDB and core maintainer of Vortex, recently detailed how the open-source columnar file format optimizes machine learning pipelines. Now hosted under the Linux Foundation's LF AI & Data, Vortex achieves S3-to-GPU scan speeds up to 30 times faster than Apache Parquet. It also outperforms Parquet by 100 times or more in random access. By streaming data directly from cloud storage to GPUs at speeds up to 60 Gbps, Vortex eliminates the traditional CPU and NVMe bottlenecks that leave expensive hardware idling.

Traditional pipelines using frameworks like PyTorch must fetch data from S3, save it to NVMe, decompress it on the CPU, and copy it over PCIe to the GPU. Vortex replaces this multi-step process with a zero-copy memory pipeline. It uses cascading lightweight encodings and layout-based segment pruning to minimize data movement. Because Vortex decouples logical and physical types, it can perform computations directly on compressed data. For example, summing a run-end encoded array requires simple multiplication rather than iterating through every single decompressed element.

To achieve these speeds, Vortex utilizes a specialized buffer pool. By accumulating HTTP buffers into pinned, page-locked memory, the system bypasses the standard CUDA bounce buffer. This reduces the transfer to a single memory copy before the data lands on the GPU. The format's sliceable compression also allows practitioners to decompress only the specific row ranges required for a query, rather than entire data blocks.

For machine learning practitioners, this architecture eliminates what Satici calls the "movement tax" and "decision tax" of training. Instead of reprocessing, re-tokenizing, and reshuffling entire datasets in S3 to test a new data mix or curriculum, developers can simply alter their scan query. This allows instant iteration, ensuring that GPUs spend their time processing compute workloads rather than waiting on slow data pipelines.