vLLM’s tiered KV cache offloading system preserves evicted key-value cache data across host memory, storage, and remote peers instead of recomputing it, enabling horizontal scaling of the cache and higher serving capacity. The architecture is host-centric: all evicted data first flows through CPU DRAM as a primary tier before optionally cascading to secondary tiers such as content-addressed filesystem storage, S3-compatible object stores, or cross-instance RDMA transfers, letting accelerator memory be freed as soon as data reaches the host. Reported testing shows the system sustains high throughput beyond 128 concurrent conversations by serving storage-backed cache hits, more than doubling performance versus serving without offloading.
