A paper describes RTP-LLM, an LLM inference engine designed to address deployment bottlenecks in production serving. It combines file-order-driven parallel I/O for faster model loading, a prefill-decode disaggregation architecture that separates compute-bound and memory-bound phases with hierarchical KV-cache management, and modular speculative decoding with adaptive KV-cache quantization. Evaluated on production workloads, the system reports model-loading speedups of roughly 4.7x-6.3x and about 35-37% lower P95 time-to-first-token latency compared to competing inference engines.
