The vLLM-Omni team details how it serves Qwen3-Omni, a multimodal text-and-speech model, using a three-stage pipeline of Thinker (reasoning), Talker (codec generation), and Code2Wav (waveform reconstruction), each optimized independently with batching, CUDA graphs, and asynchronous inter-stage handoffs. Through successive optimization layers, throughput improved roughly 5.4x, from 2.2 to 11.7 requests per second. Audio time-to-first-packet dropped from 5.9 seconds to 632 milliseconds under high concurrency, while keeping generation faster than real time.
