TokenRouter is a serving system designed for token-level routing across multiple large language models within a single inference pipeline. Rather than relying on a shared scheduler built for a single model, it runs an independent subserver for each model and dispatches requests asynchronously, which avoids the step desynchronization and batch admission delays that affect prior serving designs under token-level routing. Each subserver uses a delayed-batching scheduler whose parameters are derived from a mathematical throughput model. Across a range of routing algorithms, workloads, and model pairs, the system was measured to deliver roughly 2x to 64x higher decoding throughput than existing serving systems.
