Netflix built an in-house LLM serving platform combining Triton and vLLM to handle inference across CPUs and GPUs within its existing JVM-based infrastructure. The platform delegates larger model requests to a specialized serving system where Triton manages model loading and scheduling while vLLM performs the actual inference. The architecture addresses operational challenges including version compatibility between components, custom model integration through vLLM extension points, and constrained decoding to ensure outputs conform to required formats like JSON.
