Ventor-QTest is a black-box auditing framework for assessing the inference quality of vendor-hosted large language model APIs without access to internal token probabilities. It combines a repeated-request method that measures average fidelity loss from categorical output distributions with a long-sequence method that computes extreme fidelity loss from empirical tail statistics. Testing across multiple model-routing conditions found that extreme fidelity loss correlates with degraded performance on long-horizon, agentic tasks, while showing limited association with standard accuracy benchmarks.
