AI evaluation costs have become a significant computational bottleneck that rivals or exceeds training expenses for modern systems. The Holistic Agent Leaderboard spent approximately $40,000 to evaluate 21,730 agent rollouts across multiple models and benchmarks, with individual evaluations ranging from tens to tens of thousands of dollars depending on model pricing and task complexity. Static benchmark compression techniques that achieved 100-200x cost reductions no longer apply effectively to agent-based or training-in-the-loop evaluations, which compress only 2-3.5x at best, forcing institutions to choose between statistical rigor and affordability.
