Researchers propose Elo-per-token analysis, a method that identifies the best intermediate solution within each token budget and uses a Bradley-Terry model to convert task-specific rankings into comparable Elo ratings across benchmarks with different scoring scales. Testing four general-purpose agents on four open-ended benchmarks with sessions up to 100M tokens, plus three optimization harnesses in controlled settings, they find agents initially convert tokens into Elo gains faster than random sampling but hit diminishing and eventually negative marginal returns, unlike human experts who show superlinear improvement. Using the identified inflection point where marginal gains match random sampling, the authors redistribute a 100M-token budget across parallel sessions, gaining +264 Elo over a single long session and +355 Elo over ten short sessions.
