Researchers introduce Taste-Bench, a benchmark that measures an LLM agent’s “taste” — its ability to make good long-horizon decisions at forks where multiple directions are available but only one leads to a better outcome. The benchmark questions are mined automatically from parallel attempts and detours in agent trajectories on engineering and research tasks, without human annotation. The best frontier model answers only 59.7% of questions correctly, and the authors show that distilling a teacher’s hindsight judgment into a student model improves decision-making and end-to-end success on held-out SWE-bench Pro tasks.
