TraceDance automatically builds targeted agent-behavior benchmarks from real deployment traces rather than fixed benchmark suites, using an Anchor-and-Confirm pipeline (programmable retrieval plus candidate-level confirmation by a flash LLM) and an Anchor Synthesis Loop that generates and revises behavior specifications. It evaluates models via decision-point continuation against behavior-specific rubrics, avoiding environment replay or reference answers. From over 252,000 real sessions the system produced 107 benchmarks with 4,125 test instances (84% human-validated), and found nine frontier LLMs average only a 26.7% pass rate on the evaluated decision points.
