RealCLI is a paired benchmark that scores both task execution and human-agent collaboration quality on realistic command-line tasks, using a five-dimensional framework with 22 sub-dimensions and 51 interaction scenario templates. RealCLI-EXEC gives agents complete specifications upfront, while RealCLI-COLLAB uses a scripted user who reveals details step by step, corrects mistakes, changes constraints, and makes approval decisions. Frontier agents perform poorly: the best configuration averages 49.2% and peaks at 37.6% on the joint collaboration score, with models struggling most on steerability, trust calibration, and safety boundaries.
