Researchers analyzed 127,833 real-world clinical queries from 6,342 physicians, advanced practice providers, and nurses across 35 specialties during an eight-month deployment of an institutional AI assistant. Using a clinician-validated taxonomy called RCQ-Map, they found documentation, administration, and knowledge retrieval dominated real usage while diagnosis queries were rare, at only 3.7%. Applying the same framework to 58 public clinical AI benchmarks revealed the median benchmark included no documentation tasks and shared just 31% of real-world task types. The authors conclude that current benchmark scores say little about performance on the clinical tasks AI assistants are actually used for, and call for evaluation methods that better match real-world use.
