Simon Willison describes smevals, an evaluation framework built with Prime Radiant that separates running tasks against model variants from grading the resulting outputs. The framework lets users execute the same tasks across different model configurations and then score results using customizable checks and graders, viewable through an interactive local server or exported as static HTML reports. It introduces standardized terminology — evals, tasks, configs, runners, graders, and checkers — intended to give LLM evaluation workflows a coherent shared vocabulary.
