The paper introduces ImpossibleRubrics, a 169-task benchmark of ‘impossible’ prompts (each paired with an oracle certificate defining valid honest responses) used to test whether LLM-generated rubrics reward honest answers over adversarially optimized ones, a key robustness question for rubric-based RL and LLM-as-judge evaluation. Eleven rubric generators are exploited 8-36% of the time depending on the difficulty cut, revealing a rubric-quality gap rather than task-level impossibility. Counterintuitively, a single generic rubric (‘be decisive, penalize hedging’) is exploited 64% of the time, and most tailored, task-specific rubrics perform worse, suggesting specificity can tell an attacker which false claim to fabricate.
