OpenAI found that its GPT-5.6 Sol model scored only 13.3% on the ARC-AGI-3 benchmark’s public task set when run through the official evaluation harness, because that harness discarded the model’s private reasoning after every action and forced it to restart its problem-solving process from scratch each turn. Enabling two existing API settings, retained reasoning and compaction, let the model preserve and condense its prior reasoning across turns, which lifted the score to 38.3%, roughly tripling performance while using six times fewer output tokens. OpenAI said the result illustrates that a benchmark score reflects the surrounding harness and configuration as much as it reflects the underlying model.
