An assistant can answer a question correctly and still help someone pursue the wrong solution. Ai2’s November 13 introduction to XYBench examines that gap: whether models identify a mistaken premise and redirect attention toward the user’s actual goal.
The benchmark contains more than 8,000 questions from technical forums, curated examples and questions generated from WikiHow guides. The generated material forms the majority, so the collection should not be described as entirely authentic user conversations. Its published evaluation framework measures whether an answer includes relevant solutions, which approach it emphasizes and whether it identifies the misconception.
Ai2 reports that models often mention a useful alternative while continuing to emphasize the original mistaken approach. Claude Sonnet 4.5 acts as the automated judge; human agreement was lower for judging emphasis than for spotting whether a solution appeared.
The scope matters: these are English, mainly procedural questions evaluated with one response per question. For assistant developers, XYBench offers a useful diagnostic alongside task-completion tests. It does not establish reliability in extended conversations or demonstrate that an autonomous agent will act safely.
Chart: Ai2 / XYBench. Results shown are for the Stack Overflow subset.
