Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most
Tahreem Yasir, Wenbo Li, Sam Gilson, Sutapa Dey Tithi, Xiaoyi Tian, Tiffany Barnes
LLMs tested as tutoring agents on 10,836 propositional logic solution-feedback pairs perform well when confirming correct answers but fail systematically at the two cases that actually matter: they over-reject valid but non-standard student reasoning and over-validate outright wrong answers. The benchmark uses knowledge-graph-derived ground truth to score seven models across three feedback conditions, and the failure pattern held across all models regardless of how much solution context was provided — pointing to a structural limitation, not a fixable prompting issue. Accurate diagnosis also didn't translate to pedagogically useful feedback, meaning getting the classification right and giving helpful guidance are two separate problems LLMs aren't jointly solving.
No production traction yet. The GitHub references are all automated arxiv digest aggregators, not implementers building on the work. Zero citations on Semantic Scholar. The paper's practical recommendation — use knowledge-graph-grounded models for diagnosis and LLMs only for open-ended dialogue and scaffolding — is consistent with how serious ITS vendors already think about hybrid architectures, but this work hasn't been picked up by that community in any visible way.
Effective tutoring requires distinguishing optimal, valid but suboptimal, and incorrect student solutions, a distinction central to intelligent tutoring systems (ITS) but untested for LLM-based tutors. As LLMs are increasingly explored as conversational complements to ITS, evaluating their diagnostic precision is essential. We present a benchmark of seven LLM feedback agents in propositional logic using knowledge-graph-derived ground truth across 10,836 solution--feedback pairs and three feedback conditions. Models achieved near-ceiling performance on optimal steps but systematically over-rejected valid but suboptimal reasoning and over-validated incorrect solutions, precisely where adaptive tutoring matters most. These failures persisted across models regardless of solution context, suggesting architectural rather than informational limits. Moreover, accurate diagnosis did not reliably produce pedagogically actionable feedback, revealing a gap between diagnostic judgment and instructional effectiveness. Our findings suggest that LLMs are better suited for hybrid architectures where KG-grounded models handle diagnosis while LLMs support open-ended scaffolding and dialogue.