Computer Science Department, PhD Research Qualifier, Ethan Croteau: "Assessing GPT’s Math Problem-Solving with Critical Imagery"

Computer Science Department

PhD Research Qualifier

Title: Assessing GPT’s Math Problem-Solving with Critical Imagery

Presenter: Ethan Croteau, PhD Student

Department: WPI – Computer Science

📅 Date: Wednesday, March 26, 2025

⏰ Time: 11:00 am – 12:00 pm

📍 Location: Unity Hall 320G

Advisor: Prof. Neil Heffernan

Reader: Prof. Emmanuel Agu

Abstract:

Mathematical problem-solving frequently depends on visual elements like graphs, diagrams, and geometric figures, which provide critical context and deepen conceptual understanding in educational settings. Though Large Language Models (LLMs) like GPT-4o excel in textbased reasoning, their performance on image-dependent math problems, essential for tutoring, remains weak despite multimodal capabilities. This study evaluates GPT-4o’s performance in solving student-oriented problems from the Illustrative Mathematics curriculum (Grades 6–8), categorizing problems based on their visual necessity–Required, Useful, Not Required, or Insufficient. We compare the model’s accuracy across multiple attempts with and without images. Among 1,103 problems, 70.2% require visuals for correct solutions, yet GPT-4o achieves only 38.48% accuracy with images, often misinterpreting diagrams, and drops to 2.36% without images. Our findings reveal GPT-4o’s visual reasoning gaps, including misidentified geometric relationships and graph misreadings, underscoring images’ critical role in effective AI-driven tutoring. This research advances AI-supported math education by sharing anonymized data to inform alt-text generation for visually impaired learners, enhancing tutoring systems, and fostering human-AI collaboration across diverse classrooms. Future work will prioritize AI-generated image descriptions to improve accessibility and fine-tuning to address visual reasoning weaknesses, enhancing LLM math capabilities.