Computer Science Department, MS Thesis Presentation Keon Roohani , "Coordination in Multi-Agent LLM Systems: The Role of a Question-Asking Agent in Guiding Collaborative Consensus"

Keon Roohani

MS Student

WPI – Computer Science Department 

Wednesday, April 16th , 2025

Time: 11:00 AM – 12 PM

Location : UH 347 

Advisor: Prof. Fabricio Murai

Reader: Prof. Roee Shraga

Abstract:
Coordination, debate, and reflection have shown promising improvements in multi-agent Large Language Model (LLM) task performance. Inspired by the role of questioning in human group reasoning, this research introduces a novel component to multi-agent LLM systems: a Question-Asking Agent (QAA) that guides collaboration through targeted, uncertainty-reducing questions. The QAA selects questions based on Expected Information Gain (EIG), a metric used to quantify the value of information a question may provide.

 To evaluate the impact of the QAA, a multi-agent LLM system was implemented and tested on the chess game state tracking task, a benchmark problem that challenges LLMs to maintain consistent reasoning across a sequential input. The system included generic agents collaborating through dialogue and a QAA generating questions using template-based formulations with calculable EIG. Experiments were conducted across 15 configurations varying the number of agents (1–5) and QAA strategy (none, random, EIG-driven).

 Results show that the QAA with EIG consistently improved system accuracy compared to both the baseline and the random-question QAA. This study demonstrates that EIG-guided questioning can meaningfully improve reasoning performance in multi-agent LLM systems. These findings open new directions for enhancing coordination, interpretability, and performance in multi-agent LLM settings across a range of structured reasoning tasks beyond chess.