Computer Science Department, MS Thesis Presentation Jose Fabrizio Filizzola Ortiz " Improving LLMs Persuasiveness with Online Debates Data and Argumentation via Multi-Agent AI Systems"

Jose Fabrizio Filizzola Ortiz

MS Student

WPI – Computer Science Department 

Monday, April 28, 2025

Time: 3:00 p.m. – 4:00 p.m.

Location: Fuller Labs 311

Advisor: Prof. Fabricio Murai

Reader: Prof. Neil Heffernan

Abstract:

Debating ideas can help people learn about, reconcile, or dismiss different perspectives about a theme. LLMs are promising agents for debates as they were trained for conversations using big data, much of it directly collected from the Web. However, existing LLMs can be less persuasive than experience debaters and generate repetitive statements, which weakens their arguments and can produce overwhelming responses. In addition, LLMs tend to seek the user's approval even if they are wrong, which can make the debate biased towards the user's opinion and arrive at incorrect conclusions. To overcome these challenges, we leveraged fine-tuning from large data and multi-agent AI systems. We fine-tuned a LLama 3 model using a dataset collected from the subreddit "Change My View" containing arguments from successful debates. For the multi-agent component, we proposed a framework with a summarizer, planner, and generator. First, we evaluated the fine-tuned against the pre-trained model in a in-person lab experiment to determine the relevant debate criteria: conciseness, persuasiveness, and (lack of) sycophancy. After incorporating the human feedback, we developed the multi-agent debater that we compared against the pre-trained model using LLM evaluators. The results show that the multi-agent debater performs significantly better than the pre-trained model.