Global State Prediction for Learning Collective Transport with Distributed Reinforcement Learning
Abstract:
This dissertation introduces Global State Prediction (GSP), a novel framework addressing the fundamental challenge of non-stationarity in multi-agent reinforcement learning for collective transport tasks. While traditional approaches to multi-agent coordination rely on either centralized control or complete information sharing, GSP enables effective decentralized coordination through prediction-based mechanisms that require minimal communication.
We first present the core GSP framework, where agents predict future global states based on shared partial observations, demonstrating superior performance compared to both implicit communication and global knowledge approaches. Building on this foundation, we introduce Neighborhood-Limited GSP (GSP-N), which restricts information exchange to immediate neighbors while maintaining comparable performance, reducing communication complexity from O(n^2) to O(1) as swarm size increases.
The dissertation's most significant contribution lies in two memory-enhanced variants: Recurrent GSP-N (R-GSP-N) and Attention-based GSP-N (A-GSP-N), which incorporate temporal reasoning to improve prediction accuracy in complex environments. These memory-enhanced approaches show complementary strengths, with R-GSP-N excelling in adaptation to unfamiliar payload dynamics and A-GSP-N demonstrating superior performance in spatially complex environments.
Extensive experiments across diverse scenarios demonstrate that our proposed approaches significantly outperform baseline methods, particularly in challenging conditions involving non-uniform objects with unknown mass distributions and complex obstacle configurations. Notably, policies trained with GSP-N variants on smaller teams transfer effectively to larger swarms without retraining, demonstrating exceptional scalability. This work advances both theoretical understanding and practical capabilities for robust collective transport in real-world swarm robotics applications.
Advisor: Professor Carlo Pinciroli, Robotics Engineering, WPI
Committee: Professor Kevin Leahy, Robotics Engineering, WPI; Professor Guanrui Li, Robotics Engineering, WPI; Nicola Bezzo, Ph.D., University of Virginia