Session: 06-01-01 Computational Mechanics and Design Applications

Submission Number: 157355

Reinforcement Learning-Based Controller With NMPC-Assisted Training for Autonomous Surface Vessels 

Autonomous navigation of marine vessels has gained significant attention, but conventional model-based control strategies face challenges due to the inherent uncertainties in the marine environment, making accurate vessel modeling difficult. This study investigates the application of reinforcement learning (RL) controllers, specifically using the Deep Deterministic Policy Gradient (DDPG) algorithm, to reduce dependency on model accuracy by enabling controllers to learn through experience. To further enhance RL controller performance, this work integrates Nonlinear Model Predictive Control (NMPC) for assisted training.

The methodology involves developing a DDPG-based RL controller for the approximated test vessel’s dynamics and training the RL controller for point tracking. Secondly, the controller is evaluated for trajectory tracking tasks. Observations reveal that, while the RL controller successfully learns the control actions required for trajectory tracking, the generated actions lack the smoothness necessary for real-world applications. To address this, NMPC is introduced to assist the RL controller in optimizing control actions during training, yielding results that are more suitable for practical implementation.

The findings demonstrate that RL controllers can effectively learn vessel dynamics through iterative training, reducing the reliance on precise vessel models. Additionally, the integration of NMPC enhances control action quality. This experiment paves the way for potential combination of established controller with RL controller in order to enhance accuracy and performance. Finally, the testing and validation in experiment setup will lead to further solidify the findings.

Keywords: Nonlinear model predictive controller (NMPC), Reinforcement Learning (RL), Deep Deterministic Policy Gradient (DDPG)

Presenting Author: Hasanat Zaman Canada National Research Council

Presenting Author Biography: Master's Student in Memorial University of Newfoundland