Renewable generation is leading to rapidly shifting power flows and it is anticipated that traditional power system control may soon be inadequate to cope with these fluctuations. Traditional control include human-in-the-loop-control schemes while more autonomous control methods can be categorized into Wide-Area Monitoring, Protection and Control systems (WAMPAC). Within this latter group of more advanced systems, reinforcement learning (RL) is a potential candidate to facilitate power system control facing these new challenges. In this paper we demonstrate how a model based reinforcement learning (MBRL) algorithm, which learns and uses an internal model of the world, can be used for autonomous power system control. The proposed RL agent, called the World Model for Autonomous Power System Control (WMAP), includes a safety shield to minimize risk of poor decisions at high uncertainty. The shield can be configured to permit WMAP to take actions with the condition that WMAP asks for guidance, e.g. from a human operator, when in doubt. As an alternative, WMAP could be run in full decision support mode which would require the operator to take all the active decisions. A case study is performed on a IEEE 14-bus system where WMAP is setup to control setpoints of two FACTS devices to emulate grid stability improvements. Results show that improved grid stability is achieved using WMAP while staying within voltage limits. Furthermore, a disastrous situation is avoided when WMAP asks for help in a test scenario event that it had not been trained for.

There is limited application of closed-loop control using model-based approaches in wide area monitoring, protection, and control. Challenges that impede model-based approaches include engineering complexity, convergence issues, and model errors. Specifically, considering the rapid growth of distributed generation and renewables in the grid, maintaining an updated model without model errors is challenging. As an alternative to model-based approaches, data-driven control architectures based on reinforcement learning (RL) have shown great promise. In this work, we confront safety concerns with data-driven approaches by studying safe RL to improve voltage and power flow control. For both a model-free RL agent and a model-based RL agent, the accumulated constraint violation is investigated in a case study on the IEEE 14-bus and IEEE 57-bus systems. To evaluate performance, agents are compared against a model-based approach subject to errors. Our findings suggest that RL could be considered for optimizing voltage and current setpoints in systems when topological model errors are present.

A coordinated control of Flexible AC Transmission Systems (FACTS) reference setpoints is often absent in real systems. Despite the power quality gains demonstrated in studies, this absence can partly be derived from challenges with model-based control. As promising alternative methods of control, data driven approaches based on reinforcement learning (RL) have been considered. In this work, we study the potential gains in power quality using RL while recognizing the increasing number of installed Phasor Measurement Units, providing limited but reliable information. We demonstrate on the IEEE 14-bus and IEEE 57-bus systems that by adding a few measurements per FACTS device and a constraint violation signal, an RL scheme may significantly improve power quality compared to a baseline of fixed setpoints. To evaluate robustness, several configurations are simulated and for larger systems, we identify unobserved constraint violations as the main risk and propose a potential path for new research.

With the growing electrification and integration of renewables, network operators face unprecedented challenges. Coordinated control of Flexible AC Transmission Systems (FACTS) setpoints using real-time optimization techniques has been proposed to substantially improve voltage and power flow control. However, optimizing the setpoints of several FACTS devices is rarely done in practice. In part, this can be derived from the challenges with model-based methods. As alternative control methods, data-driven methods based on reinforcement learning (RL) have shown great promise. However, RL has its own challenges that include data and safety during learning. Motivated by the increasing collection of data, we study an RL-based optimization of FACTS setpoints and how datasets can be leveraged for pre-training to improve safety. We demonstrate on the IEEE 14-bus and IEEE 57-bus systems that an offline to online RL algorithm can significantly reduce voltage deviations and constraint violations. The performance is compared against an RL agent learning from scratch and the original control policy that generated the dataset. Moreover, our analysis shows that dataset coverage and the amount of pre-training updates affect the performance considerably. Finally, to identify the gap to an optimal policy, the proposed approach is benchmarked against an optimal controller with perfect information.