Safe Battery Use and Grid-Convenience in an Intelligent Energy Control System

Energy control systems need to balance the use of locally produced and consumed power. Their goal is to maximize self-use by charging local storage facilities while maintaining grid-convenience, i.e. avoiding largely different successive feed-in rates to the grid. To avoid over- or undercharging of the battery, control needs to respect certain safety thresholds. Recently, energy control systems have been enhanced with learning components for optimized control. This however, makes it more difficult to ensure that control respects above-mentioned safety thresholds, especially in the presence of incomplete information.

This paper proposes an approach for safe learning in energy control systems by combining a shielded reinforcement learning (RL) agent that determines which percentage of the locally produced energy is stored in a battery and which percentage is fed into the grid, with a digital twin of the battery that maintains information on the State of Charge of the battery. To ensure safe learning, we use a formally verified shield, which ensures under certain assumptions that unsafe actions, i.e. overcharging of the battery, are avoided. For formal verification, the RL agent is replaced by a contract and we assume that the digital twin is always available and provides missing information to the RL agent in case of communication losses. This combination then allows us to formally verify the safety of the resulting energy control system with complex discrete and continuous dynamics. We illustrate our approach by developing a Simulink model of a real smart house in Heeten, NL [35]. Our experimental results demonstrate that both self-use and grid-convenience can be achieved while maintaining safe battery use.