Data-driven learning-based control methods, such as reinforcement learning (RL), have become increasingly popular with recent proliferation of the machine learning paradigm. These methods address the parameter sensitiveness and unmodeled dynamics in model-based controllers, such as finite control-set model predictive control. RL agents are typically utilized in simulation environments, where they are allowed to explore multiple 'unsafe' actions during the learning process. However, this type of learning is not applicable to online self-learning of controllers in physical power converters, because unsafe actions would damage them. To address this, this letter proposes a safe online RL-based control framework to autonomously find the optimal switching strategy for the power converters, while ensuring system safety during the entire self-learning process. The proposed safe online RL-based control is validated in a practical testbed on a two-level voltage source converter system, and the results confirm the effectiveness of the proposed method.