Reinforcement learning (RL) has gained popularity in power electronics due to its ability to handle nonlinearities and self-learning characteristics. When properly configured, an RL agent can autonomously learn the optimal control policy by interacting with the converter system. In particular, similar to conventional finite-control-set model predictive control (FCS-MPC), the RL agent can learn the optimal switching strategy for the power converter and achieve desirable control performance. However, the alteration of closed-loop dynamics by the RL controller poses challenges in ensuring and assessing system stability. To address this, the article proposes formulating a Lyapunov function to guide the agent in learning an optimal control policy that enhances desirable control performance while ensuring closed-loop stability. Additionally, the practical stability region of the system is quantified by deriving a compact set regarding the convergence of voltage control error. Finally, the proposed Lyapunov-guided RL controller is validated through a demonstration framework with a practical experimental setup. Both simulation and experimental results confirm the effectiveness of the proposed method.