Finite control set model predictive control (FCS-MPC) is widely researched for converter control, as it incorporates the control objective and system constraints straightforwardly into the control. To address the drawbacks of conventional FCS-MPC, data-driven predictive control schemes, such as reinforcement learning (RL), have been proposed to tackle issues related to parameter sensitivity and unmodeled dynamics. Another challenge with FCS-MPC is the computational burden associated with complex converter systems and longer prediction horizon optimization problems. For FCS-MPC with a longer prediction horizon, the computation cost increases exponentially, rendering it impractical for real-time implementation. Additionally, RL controllers have not been explored for achieving long prediction horizon predictive control in power converters. To bridge this gap, this paper proposes introducing prediction horizons into RL for optimal power converter control, leveraging the nonlinear mapping capability and self-learning characteristics of RL. Thus, a multi-step RL controller is developed to enhance steady-state performance without increasing the computational burden. Both simulation and experimental results confirm the effectiveness of the proposed method.