Long prediction horizon finite control-set model predictive control (FCS-MPC) exhibits excellent performance regarding closed-loop stability, harmonic distortions, and switching frequency. However, the computational burden for practical implementation increases exponentially for the traditional exhaustive enumeration approach. Conventional methods include reformulating it as an integer least-square (ILS) problem and employing supervised imitation learning techniques based on artificial neural networks (ANNs) to mitigate the computation burden issue from longer prediction horizons. In this article, a novel autonomous controller is developed by fusing the reinforcement learning (RL) framework with a long prediction horizon for converter control. In this way, the RL agent learns autonomously the optimal switching strategy by interacting with the converter system. In addition, to ensure a tractable training process and convergence, a hierarchical action space partitioning approach is proposed based on the receding horizon principle. Finally, an online demonstration framework is presented to realize the practical implementation of the algorithm. The effectiveness of the proposed long prediction horizon RL controller is validated via both simulations and experiments.