This article fills a gap by presenting the first experimental benchmarking study that compares the state-of-the-art model-free reinforcement learning (MFRL) controller with a widely adopted proportional integral (PI) controller with space vector modulation (SVM), finite control set model predictive control (FCS-MPC), and a supervised imitation artificial neural network (ANN) controller. The methods are implemented on a benchmark two-level voltage source converter (VSC) platform representative of uninterruptible power supply (UPS) applications. The evaluation covers steady-state performance, dynamic response, computational burden, robustness, and practical implementation considerations. By providing a holistic qualitative and quantitative comparison, the article delivers insights into the practical benefits and limitations of state-of-the-art converter control strategies. The results also provide the power electronics community with a clear reference and guidance for adopting RL-based controllers compared to well-established model-based or classical linear controllers.