With the advancement of radio technologies, wireless systems have become more convoluted. This complexity, accompanied by an increase in the number of connections, is translated into a need for more parameters to analyze and decisions to take at each instant. Artificial Intelligence (AI) comes into play by automating these processes, particularly with Deep Learning techniques, that often show the best accuracies. However, the high performance of these methods comes at the cost of lower understandability and interpretability for humans. To this end, eXplainable AI (XAI) serves as a technique to better understand the decision process of these algorithms. This paper proposes an XAI framework that can be used in a Reinforcement Learning (RL) scenario. We focus on the use case of Radio Resource Management (RRM) for improved network energy efficiency. The featured framework presents a pre-model block using Concrete Autoencoders for feature reduction and a post model block using self-supervised learning to estimate feature importance. It also incorporates DeepSHAP to provide more detailed explanations. The explanations provided by the pipeline prove useful to reduce model complexity without loss of accuracy and to understand the usage of the input features by the AI model.