Electricity price forecasting is a critical aspect of energy market operations and investments, therefore, the development of new, more accurate forecast models is a continuous effort. However, the evaluation of forecasting models is typically limited to general accuracy metrics without a deep understanding of the underlying root causes of making these predictions, which limits their real world applicability in critical infrastructure. In this work, we extend two state-of-the-art data-driven forecast models to predict the day-ahead system prices of the NordPool market. The goal is to attribute the importance that the models give to the input features, and, with this, see if the cause-effect relationship learnt by the models is consistent with reality. When this is not the case, the reliability of these models in real-world applications, where forecasts inform downstream decision-making, cannot be trusted, despite their overall accuracy. The findings of this study indicate that, while generally precise, even state-of-the-art forecasting models face challenges in maintaining consistency with real-world conditions.