Various approaches to explaining predictions of black box models have been proposed, including model-agnostic techniques that measure feature importance (or effect) by presenting modified test instances to the underlying black-box model. These modifications rely on choosing feature values from the complete range of observed values. However, when applying machine learning algorithms to the task of forecasting from multivariate time-series, it is suggested that the temporal aspect should be taken into account when analyzing the feature effect. A modification of individual conditional expectation (ICE) plots is proposed, called ICE-T plots, which displays the prediction change for temporally ordered feature values. Results are presented from a case study on predicting the Swedish gross domestic product (GDP) based on a comprehensive set of indicator and prognostic variables. The effect of calculating feature effect with and without temporal constraints is demonstrated, as well as the impact of transformations and forecast horizons on what features are found to have a large effect, and the use of ICE-T plots as a complement to ICE plots.