In this work we present a framework to explain the prediction of the velocity fluctuation at a certain wall-normal distance from wall measurements with a deep-learning model. For this purpose, we apply the deep-SHAP (deep Shapley additive explanations) method to explain the velocity fluctuation prediction in wall-parallel planes in a turbulent open channel at a friction Reynolds number ${\textit{Re}}_\tau =180$ . The explainable-deep-learning methodology comprises two stages. The first stage consists of training the estimator. In this case, the velocity fluctuation at a wall-normal distance of 15 wall units is predicted from the wall-shear stress and wall-pressure. In the second stage, the deep-SHAP algorithm is applied to estimate the impact each single grid point has on the output. This analysis calculates an importance field, and then, correlates the high-importance regions calculated through the deep-SHAP algorithm with the wall-pressure and wall-shear stress distributions. The grid points are then clustered to define structures according to their importance. We find that the high-importance clusters exhibit large pressure and shear-stress fluctuations, although generally not corresponding to the highest intensities in the input datasets. Their typical values averaged among these clusters are equal to one to two times their standard deviation and are associated with streak-like regions. These high-importance clusters present a size between 20 and 120 wall units, corresponding to approximately 100 and 600 $\unicode{x03BC} \textrm {m}$ for the case of a commercial aircraft.