The present work aims to answer three essential research questions (RQs) that have previously not been explicitly dealt with in the field of applied machine learning (ML) in steel process engineering. RQ1: How many training data points are needed to create a model with near-upper-bound predictive performance on test data? RQ2: What is the near-upper-bound predictive performance on test data? RQ3: For how long can a model be used before its predictive performance starts to decrease? A methodology to answer these RQs is proposed. The methodology uses a developed sampling algorithm that samples numerous unique training and test datasets. Each sample was used to create one ML model. The predictive performance of the resulting ML models was analyzed using common statistical tools. The proposed methodology was applied to four disparate datasets from the steel industry in order to externally validate the experimental results. It was shown that the proposed methodology can be used to answer each of the three RQs. Furthermore, a few findings that contradict established ML knowledge were also found during the application of the proposed methodology.