Evaluating Different Approaches to Calibrating Conformal Predictive Systems
2020 (English)In: Proceedings of the 9th Symposium on Conformal and Probabilistic Prediction and Applications, COPA 2020, ML Research Press , 2020, p. 134-150Conference paper, Published paper (Refereed)
Abstract [en]
Conformal predictive systems (CPSs) provide probability distributions for real-valued labels of test examples, rather than point predictions (as output by regular regression models) or confidence intervals (as output by conformal regressors). The performance of a CPS is dependent on both the underlying model and the way in which the quality of its predictions is estimated; a stronger underlying model and a better quality estimation can significantly improve the performance. Recent studies have shown that conformal regressors that use random forests as the underlying model may benefit from using out-of-bag predictions for the calibration, rather than setting aside a separate calibration set, allowing for more data to be used for training and thereby improving the performance of the underlying model. These studies have furthermore shown that the quality of the individual predictions can be effectively estimated using the variance of the predictions or by k-nearest-neighbor models trained on the prediction errors. It is here investigated whether these methods are also effective in the context of split conformal predictive systems. Results from a large empirical study are presented, using 33 publicly available datasets. The results show that by using either variance or the k-nearest-neighbor method for estimating prediction quality, a significant increase in performance, as measured by the continuous ranked probability score, can be obtained compared to omitting the quality estimation. The results furthermore show that the use of out-of-bag examples for calibration is competitive with the most effective way of splitting training data into a proper training set and a calibration set, without requiring tuning of the calibration set size.
Place, publisher, year, edition, pages
ML Research Press , 2020. p. 134-150
Keywords [en]
Conformal predictive distributions, Quality estimation, Random forest, Regression, Split conformal predictive systems
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:kth:diva-350335Scopus ID: 2-s2.0-85126598294OAI: oai:DiVA.org:kth-350335DiVA, id: diva2:1883763
Conference
9th Symposium on Conformal and Probabilistic Predictions with Applications, COPA 2020, Virtual, Online, Italy, Sep 9 2020 - Sep 11 2020
Note
QC 20240711
2024-07-112024-07-112024-07-11Bibliographically approved