Endre søk
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Evaluating Different Approaches to Calibrating Conformal Predictive Systems
KTH, Skolan för elektroteknik och datavetenskap (EECS), Datavetenskap, Programvaruteknik och datorsystem, SCS. Stena Line, Sweden.
Stena Line, Sweden; Centre for Reliable Machine Learning, University of London, UK.
Stena Line, Sweden; Department of Pharmaceutical Biosciences, Uppsala University, Uppsala, Sweden.
KTH, Skolan för elektroteknik och datavetenskap (EECS), Datavetenskap, Programvaruteknik och datorsystem, SCS.ORCID-id: 0000-0001-8382-0300
2020 (engelsk)Inngår i: Proceedings of the 9th Symposium on Conformal and Probabilistic Prediction and Applications, COPA 2020, ML Research Press , 2020, s. 134-150Konferansepaper, Publicerat paper (Fagfellevurdert)
Abstract [en]

Conformal predictive systems (CPSs) provide probability distributions for real-valued labels of test examples, rather than point predictions (as output by regular regression models) or confidence intervals (as output by conformal regressors). The performance of a CPS is dependent on both the underlying model and the way in which the quality of its predictions is estimated; a stronger underlying model and a better quality estimation can significantly improve the performance. Recent studies have shown that conformal regressors that use random forests as the underlying model may benefit from using out-of-bag predictions for the calibration, rather than setting aside a separate calibration set, allowing for more data to be used for training and thereby improving the performance of the underlying model. These studies have furthermore shown that the quality of the individual predictions can be effectively estimated using the variance of the predictions or by k-nearest-neighbor models trained on the prediction errors. It is here investigated whether these methods are also effective in the context of split conformal predictive systems. Results from a large empirical study are presented, using 33 publicly available datasets. The results show that by using either variance or the k-nearest-neighbor method for estimating prediction quality, a significant increase in performance, as measured by the continuous ranked probability score, can be obtained compared to omitting the quality estimation. The results furthermore show that the use of out-of-bag examples for calibration is competitive with the most effective way of splitting training data into a proper training set and a calibration set, without requiring tuning of the calibration set size.

sted, utgiver, år, opplag, sider
ML Research Press , 2020. s. 134-150
Emneord [en]
Conformal predictive distributions, Quality estimation, Random forest, Regression, Split conformal predictive systems
HSV kategori
Identifikatorer
URN: urn:nbn:se:kth:diva-350335Scopus ID: 2-s2.0-85126598294OAI: oai:DiVA.org:kth-350335DiVA, id: diva2:1883763
Konferanse
9th Symposium on Conformal and Probabilistic Predictions with Applications, COPA 2020, Virtual, Online, Italy, Sep 9 2020 - Sep 11 2020
Merknad

QC 20240711

Tilgjengelig fra: 2024-07-11 Laget: 2024-07-11 Sist oppdatert: 2024-07-11bibliografisk kontrollert

Open Access i DiVA

Fulltekst mangler i DiVA

Scopus

Person

Werner, HugoBoström, Henrik

Søk i DiVA

Av forfatter/redaktør
Werner, HugoBoström, Henrik
Av organisasjonen

Søk utenfor DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric

urn-nbn
Totalt: 67 treff
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf