kth.sePublikationer KTH
Ändra sökning
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Evaluating Different Approaches to Calibrating Conformal Predictive Systems
KTH, Skolan för elektroteknik och datavetenskap (EECS), Datavetenskap, Programvaruteknik och datorsystem, SCS. Stena Line, Sweden.
Stena Line, Sweden; Centre for Reliable Machine Learning, University of London, UK.
Stena Line, Sweden; Department of Pharmaceutical Biosciences, Uppsala University, Uppsala, Sweden.
KTH, Skolan för elektroteknik och datavetenskap (EECS), Datavetenskap, Programvaruteknik och datorsystem, SCS.ORCID-id: 0000-0001-8382-0300
2020 (Engelska)Ingår i: Proceedings of the 9th Symposium on Conformal and Probabilistic Prediction and Applications, COPA 2020, ML Research Press , 2020, s. 134-150Konferensbidrag, Publicerat paper (Refereegranskat)
Abstract [en]

Conformal predictive systems (CPSs) provide probability distributions for real-valued labels of test examples, rather than point predictions (as output by regular regression models) or confidence intervals (as output by conformal regressors). The performance of a CPS is dependent on both the underlying model and the way in which the quality of its predictions is estimated; a stronger underlying model and a better quality estimation can significantly improve the performance. Recent studies have shown that conformal regressors that use random forests as the underlying model may benefit from using out-of-bag predictions for the calibration, rather than setting aside a separate calibration set, allowing for more data to be used for training and thereby improving the performance of the underlying model. These studies have furthermore shown that the quality of the individual predictions can be effectively estimated using the variance of the predictions or by k-nearest-neighbor models trained on the prediction errors. It is here investigated whether these methods are also effective in the context of split conformal predictive systems. Results from a large empirical study are presented, using 33 publicly available datasets. The results show that by using either variance or the k-nearest-neighbor method for estimating prediction quality, a significant increase in performance, as measured by the continuous ranked probability score, can be obtained compared to omitting the quality estimation. The results furthermore show that the use of out-of-bag examples for calibration is competitive with the most effective way of splitting training data into a proper training set and a calibration set, without requiring tuning of the calibration set size.

Ort, förlag, år, upplaga, sidor
ML Research Press , 2020. s. 134-150
Nyckelord [en]
Conformal predictive distributions, Quality estimation, Random forest, Regression, Split conformal predictive systems
Nationell ämneskategori
Data- och informationsvetenskap
Identifikatorer
URN: urn:nbn:se:kth:diva-350335Scopus ID: 2-s2.0-85126598294OAI: oai:DiVA.org:kth-350335DiVA, id: diva2:1883763
Konferens
9th Symposium on Conformal and Probabilistic Predictions with Applications, COPA 2020, Virtual, Online, Italy, Sep 9 2020 - Sep 11 2020
Anmärkning

QC 20240711

Tillgänglig från: 2024-07-11 Skapad: 2024-07-11 Senast uppdaterad: 2024-07-11Bibliografiskt granskad

Open Access i DiVA

Fulltext saknas i DiVA

Scopus

Person

Werner, HugoBoström, Henrik

Sök vidare i DiVA

Av författaren/redaktören
Werner, HugoBoström, Henrik
Av organisationen
Programvaruteknik och datorsystem, SCS
Data- och informationsvetenskap

Sök vidare utanför DiVA

GoogleGoogle Scholar

urn-nbn

Altmetricpoäng

urn-nbn
Totalt: 67 träffar
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf