kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Conditional Noise-Contrastive Estimation of Unnormalised Models
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Robotics, Perception and Learning, RPL. Rhein Westfal TH Aachen, UMIC, Aachen, Germany.; Univ Edinburgh, Edinburgh, Midlothian, Scotland..ORCID iD: 0000-0002-8044-4773
Univ Edinburgh, Sch Informat, Edinburgh, Midlothian, Scotland..
2018 (English)In: 35th International Conference on Machine Learning, ICML 2018 / [ed] Dy, J Krause, A, International Machine Learning Society (IMLS) , 2018, Vol. 80, p. 1334-1442Conference paper, Published paper (Refereed)
Abstract [en]

Many parametric statistical models are not properly normalised and only specified up to an intractable partition function, which renders parameter estimation difficult. Examples of unnormalised models are Gibbs distributions, Markov random fields, and neural network models in unsupervised deep learning. In previous work, the estimation principle called noise-contrastive estimation (NCE) was introduced where unnormalised models are estimated by learning to distinguish between data and auxiliary noise. An open question is how to best choose the auxiliary noise distribution. We here propose a new method that addresses this issue. The proposed method shares with NCE the idea of formulating density estimation as a supervised learning problem but in contrast to NCE, the proposed method leverages the observed data when generating noise samples. The noise can thus be generated in a semi-automated manner. We first present the underlying theory of the new method, show that score matching emerges as a limiting case, validate the method on continuous and discrete valued synthetic data, and show that we can expect an improved performance compared to NCE when the data lie in a lower-dimensional manifold. Then we demonstrate its applicability in unsupervised deep learning by estimating a four-layer neural image model.

Place, publisher, year, edition, pages
International Machine Learning Society (IMLS) , 2018. Vol. 80, p. 1334-1442
Series
Proceedings of Machine Learning Research, ISSN 2640-3498
National Category
Other Engineering and Technologies
Identifiers
URN: urn:nbn:se:kth:diva-318709ISI: 000683379200075Scopus ID: 2-s2.0-85057220926OAI: oai:DiVA.org:kth-318709DiVA, id: diva2:1698104
Conference
35th International Conference on Machine Learning (ICML), JUL 10-15, 2018, Stockholm, Sweden
Note

QC 20220922

Part of books: ISBN 978-151086796-3

Available from: 2022-09-22 Created: 2022-09-22 Last updated: 2023-09-22Bibliographically approved

Open Access in DiVA

No full text in DiVA

Scopus

Authority records

Ceylan, Ciwan

Search in DiVA

By author/editor
Ceylan, Ciwan
By organisation
Robotics, Perception and Learning, RPL
Other Engineering and Technologies

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 33 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf