kth.sePublications KTH
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Adaptive random fourier features with metropolis sampling
KTH, School of Engineering Sciences (SCI), Mathematics (Dept.), Numerical Analysis, NA.ORCID iD: 0000-0002-8458-0852
H-Ai AB, Grevgatan 23,114 53, Stockholm, Sweden.
Department of Mathematical Sciences, University of Delaware, Newark, DE 19716, USA.
KTH, School of Engineering Sciences (SCI), Mathematics (Dept.), Numerical Analysis, NA.ORCID iD: 0000-0003-2669-359X
Show others and affiliations
2019 (English)In: Foundations of Data Science, E-ISSN 2639-8001, Vol. 0, no 0, p. 0-0Article in journal (Refereed) Published
Abstract [en]

The supervised learning problem todetermine a neural network approximation $\mathbb{R}^d\ni x\mapsto\sum_{k=1}^K\hat\beta_k e^{{\mathrm{i}}\omega_k\cdot x}$with one hidden layer is studied asa random Fourier features algorithm.  The Fourier features, i.e., the frequencies $\omega_k\in\mathbb{R}^d$,are sampled using an adaptive Metropolis sampler.The Metropolis test accepts proposal frequencies $\omega_k'$, having corresponding amplitudes $\hat\beta_k'$, with the probability$\min\big\{1, (|\hat\beta_k'|/|\hat\beta_k|)^\gamma\big\}$,for a certain positive parameter $\gamma$, determined by minimizing the approximation error for given computational work.This adaptive, non-parametric stochastic method leads asymptotically, as $K\to\infty$, to equidistributed amplitudes $|\hat\beta_k|$, analogous  to deterministic adaptive algorithms for differential equations. The equidistributed amplitudes are shown to asymptotically correspond to the optimal density for independent samples in random Fourier features methods.Numerical evidence is provided in order to demonstrate the approximation properties and efficiency of the proposed algorithm. The algorithm is testedboth on synthetic data and a real-world high-dimensional benchmark.

Place, publisher, year, edition, pages
American Institute of Mathematical Sciences, 2019. Vol. 0, no 0, p. 0-0
Keywords [en]
Random Fourier features, neural networks, Metropolis algorithm, stochastich gradient descent
National Category
Computational Mathematics Probability Theory and Statistics
Research subject
Applied and Computational Mathematics, Numerical Analysis
Identifiers
URN: urn:nbn:se:kth:diva-287767DOI: 10.3934/fods.2020014ISI: 000663367000004Scopus ID: 2-s2.0-85098437855OAI: oai:DiVA.org:kth-287767DiVA, id: diva2:1511207
Funder
Swedish Research Council, 2019-03725
Note

QC 20201221

Available from: 2020-12-17 Created: 2020-12-17 Last updated: 2023-06-08Bibliographically approved
In thesis
1. Numerical algorithms for high dimensional integration with application to machine learning and molecular dynamics
Open this publication in new window or tab >>Numerical algorithms for high dimensional integration with application to machine learning and molecular dynamics
2021 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

This thesis contains results on high dimensional integration with two papers, paper I and paper II, presenting applications in machine learning and two papers, paper III and paper IV, presenting applications to molecular dynamics.

In paper I we present algorithms based on a Metropolis test for training shallow neural networks with trigonometric activation functions. Numerical experiments are performed on both synthetic and real data. The trigonometric activation function gives access to the Fourier transform and its inverse transform. The algorithms gives equidistributed amplitudes.

In paper II we derive smaller generalization error for deep residual neural networks compared to shallow ones. An algorithm that builds the residual neural network layer by layer based on an algorithm from paper I is presented both as a stand alone algorithm as well as a pre-step for a global optimizer like Stochastic gradient descent or Adam. Numerical test are performed with promising results.

In paper III we make use of the semiclassical Weyl law to show that canonical quantum observables can be approximated by molecular dynamics with an error rate proportional to the electron-nuclei mass ratio. Numerical experiments are presented that confirms the expected theoretical result.

In paper IV we consider canonical ensembles of molecular systems. We propose four numerical algorithms for efficient computation of the canonical ensemble molecular dynamics observables. The four algorithms can each be efficient in different situations. For example in low temperatures we can make use of the fact that the lowest electron energy levels contributes most to the observable. The work is an extension of the results in paper III.

Abstract [sv]

Den här avhandlingen innehåller resultat inom högdimensionell integration med två rapporter, rapport I och rapport II, som presenterarapplikationer inom maskininlärning och två rapporter, rapport III ochrapport IV, som presenterar applikationer inom molekyldynamik.I rapport I presenterar vi algoritmer baserade på ett Metropolistest för träning av grunda neurala nätverk med trigonometriska aktiveringsfunktioner. Numeriska experiment utförs på både syntetiskoch riktig data. Den trigonometriska aktiveringsfunktionen ger tillgångtill Fouriertransformen och dess inverstransform. Algoritmerna ger likafördelade amplituder.I rapport II härleds mindre generaliseringsfel för djupa residualnäti jämförelse med grunda. En algoritm som bygger residualnät lagerför lager baserat på en algoritm från rapport I presenteras både somen fristående algoritm såväl som ett försteg till en global optimerareså som Stochastic gradient descent eller Adam. Numeriska test utförsmed lovande resultat.I rapport III använder vi Weyls semiklassiska lag för att visa att kanoniska kvantobservabler kan approximeras med molekyldynamik medett fel som är proportionellt mot massförhållandet mellan elektroneroch atomkärnor. Numeriska experiment presenteras som bekräftar detförväntade teoretiska resultatet.I rapport IV betraktar vi kanoniska ensembler av molekylära system. Vi föreslår fyra numeriska algoritmer för effektiv beräkning avmolekyldynamikobservabler i den kanonisk ensemblen. De fyra algoritmerna kan var och en vara effektiva i olika situationer. Till exempel vidlåga temperaturer kan vi använda det faktum att de lägsta elektronenerginivåerna bidrar mest till observablerna. Arbetet är en utvidgningav resultaten i rapport III.

Place, publisher, year, edition, pages
KTH Royal Institute of Technology, 2021. p. 51
Series
TRITA-SCI-FOU ; 2020:38
National Category
Computational Mathematics
Research subject
Applied and Computational Mathematics, Numerical Analysis
Identifiers
urn:nbn:se:kth:diva-287773 (URN)978-91-7873-718-5 (ISBN)
Public defence
2021-01-29, Via Zoom https://kth-se.zoom.us/webinar/register/WN_iey9KBBkTIuZCx03HbQ3rQ, Stockholm, 14:00 (English)
Opponent
Supervisors
Available from: 2021-01-08 Created: 2020-12-18 Last updated: 2022-12-12Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopushttp://dx.doi.org/10.3934/fods.2020014

Authority records

Kammonen, AkuSandberg, MattiasSzepessy, Anders

Search in DiVA

By author/editor
Kammonen, AkuSandberg, MattiasSzepessy, Anders
By organisation
Numerical Analysis, NA
Computational MathematicsProbability Theory and Statistics

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 155 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf