Open this publication in new window or tab >>Show others...
2019 (English)In: Foundations of Data Science, E-ISSN 2639-8001, Vol. 0, no 0, p. 0-0Article in journal (Refereed) Published
Abstract [en]
The supervised learning problem todetermine a neural network approximation $\mathbb{R}^d\ni x\mapsto\sum_{k=1}^K\hat\beta_k e^{{\mathrm{i}}\omega_k\cdot x}$with one hidden layer is studied asa random Fourier features algorithm. The Fourier features, i.e., the frequencies $\omega_k\in\mathbb{R}^d$,are sampled using an adaptive Metropolis sampler.The Metropolis test accepts proposal frequencies $\omega_k'$, having corresponding amplitudes $\hat\beta_k'$, with the probability$\min\big\{1, (|\hat\beta_k'|/|\hat\beta_k|)^\gamma\big\}$,for a certain positive parameter $\gamma$, determined by minimizing the approximation error for given computational work.This adaptive, non-parametric stochastic method leads asymptotically, as $K\to\infty$, to equidistributed amplitudes $|\hat\beta_k|$, analogous to deterministic adaptive algorithms for differential equations. The equidistributed amplitudes are shown to asymptotically correspond to the optimal density for independent samples in random Fourier features methods.Numerical evidence is provided in order to demonstrate the approximation properties and efficiency of the proposed algorithm. The algorithm is testedboth on synthetic data and a real-world high-dimensional benchmark.
Place, publisher, year, edition, pages
American Institute of Mathematical Sciences, 2019
Keywords
Random Fourier features, neural networks, Metropolis algorithm, stochastich gradient descent
National Category
Computational Mathematics Probability Theory and Statistics
Research subject
Applied and Computational Mathematics, Numerical Analysis
Identifiers
urn:nbn:se:kth:diva-287767 (URN)10.3934/fods.2020014 (DOI)000663367000004 ()2-s2.0-85098437855 (Scopus ID)
Funder
Swedish Research Council, 2019-03725
Note
QC 20201221
2020-12-172020-12-172023-06-08Bibliographically approved