Change search
ReferencesLink to record
Permanent link

Direct link
Kernel Density Estimation-Based Markov Models with Hidden State
KTH, School of Electrical Engineering (EES), Communication Theory. The University of Edinburgh, United Kingdom. (The Centre for Speech Technology Research)
KTH, School of Electrical Engineering (EES), Communication Theory.
KTH, School of Electrical Engineering (EES), Communication Theory. Victoria University of Wellington, New Zealand. (Communications and Signal Processing Group, School of Engineering and Computer Science)
(English)Manuscript (preprint) (Other academic)
Abstract [en]

We consider Markov models of stochastic processes where the next-step conditional distribution is defined by a kernel density estimator (KDE), similar to certain time-series bootstrap schemes from the economic forecasting literature. The KDE Markov models (KDE-MMs) we discuss are nonlinear, nonparametric, fully probabilistic representations of stationary processes with strong asymptotic convergence properties. The models generate new data simply by concatenating points from the training data sequences in a context-sensitive manner, with some added noise. We present novel EM-type maximum-likelihood algorithms for data-driven bandwidth selection in KDE-MMs. Additionally, we augment the KDE-MMs with a hidden state, yielding a new model class, KDE-HMMs. The added state-variable enables long-range memory and signal structure representation, complementing the short-range correlations captured by the Markov process. This is compelling for modelling complex real-world processes such as speech and language data. The paper presents guaranteed-ascent EM-update equations for model parameters in the case of Gaussian kernels, as well as relaxed update formulas that greatly accelerate training in practice. Experiments demonstrate increased held-out set probability for KDE-HMMs on several challenging natural and synthetic data series, compared to traditional techniques such as autoregressive models, HMMs, and their combinations.

Keyword [en]
hidden Markov models, nonparametric methods, kernel density estimation, autoregressive models, Markov forecast density
National Category
Probability Theory and Statistics
URN: urn:nbn:se:kth:diva-134692OAI: diva2:667672
LISTA – The Listening Talker
EU, FP7, Seventh Framework Programme, 256230

QS 2013

Available from: 2013-11-27 Created: 2013-11-27 Last updated: 2013-11-28Bibliographically approved
In thesis
1. Probabilistic Sequence Models with Speech and Language Applications
Open this publication in new window or tab >>Probabilistic Sequence Models with Speech and Language Applications
2013 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Series data, sequences of measured values, are ubiquitous. Whenever observations are made along a path in space or time, a data sequence results. To comprehend nature and shape it to our will, or to make informed decisions based on what we know, we need methods to make sense of such data. Of particular interest are probabilistic descriptions, which enable us to represent uncertainty and random variation inherent to the world around us.

This thesis presents and expands upon some tools for creating probabilistic models of sequences, with an eye towards applications involving speech and language. Modelling speech and language is not only of use for creating listening, reading, talking, and writing machines---for instance allowing human-friendly interfaces to future computational intelligences and smart devices of today---but probabilistic models may also ultimately tell us something about ourselves and the world we occupy.

The central theme of the thesis is the creation of new or improved models more appropriate for our intended applications, by weakening limiting and questionable assumptions made by standard modelling techniques. One contribution of this thesis examines causal-state splitting reconstruction (CSSR), an algorithm for learning discrete-valued sequence models whose states are minimal sufficient statistics for prediction. Unlike many traditional techniques, CSSR does not require the number of process states to be specified a priori, but builds a pattern vocabulary from data alone, making it applicable for language acquisition and the identification of stochastic grammars. A paper in the thesis shows that CSSR handles noise and errors expected in natural data poorly, but that the learner can be extended in a simple manner to yield more robust and stable results also in the presence of corruptions.

Even when the complexities of language are put aside, challenges remain. The seemingly simple task of accurately describing human speech signals, so that natural synthetic speech can be generated, has proved difficult, as humans are highly attuned to what speech should sound like. Two papers in the thesis therefore study nonparametric techniques suitable for improved acoustic modelling of speech for synthesis applications. Each of the two papers targets a known-incorrect assumption of established methods, based on the hypothesis that nonparametric techniques can better represent and recreate essential characteristics of natural speech.

In the first paper of the pair, Gaussian process dynamical models (GPDMs), nonlinear, continuous state-space dynamical models based on Gaussian processes, are shown to better replicate voiced speech, without traditional dynamical features or assumptions that cepstral parameters follow linear autoregressive processes. Additional dimensions of the state-space are able to represent other salient signal aspects such as prosodic variation. The second paper, meanwhile, introduces KDE-HMMs, asymptotically-consistent Markov models for continuous-valued data based on kernel density estimation, that additionally have been extended with a fixed-cardinality discrete hidden state. This construction is shown to provide improved probabilistic descriptions of nonlinear time series, compared to reference models from different paradigms. The hidden state can be used to control process output, making KDE-HMMs compelling as a probabilistic alternative to hybrid speech-synthesis approaches.

A final paper of the thesis discusses how models can be improved even when one is restricted to a fundamentally imperfect model class. Minimum entropy rate simplification (MERS), an information-theoretic scheme for postprocessing models for generative applications involving both speech and text, is introduced. MERS reduces the entropy rate of a model while remaining as close as possible to the starting model. This is shown to produce simplified models that concentrate on the most common and characteristic behaviours, and provides a continuum of simplifications between the original model and zero-entropy, completely predictable output. As the tails of fitted distributions may be inflated by noise or empirical variability that a model has failed to capture, MERS's ability to concentrate on high-probability output is also demonstrated to be useful for denoising models trained on disturbed data.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2013. xviii, 80 p.
Trita-EE, ISSN 1653-5146 ; 2013:042
Time series, acoustic modelling, speech synthesis, stochastic processes, causal-state splitting reconstruction, robust causal states, pattern discovery, Markov models, HMMs, nonparametric models, Gaussian processes, Gaussian process dynamical models, nonlinear Kalman filters, information theory, minimum entropy rate simplification, kernel density estimation, time-series bootstrap
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
urn:nbn:se:kth:diva-134693 (URN)978-91-7501-932-1 (ISBN)
Public defence
2013-12-17, D3, Lindstedtsvägen 5, KTH, Stockholm, 09:00 (English)
ACORNS: Acquisition of Communication and Recognition SkillsLISTA – The Listening Talker
EU, European Research Council, FP6-034362EU, FP7, Seventh Framework Programme, 256230

QC 20131128

Available from: 2013-11-28 Created: 2013-11-27 Last updated: 2013-11-28Bibliographically approved

Open Access in DiVA

No full text

Search in DiVA

By author/editor
Henter, Gustav EjeLeijon, ArneKleijn, W. Bastiaan
By organisation
Communication Theory
Probability Theory and Statistics

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 74 hits
ReferencesLink to record
Permanent link

Direct link