On the Estimation of Differential Entropy from Data Located on Embedded Manifolds
2007 (English)In: IEEE Transactions on Information Theory, ISSN 0018-9448, Vol. 53, no 7, 2330-2341 p.Article in journal (Refereed) Published
Estimation of the differential entropy from observations of a random variable is of great importance for a wide range of signal processing applications such as source coding, pattern recognition, hypothesis testing, and blind source separation. In this paper, we present a method for estimation of the Shannon differential entropy that accounts for embedded manifolds. The method is based on high-rate quantization theory and forms an extension of the classical nearest-neighbor entropy estimator. The estimator is consistent in the mean square sense and an upper bound on the rate of convergence of the estimator is given. Because of the close connection between compression and Shannon entropy, the proposed method has an advantage over methods estimating the Renyi entropy. Through experiments on uniformly distributed data on known manifolds and real-world speech data we show the accuracy and usefulness of our proposed method.
Place, publisher, year, edition, pages
2007. Vol. 53, no 7, 2330-2341 p.
convergence rate, manifolds, nearest-neighbor distance, Shannon differential entropy
Fluid Mechanics and Acoustics
IdentifiersURN: urn:nbn:se:kth:diva-5787DOI: 10.1109/TIT.2007.899533ISI: 000247606300002ScopusID: 2-s2.0-34447316139OAI: oai:DiVA.org:kth-5787DiVA: diva2:10282
QC 201009142006-05-232006-05-232010-09-14Bibliographically approved