Large-scale data generation, acquisition, and processing are happening at everymoment in our society. This thesis explores the opportunities for applying lossycompression methods and concepts for improving information engineering techniquesthat transform a large amount of collected data into useful applications. Two specificapplications are investigated: data identification and representation learning.The lossy compression methods, such as product quantization and hierarchicalvector quantization, can be used to build the data structures for efficient retrievaland identification. There exists a trade-off between the rate of compressed data andgeneral retrieval performance. This thesis focus on studying this trade-off under thesimilarity identification framework. The similarity identification task is to identifythe items in a database that are similar to a given query item for a given metric. Thisthesis studies the trade-off between the rate and identifiability of the compresseddata for correlated query and source signals. Signal processing methods such asKarhunen-Loève transform and linear prediction are applied to exploit the lineardependence of the source and query signals. In addition, practical schemes basedon tree-structured vector quantizers and transform-based models are proposed forsimilarity identification.Representation learning aims to transform the real-world observations into an-other feature space that is more amiable to particular applications. Ideally, thelearned representation should only contain essential information of the original data,and irrelevant information should be removed. This thesis focuses on integratingdeep learning models with lossy compression methods and concepts to improverepresentation learning. For learning representation for large-scale image retrieval,the product quantizer is incorporated into the bottleneck stage of autoencodermodels and trained in an end-to-end fashion for image retrieval tasks. The trainedencoder neural network concatenated with a product quantizer is then used toproduce short indices for each image for fast retrieval. For improving unsuper-vised representation learning, a quantization-based regularizer is introduced tothe autoencoder-based models for fostering a similarity-preserving mapping at theencoder. It is demonstrated that the proposed regularization method results inimproved latent representations for downstream tasks such as classification andclustering. Finally, a contrastive loss based on conditional mutual information (CMI)for learning representations of time series data is proposed. An encoder is firsttrained to maximize the mutual information between the latent variables and thetrend information conditioned on the encoded observed variables. Then the featuresextracted from the trained encoder are used to learn a subsequent logistic regressionmodel for predicting time series movements. The CMI maximization problem canbe transformed into a classification problem of determining whether two encodedivrepresentations are sampled from the same class or not. It is shown that the proposedmethod is effective for improving the generalization of deep learning models thatare trained on limited data.

We present a representation learning framework for financial time series forecasting. One challenge of using deep learning models for finance forecasting is the shortage of available training data when using small datasets. Direct trend classification using deep neural networks trained on small datasets is susceptible to the overfitting problem. In this paper, we propose to first learn compact representations from time series data, then use the learned representations to train a simpler model for predicting time series movements. We consider a class-conditioned latent variable model. We train an encoder network to maximize the mutual information between the latent variables and the trend information conditioned on the encoded observed variables. We show that conditional mutual information maximization can be approximated by a contrastive loss. Then, the problem is transformed into a classification task of determining whether two encoded representations are sampled from the same class or not. This is equivalent to performing pairwise comparisons of the training datapoints, and thus, improves the generalization ability of the encoder network. We use deep autoregressive models as our encoder to capture long-term dependencies of the sequence data. Empirical experiments indicate that our proposed method has the potential to advance state-of-the-art performance.

Autoencoders and their variations provide unsupervised models for learning low-dimensional representations for downstream tasks. Without proper regularization, autoencoder models are susceptible to the overfitting problem and the so-called posterior collapse phenomenon. In this paper, we introduce a quantization-based regularizer in the bottleneck stage of autoencoder models to learn meaningful latent representations. We combine both perspectives of Vector Quantized-Variational AutoEncoders (VQ-VAE) and classical denoising regularization methods of neural networks. We interpret quantizers as regularizers that constrain latent representations while fostering a similarity-preserving mapping at the encoder. Before quantization, we impose noise on the latent codes and use a Bayesian estimator to optimize the quantizer-based representation. The introduced bottleneck Bayesian estimator outputs the posterior mean of the centroids to the decoder, and thus, is performing soft quantization of the noisy latent codes. We show that our proposed regularization method results in improved latent representations for both supervised learning and clustering downstream tasks when compared to autoencoders using other bottleneck structures.

Vector-Quantized Variational Autoencoders (VQ-VAE)[1] provide an unsupervised model for learning discrete representations by combining vector quantization and autoencoders. In this paper, we study the use of VQ-VAE for representation learning of downstream tasks, such as image retrieval. First, we describe the VQ-VAE in the context of an information-theoretic framework. Then, we show that the regularization effect on the learned representation is determined by the size of the embedded codebook before the training. As a result, we introduce a hyperparameter to balance the strength of the vector quantizer and the reconstruction error. By tuning the hyperparameter, the embedded bottleneck quantizer is used as a regularizer that forces the output of the encoder to share a constrained coding space. With that, the learned latent features better preserve the similarity relations of the data space. Finally, we incorporate the product quantizer into the bottleneck stage of VQ-VAE and use it as an end-to-end unsupervised learning model for image retrieval tasks. The product quantizer has the advantage of generating large and structured codebooks. Fast retrieval can be achieved by using lookup tables that store the distance between any pair of sub-codewords. State-of-the-art retrieval results are achieved by the proposed codebooks. 

This paper considers the problem of compression for similarity identification. Unlike classic compression problems, the focus is not on reconstructing the original data. Instead, compression is determined by the reliability of answering given queries. The problem is characterized by the identification rate of a source which is the minimum compression rate which allows reliable answers for a given similarity threshold. In this work, we investigate the component-based quadratic similarity identification for multivariate Gaussian sources. The decorrelated original data is processed by a distinct D- A dmissible system for each component. For a special case, we characterize the component-based identification rate for a correlated Gaussian source. Furthermore, we derived the optimal bit allocation for a given total rate constraint.

Among many current data processing systems, the objectives are often not the reproduction of data, but to compute some answers based on the data responding to sonic queries. The similarity identification task is to identify the items in a database which are similar to a given query item regarding to a certain metric. The problem of compression for similarity identification has been studied in [1]. Unlike classic compression problems, the focus is not on reconstructing the original data. Instead, the compression rate is determined by the desired reliability of the answers. Specifically, the information measure identification rate of a compression scheme characterizes the minimum compression rate that can be achieved which guarantees reliable answers with respect to a given similarity threshold. In this paper, we study the component-based quadratic similarity identification for correlated sources. The blocks are first decorrelated by Karhunen-Loeve transform. Then, the decorrelated data is processed by a distinct D-admissible system for each component. We derive the identification rate of component-based scheme for block correlated Gaussian sources. In addition, we characterize the identification rate of a special setting where any information regarding to the component similarity thresholds is unknown while only the similarity threshold of the whole scheme is given. Furthermore, we prove that block-correlated Gaussian sources are the "most difficult" to code under the special setting.

The task of similarity identification is to identify items in a database which are similar to a given query item for a given metric. The identification rate of a compression scheme characterizes the minimum rate that can be achieved which guarantees reliable answers with respect to a given similarity threshold [1]. In this paper, we study a prediction-based quadratic similarity identification for autoregressive processes. We use an ideal linear predictor to remove linear dependencies in autoregressive processes. The similarity identification is conducted on the residuals. We show that the relation between the distortion of query and database processes and the distortion of their residuals is characterized by a sequence of eigenvalues. We derive the identification rate of our prediction-based approach for autoregressive Gaussian processes. We characterize the identification rate for the special case where only the smallest value in the sequence of eigenvalues is required to be known and derive its analytical upper bound by approximating a sequence of matrices with a sequence of Toeplitz matrices.

This paper considers the problem of compression for similarity queries [1] and discusses transform-based compression schemes. Here, the focus is on the tradeoff between the rate of the compressed data and the reliability of the answers to a given query. We consider compression schemes that do not allow false negatives when answering queries. Hence, classical compression techniques need to be modified. We propose transform-based compression schemes which decorrelate original data and regard each transform component as a distinct D-admissible system. Both compression and retrieval will be performed in the transform domain. The transform-based schemes show advantages in terms of encoding speed and the ability of handling high-dimensional correlated data. In particular, we discuss component-based and vector-based schemes. We use P{maybe}, a probability that is related to the occurrence of false positives to assess our scheme. Our experiments show that component-based schemes offer both good performance and low search complexity.

The scoring function is a central component in mobile visual search. In this paper, we propose an embedded 3D geometry score for mobile 3D visual search (M3DVS). In contrast to conventional mobile visual search, M3DVS uses not only the visual appearance of query objects, but utilizes also the underlying 3D geometry. The proposed scoring function interprets visual search as a process that reduces uncertainty among candidate objects when observing a query. For M3DVS, the uncertainty is reduced by both appearance-based visual similarity and 3D geometric similarity. For the latter, we give an algorithm for estimating the query-dependent threshold for geometric similarity. In contrast to visual similarity, the threshold for geometric similarity is relative due to the constraints of image-based 3D reconstruction. The experimental results show that the embedded 3D geometry score improves the recall-datarate performance when compared to a conventional visual score or 3D geometry-based re-ranking.