Time series data correspond to observations of phenomena that are recorded over time. Such data are encountered regularly in a wide range of applications, such as speech and music recognition, monitoring health and medical diagnosis, financial analysis, motion tracking, and shape identification, to name a few. With such a diversity of applications and the large variations in their characteristics, time series classification is a complex and challenging task. One of the fundamental steps in the design of time series classifiers is that of defining or constructing the discriminant features that help differentiate between classes. This is typically achieved by designing novel representation techniques that transform the raw time series data to a new data domain, where subsequently a classifier is trained on the transformed data, such as one-nearest neighbors or random forests. In recent time series classification approaches, deep neural network models have been employed that are able to jointly learn a representation of time series and perform classification. In many of these sophisticated approaches, the discriminant features tend to be complicated to analyze and interpret, given the high degree of nonlinearity.

This paper investigates the problem of controller identification given the data from a linear quadratic Model Predictive Controller (MPC) with constraints. We propose an approach for learning MPC that explicitly uses the gradient information in the training process. This is motivated by the observation that recent differentiable convex optimization MPC solvers can provide both the optimal feedback law from the state to control input as well as the corresponding gradient. As a proof of concept, we apply this approach to explicit MPC (eMPC), for which the feedback law is a piece-wise affine function of the state, but the number of pieces grows rapidly with the state dimension. Controller identification can here be used to find an approximate low complexity functional approximation of the controller. The eMPC is modelled using a Neural Network (NN) with Rectified Linear Units (ReLUs), since such NNs can represent any piece-wise affine function. A key motivation is to replace on-line solvers with neural networks to implement MPC and to simplify the evaluation of the function in larger input dimensions. We also study experimental design and model evaluation in this framework, and propose a hit-and-run sampling algorithm for input design. The proposed algorithms are illustrated and numerically evaluated on a second order MPC problem.

Kernel Regression over Graphs (KRG) was recently proposed for predicting graph signals in a supervised learning setting, where the inputs are agnostic to the graph. KRG model predicts targets that are smooth graph signals as over the given graph, given the input when all the signals are deterministic. In this work, we consider the development of a stochastic or Bayesian variant of KRG. Using priors and likelihood functions, our goal is to systematically derive a predictive distribution for the smooth graph signal target given the training data and a new input. We show that this naturally results in a Gaussian process formulation which we call Gaussian Processes over Graphs (GPG). Experiments with real-world datasets show that the performance of GPG is superior to a conventional Gaussian Process (without the graph-structure) for small training data sizes and under noisy training.

We design a rectified linear unit-based multilayer neural network by mapping the feature vectors to a higher dimensional space in every layer. We design the weight matrices in every layer to ensure a reduction of the training cost as the number of layers increases. Linear projection to the target in the higher dimensional space leads to a lower training cost if a convex cost is minimized. Anl(2)-norm convex constraint is used in the minimization to reduce the generalization error and avoid overfitting. The regularization hyperparameters of the network are derived analytically to guarantee a monotonic decrement of the training cost, and therefore, it eliminates the need for cross-validation to find the regularization hyperparameter in each layer. We show that the proposed architecture is norm-preserving and provides an invertible feature vector and, therefore, can be used to reduce the training cost of any other learning method which employs linear projection to estimate the target.

We design a ReLU-based multilayer neural network to generate a rich high-dimensional feature vector. The feature guarantees a monotonically decreasing training cost as the number of layers increases. We design the weight matrix in each layer to extend the feature vectors to a higher dimensional space while providing a richer representation in the sense of training cost. Linear projection to the target in the higher dimensional space leads to a lower training cost if a convex cost is minimized. An l(2)-norm convex constraint is used in the minimization to improve the generalization error and avoid overfitting. The regularization hyperparameters of the network are derived analytically to guarantee a monotonic decrement of the training cost and therefore, it eliminates the need for cross-validation to find the regularization hyperparameter in each layer.

We address the issue of estimating the topology and dynamics of sparse linear dynamic networks in a hyperparameter-free setting. We propose a method to estimate the network dynamics in a computationally efficient and parameter tuning-free iterative framework known as SPICE (Sparse Iterative Covariance Estimation). Our approach does not assume that the network is undirected and is applicable even with varying noise levels across the modules of the network. We also do not assume any explicit prior knowledge on the network dynamics. Numerical experiments with realistic dynamic networks illustrate the usefulness of our method. 

Kernel and linear regression have been recently explored in the prediction of graph signals as the output, given arbitrary input signals that are agnostic to the graph. In many real-world problems, the graph expands over time as new nodes get introduced. Keeping this premise in mind, we propose a method to recursively obtain the optimal prediction or regression coefficients for the recently proposed Linear Regression over Graphs (LRG), as the graph expands with incoming nodes. This comes as a natural consequence of the structure of the regression problem, and obviates the need to solve a new regression problem each time a new node is added. Experiments with real-world graph signals show that our approach results in a good prediction performance which tends to be close to that obtained from knowing the entire graph apriori.

In presence of sparse noise we propose kernel regression for predicting output vectors which are smooth over a given graph. Sparse noise models the training outputs being corrupted either with missing samples or large perturbations. The presence of sparse noise is handled using appropriate use of l(1)-norm along-with use of l(2)-norm in a convex cost function. For optimization of the cost function, we propose an iteratively reweighted least-squares (IRLS) approach that is suitable for kernel substitution or kernel trick due to availability of a closed form solution. Simulations using real-world temperature data show efficacy of our proposed method, mainly for limited-size training datasets.

We address the problem of prediction of multivariate data process using an underlying graph model. We develop a method that learns a sparse partial correlation graph in a tuning-free and computationally efficient manner. Specifically, the graph structure is learned recursively without the need for cross validation or parameter tuning by building upon a hyperparameter-free framework. Our approach does not require the graph to be undirected and also accommodates varying noise levels across different nodes. Experiments using real-world datasets show that the proposed method offers significant performance gains in prediction, in comparison with the graphs frequently associated with these datasets.

We propose Hilbert transform and analytic signal construction for signals over graphs. This is motivated by the popularity of Hilbert transform, analytic signal, and modulation analysis in conventional signal processing, and the observation that complementary insight is often obtained by viewing conventional signals in the graph setting. Our definitions of Hilbert transform and analytic signal use a conjugate symmetry-like property exhibited by the graph Fourier transform (GFT), resulting in a 'one-sided' spectrum for the graph analytic signal. The resulting graph Hilbert transform is shown to possess many interesting mathematical properties and also exhibit the ability to highlight anomalies/discontinuities in the graph signal and the nodes across which signal discontinuities occur. Using the graph analytic signal, we further define amplitude, phase, and frequency modulations for a graph signal. We illustrate the proposed concepts by showing applications to synthesized and real-world signals. For example, we show that the graph Hilbert transform can indicate presence of anomalies and that graph analytic signal, and associated amplitude and frequency modulations reveal complementary information in speech signals.