We derive a new formulation of nonanticipative rate distortion function (NRDF) for time-varying multivariate Gauss-Markov processes driven by correlated noise described by a first order autoregressive moving average (ARMA(1,1)) process with mean-squared error (MSE) distortion constraint. To arrive to this formulation, we first show that the Gauss-Markov process with correlated noise can be compactly written as a linear functional of the lagged by one sample of the sufficient statistic of the correlated noise and its orthogonal innovations process. Then, we use this structural result to a general low delay quantization problem where we choose to design the encoder and the decoder policies of a multi-input multi-output (MIMO) system with given the past sample of the sufficient statistic of the correlated noise process. For jointly Gaussian processes, we find the minimization problem that needs to be solved, obtain its optimal realization and solve it by showing that is semidefinite representable. Interestingly, the optimal realization of this problem reveals that it suffices only the decoder to have access to the given past sufficient statistic of the correlated noise process but not necessarily the encoder. For scalar-valued processes, we also derive a new analytical expression. The generality of our results (both for vector and scalar processes) is shown by recovering various special cases and known results obtained for independent noise processes.