In this paper, we propose a method for dynamic systems to operate over homomorphically encrypted data for an infinite time horizon, where we do not make use of reset, re-encryption, or bootstrapping for the encrypted messages. The given system is first decomposed into the stable part and the anti-stable part. Then, the stable part is approximated to have finite impulse response, and by a novel conversion scheme, the eigenvalues of the state matrix of the anti-stable part are approximated to algebraic integers. This allows that the given system can be implemented to operate for an infinite time horizon using only addition and multiplication over encrypted data, without re-encrypting any portion of data. The performance error caused by the approximation and quantization can be made arbitrarily small, with appropriate choice of parameters.