While methods for estimating the entropy production rate of a stationary process are relatively well established, this is still a challenge in non-stationary conditions. Here, the authors propose a scheme to infer the exact value of the time-dependent entropy production rate as well as entropy production along with single realizations directly from trajectory data. The rate of entropy production provides a useful quantitative measure of a non-equilibrium system and estimating it directly from time-series data from experiments is highly desirable. Several approaches have been considered for stationary dynamics, some of which are based on a variational characterization of the entropy production rate. However, the issue of obtaining it in the case of non-stationary dynamics remains largely unexplored. Here, we solve this open problem by demonstrating that the variational approaches can be generalized to give the exact value of the entropy production rate even for non-stationary dynamics. On the basis of this result, we develop an efficient algorithm that estimates the entropy production rate continuously in time by using machine learning techniques and validate our numerical estimates using analytically tractable Langevin models in experimentally relevant parameter regimes. Our method only requires time-series data for the system of interest without any prior knowledge of the system's parameters.