The recurrent neural networks (RNN) with richly distributed internal states and flexible non-linear transition functions, have gradually overtaken the dynamic Bayesian networks in modeling highly structured sequential data. These data, which may come from speech and handwriting, often contain complex relationships between the underlying variational factors such as speaker characteristic and the observed data. The standard RNN model has very limited randomness or variability in its structure, which comes from the output conditional probability model. To improve the variability and performance, we study the new latent variable models with novel regularization methods. This paper will present different ways of using high level latent random variables in RNN to model the variability in the sequential data. We will explore possible ways of using adversarial methods to train a variational RNN model. Through theoretical analysis we show that, contrary to competing approaches our schemes are theoretical optimum in the model training and the symmetric objective function in the adversarial training provides better model training stability. Our approach also improves the posterior approximation in the variational inference network by a separated adversarial training step. Numerical results simulated from TIMIT speech data show that reconstruction loss and evidence lower bound converge to the same level and adversarial training loss converges in a stable course. The results also show our approach of regularization provides stability and smoothness on probability distribution distance minimization between prior and posterior of the latent variables.