This study explores the application of Time Series GAN in a Programmable Data Plane (PDP) for enhancing Reinforcement Learning within the context of computer networks, particularly in video applications. We address various challenges, including dataset augmentation, balancing, and extended RL training times in real setups. By leveraging synthetic data generated by TimeGAN, we accelerate experimentation, enhance dataset diversity, and simplify RL model training, ultimately evaluating TimeGAN's performance against real setups in resource optimization for PDPs using an RL agent. This research contributes by directly comparing GAN usage and real setups, bridging a gap in computer network literature, and highlighting a 99% similarity in Quality of Service achieved by an RL model trained with synthetic data, affirming TimeGAN's potential as a valuable simulator without compromising RL training efficacy.