While base station (BS) sleeping has long been recognized as an effective energy-saving approach, its real-world adoption remains very limited due to concerns over service quality degradation. In this paper, we address this challenge by integrating on-site BS batteries into the sleep scheduling process. Specifically, our reinforcement learning (RL)-based framework intelligently synchronizes battery usage with dynamic electricity prices and traffic fluctuations, enabling deeper sleep opportunities without compromising critical key performance indicators (KPIs). We further propose a complementary charging strategy that leverages lower-cost energy periods, thereby reducing operational expenses. Through extensive simulations using real-world traffic traces, we show that our solution leads to notable cost savings while incurring only negligible increases in service delay. Our results highlight the substantial promise of battery-powered BSs in bridging the gap between energy efficiency and service quality for sustainable and cost-effective mobile networks.