This paper presents a reinforcement learning (RL) approach for evaluating and controlling residential building flexibility in response to dynamic electricity pricing and intermit-tent grid requests. To address the challenge of sparse rewards and fluctuating states, a curriculum learning (CL) strategy is integrated with reward shaping, enabling a model-free RL agent to learn complex, multi-objective control behavior. A custom Gymnasium-EnergyPlus environment is developed to simulate thermal dynamics, price signals, and USEF-aligned flexibility requests. The agent is trained through a three-stage curriculum, progressing from comfort maintenance to cost minimization, and finally, to flexibility response. Results show that CL significantly improves training stability and responsiveness compared to a non-curriculum approach, with agents able to balance comfort, cost efficiency, and flexibility provision under varying conditions.