Power systems are experiencing a decrease of synchronous generation along with increased penetration of inverter based renewable generation leading to reduced system inertia and a need for flexible resources. Non-generating resources such as thermostatically controlled loads (TCLs) are flexible due to their thermal energy storage capacity. When aggregated, TCLs can arbitrage energy prices and provide reserves to the power system. We approach the operational flexibility of the TCLs by modeling a risk-averse aggregator that controls decentralized TCLs and aims to maximize its own profit. The high number and low power rating of residential TCLs makes it difficult to model and assess their flexibility potential on national level. Thus, we make use of a high-level thermal energy storage model for aggregations of TCLs to quantify their flexibility potential. We present a method to aggregate temperature, TCL parameters, and building stock data into a thermal battery equivalent. We propose a multi-period multi-market multi-zonal two-stage chance constrained rolling horizon optimization problem formulation for the risk-averse day-ahead self-scheduling problem of a price-taker TCL aggregator bidding in energy and reserve markets under uncertainty and recast the problem as a linear program. We perform several case studies in the Swedish power system based on a survey of single -and two-family dwellings with electric heating and assess the flexibility potential. Additionally, a sensitivity analysis provides insights regarding market design and policy implications.