The performance of sub-grid scale heat flux closure models is investigated in the context of large eddy simulation. The study is based on filtered direct numerical simulation data of a thermal channel flow for various Prandtl numbers, where temperature is transported as a passive scalar. Three-dimensional and two-dimensional filtering of the direct numerical simulation data is carried out based on a diffusion-based filtering method. The gradient diffusion hypothesis model and Clark's gradient model (CGM) are investigated in the a-priori analysis, representing a functional and a structural sub-grid scale model, respectively. The sub-grid scale models are compared to the exact sub-grid scale heat flux term by evaluating the alignment, correlation coefficient, and isotropy ratio for different filter widths. It can be observed that the generally good performance of CGM deteriorates from low to high Prandtl numbers. Regions where CGM shows deficits can be localized by a sub-grid activity sensor. In addition, the near-wall behavior of the sub-grid scale heat flux and CGM are investigated. The near-wall scaling of CGM also depends on the near-wall scaling of the filter width. The correct near-wall scaling of CGM can be achieved by a two-dimensional deconvolution formulation in the wall parallel directions.