Traffic state predictions are critical for the traffic management and control of transport systems. This study introduces an innovative contrastive learning framework coupled with a transformer architecture for spatiotemporal traffic state prediction, designed to capture the spatio-temporal heterogeneity inherent in traffic. The transformer structure functions as the upper level of the prediction framework to minimize the prediction errors between the input and predicted output. Based on the self-supervised contrastive learning, the lower level in the framework is proposed to discern the spatio-temporal heterogeneity and embed the latent characteristic of traffic flow by regenerating the augmentation features. Then, a soft clustering problem is applied between the upper level and lower level to category the types of traffic flow characteristics by minimizing the joint loss across each cluster. Subsequently, the proposed model is evaluated through a real-world highway traffic flow dataset for bench marking against several latest existing models. The experimental results affirm that the proposed model considerably enhances traffic state prediction accuracy. In terms of precision metrics, the model records a Mean Absolute Error of 13.31 and a Mean Absolute Percentage Error of 7.85%, reflecting marked improvements of 2.0% and 14.5% respectively over the latest and most competitive baseline model. Furthermore, the analysis reveals that capacity of the proposed method to learn the cluster patterns of spatio-temporal traffic dynamics reflected by calibrated fundamental diagrams.