Asphalt concrete (AC) shows significant tension-compression (TC) asymmetry, i.e., different properties in tension and compression (T&C). This asymmetry may profoundly affect AC's performance and deterioration in the field, but limited studies have been performed to quantify this behavior. This study aims to quantitively characterize the global and local mechanical responses of AC in T&C through numerical modeling. To this end, three AC mixtures: the gap-graded SMA10, dense-graded AC20, and open-graded mixtures PA13, were evaluated experimentally and numerically. Digital image processing was used to generate image-based AC models with contact regions (CR), and dynamic simulations were conducted using the steady-state dynamics (SSD) approach. The results indicated that the measured and predicted master curves for AC in T&C qualitatively agree and demonstrate significant asymmetry, with higher moduli but lower phase angles in compression compared to tension. Among the mixtures, PA13 exhibited the most pronounced asymmetry, followed by SMA10 and AC20. Statistical analyses of local stress and strain found that the stress and strain in different phases show significant variations, with more pronounced disparities observed at lower frequencies. Notably, at 10−6 Hz for PA13 in compression, the stress within the aggregate phase exceeded that of the matrix phase by over 250 times, while the strain within the matrix phase surpassed the aggregate phase by more than 600 times. To enhance pavement durability, it is recommended to consider AC's TC asymmetry in pavement design.