Anneal hardening has been commonly observed in single-phase solid solutions, including face-centered cubic (FCC) alloys containing transitional-metal elements. However, the underlying mechanisms governing this effect have remained unclear due to a lack of direct evidence. In this study, we utilize multi-scale in-situ characterizations to thoroughly investigate the microstructural evolution during annealing of an MP35 N (Co35Ni35Cr24Mo6, at %) alloy. Our findings reveal negligible differences in the crystal structure, grain boundary (GB) character, and dislocation structure before and after annealing at 550 °C. However, in-situ transmission electron microscopy heating experiments and atomic-resolution energy-dispersive spectroscopy mappings disclose that the 550 °C annealing promotes nanoscale segregation of Mo into GBs, driven by the reduced GB energy. These segregated Mo atoms engage in strong charge exchanges with neighboring atoms, enhancing the GB's cohesive strength and improving the resistance to dislocation motion due to the increased strain field near the GBs. Consequently, the GB strengthening effect is enhanced, leading to significant anneal hardening in the fine-grained sample (3.2 μm) with Mo segregation, while no hardening is observed in the coarse-grained sample (202.2 μm) lacking Mo segregation. Furthermore, we demonstrate that annealing at higher temperatures triggers an interfacial phase transition from the FCC to a μ phase through spinodal decomposition accompanied by significant dislocation recovery, which paradoxically weakens the anneal hardening effect. These findings provide deeper insights into the anneal hardening phenomena and offer valuable guidance for optimizing cold working and heat treatment processes in the further development of high-performance structural alloys.