Digital Twin (DT) of a manufacturing system mainly involving materials and machines has been widely explored in the past decades to facilitate the mass customization of modern products. Recently, the new vision of Industry 5.0 has brought human operators back to the core part of work cells. To this end, designing human-centric DT systems is vital for an ergonomic and symbiotic working environment. However, one major challenge is the construction and utilization of high-fidelity digital human models. In the literature, preset universal human avatar models such as skeletons are mostly employed to represent the human operators, which overlooks the individual differences of physical traits. Besides, the fundamental utilization features such as motion tracking and procedure recognition still do not well address the practical issues such as occlusions and incomplete observations. To deal with the challenge, this paper proposes a systematic design framework to quickly and precisely build and utilize the human-centric DT systems. The mesh-based customized human operator models with rendered appearances are first generated within one minute from a short motion video. Then transformer-based deep learning networks are developed to realize the motion-related operator status synchronization in complex conditions. Extensive experiments on multiple real-world human–robot collaborative work cells show the superior performance of the proposed framework over the state-of-the-art.