Machining deformation control for large structural parts is an intractable problem, which is highly important for the dimensional accuracy and fatigue life of parts, and deformation prediction is the basis for deformation control. Existing prediction methods rely on the measurement of residual stress, which is limited by the measurement accuracy of residual stress distributed within thick blanks, and it is still a worldwide challenge. To address the above issue, this paper proposes a machining deformation prediction method based on estimation of latent filed for residual stress field using deformation force. The residual stress field is represented by latent field, which is estimated by deformation force monitoring data during the machining process based on the proposed physical-field estimation neural network. The estimated latent field is used to predict the subsequent deformation force and deformation via an inference network by combining the machining process information. The proposed method is verified by both simulation and actual environment, and it can provide a helpful reference for other machining related difficult-to-measure field.