The residual stress field of structural components significantly influences their comprehensive performance and service life. Due to the lack of effective representation means and inference methods, existing methods are confined to inspecting local residual stress rather than the entire residual stress field, rendering the inference of complex residual stress fields quite difficult. In response to the challenges associated with the requirement for extensive sets of deformation force data from the current workpiece and the inherent difficulty in establishing a stable relationship between deformation forces and residual stress fields, this paper introduces a novel inference method of residual stress field is proposed based on a data-causal knowledge fusion model, where causal knowledge is introduced to eliminate the coupling effect of geometric change on residual stress, which can make up the drawback of pure data driven model. The proposed approach can accurately inference the residual stress within the workpieces, which provides an important basis for deformation control and part property improvement.