Modeling the relationship of properties with composition, process, and microstructure is important to designing and developing new steel products. As traditional Machine Learning (ML) relies only on digital data, it is incapable of treating multimodal information. In this paper, a Deep Learning (DL) method is proposed to predict mechanical properties of High-Strength Low-Alloy (HSLA) steels, in which both microstructural evolution during hot rolling and transformations during cooling are taken into account. Continuous Cooling Transformation (CCT) diagrams are generated based on hot rolling parameters and compositions and superimposed with Cooling Path (CP) curves to represent the dynamic changes of transformed products, which is perceived and processed by the Convolutional Neural Network (CNN) as inputs. By doing so, the multimodal model for predicting mechanical properties of high-grade pipeline steels was developed, which demonstrates superior prediction accuracy and stability over traditional data-driven ML models. Also, reverse visualization is performed to work out hotspots in cooling processes, which clearly demonstrates the interpretability of the DL model. This framework provides useful guidance for designing production routes of HSLA steels and can also be implemented for other high-strength steels.