Surface roughness plays an indispensable and fundamental role as a leading indicator of the surface quality of machined parts in the manufacturing process. The precise and effective monitoring and prediction of surface roughness is crucial for surface quality control. In this regard, the development of an in-process surface quality monitoring system is necessary, which has the promising potential to achieve this goal. Such a system typically comprises data-driven models for decision-making and sensing techniques for detecting associated process information. However, some challenges still exist in building such systems. Firstly, the architecture design and deployment of data-driven models, specifically deep learning (DL)-based models, demand adequate domain knowledge. Secondly, most models trained on specific tasks with limited datasets are prone to suppressing their versatility and generalization across different machining conditions. Additionally, in most cases, reliance on handcrafted features to represent dynamic information on various signals during model training necessitates extensive expertise in selecting appropriate feature types. Furthermore, due to the nature of their low dimensionality, handcrafted features have difficulty in capturing of overall process-related underlying patterns from dynamics signatures, which is time-varying and often occurs in transient events. To address these challenges, this paper proposes the regression-based pre-trained convolutional neural network (pre-trained CNN) combined with Mel-spectrogram images based on the transfer learning method for surface roughness prediction. Within the context, the architecture of the transfer model is slightly adapted from already well-trained CNNs. Initial weights in each layer of the CNN model are directly inherited and then fine-tuned through the Bayesian optimization tuning method. Besides, the audible sound signals are captured and subsequently converted into 2D Mel-spectrogram images with variant time lengths, which are separately engaged to retrain and validate four existing pre-trained CNN models (VGG16, VGG19, ResNet50V2 and InceptionResNetV2). Eventually, the effectiveness of proposed models and comparison of their predictive capabilities are further validated through a case study in the turning process. The results demonstrate that each applied pre-trained CNN model is capable of effectively predicting surface quality with satisfactory prediction results. Therefore, the proposed method can facilitate the establishment of a machining monitoring system concerning its accuracy, reliability, and robustness.