The shear strength of rock fractures serves as a crucial control on the strength and deformation behavior of engineering rock masses. To reduce the uncertainties in the shear strength evaluation, a hybrid machine learning model (GS-SVR model) of the support vector regression (SVR) underpinned by the grid search optimization algorithm (GS) was proposed. It achieves the prediction of shear strength by generalization and deduction of a large amount of data on rock fracture parameters, which avoids the complex derivation of theoretical equations. For practical application, a dataset comprising more than 134 shear tests on various rocks was compiled to collect the relevant three-dimensional morphological and mechanical parameters for training and prediction. Three classical shear strength models and the original SVR model were introduced for further comparison. Finally, sensitivity analysis was carried out to explore the relative importance of input variables to the shear strength. The results showed that the GS-SVR model (correlation coefficient R2 = 0.984, root mean squared error RMSE=0.383) outperformed the original SVR model (R2 = 0.936, RMSE=0.568). Moreover, compared with three classical shear strength models, the prediction results of the GS-SVR model were also most consistent with the experimental results (with the lowest RMSE and the highest R2). This machine learning model enhanced by GS can be used as a reliable and accurate shear strength prediction tool to partially replace laboratory tests to save costs.