This paper presents an approach to automatic surface defect detection by a deep learning-based object detection method, particularly in challenging scenarios where defects are rare, i.e., with limited training data. We base our approach on an object detection model YOLOv8, preceded by a few steps: 1) filtering out irrelevant information, 2) enhancing the visibility of defects, namely brightness contrast, and 3) increasing the diversity of the training data through data augmentation. We evaluated the method in an industrial case study of crown wheel surface inspection in detecting Unclean Gear as well as Deburring defects, resulting in promising performances. With the combination of the three preprocessing steps, we improved the detection accuracy by 22.2% and 37.5% respectively while detecting those two defects. We believe that the proposed approach is also adaptable to various applications of surface defect detection in other industrial environments as the employed techniques, such as image segmentation, are available off the shelf.