This paper introduces a holistic approach to develop a Smart Fault Detection and Diagnosis (SFDD) system which can be integrated and widely used in both old and new heat pump systems worldwide. A SFDD acts as the heart of smart heat pump system. It is able to minimize the installation and control errors, couple or decouple the renewable energy sources, make load-shifting possible, detect the performance degradation during operation, avoid unnecessary visual inspections and components replacement, and reduce the maintenance cost and down-time of the system. In order to determine the most important faults in the heat pump system, a comprehensive study is done on the faults reported to both OEMs and a main insurance company in Sweden and the results are briefly presented. The paper also discusses how a knowledge-based Smart Fault Detection and Diagnosis unit should be designed to support the heat pump systems at commissioning, operation, and maintenance phases.