Due to the equipment failure and inappropriate operation strategy, it is often difficult to achieve energy-efficient building. Anomaly detection of building energy consumption is an important approach to improve building energy-saving. This study proposes a dynamic anomaly detection algorithm, which realizes the dynamic detection of point anomalies and collective anomalies. This investigation established a semi-supervised matching mechanism, which avoids the influence of error label and improves the efficiency of anomaly detection. A particle swarm optimization (PSO) is used to optimize the unsupervised clustering algorithm. This investigation tests the effectiveness of the proposed algorithm and evaluates the performance of the energy consumption clustering algorithm by using the annual electricity consumption data of an experimental building in a university. The results show that the clustering accuracy of the algorithm can reach more than 80%, and it can effectively detect the building energy consumption data of two different forms of outliers.