We present an anomaly-based drone classification scheme. High dimensional spectrum data is encoded using a convolutional neural network autoencoder. This is trained on data generated from a generic mathematical drone model. Once encoded, we use quadratic discriminant analysis for non-drone classes and define anomalies in terms of the log likelihood and prior knowledge from the drone model. When integrating ten samples, we can discriminate drones from non-drone samples such as birds, with an average accuracy of 98% at 20 dB signal to noise ratio. This corresponds to an effective observation time of 90 ms.