Supervised topic models are important machine learning tools whichhave been widely used in computer vision as well as in other domains. However,there is a gap in the understanding of the supervision impact on the model. Inthis paper, we present a thorough analysis on the behaviour of supervised topicmodels using Supervised Latent Dirichlet Allocation (SLDA) and propose twofactorized supervised topic models, which factorize the topics into signal andnoise. Experimental results on both synthetic data and real-world data for computer vision tasks show that supervision need to be boosted to be effective andfactorized topic models are able to enhance the performance.