In this paper we present a modification to a latent topic model, which makes themodel exploit supervision to produce a factorized representation of the observeddata. The structured parameterization separately encodes variance that is sharedbetween classes from variance that is private to each class by the introduction of anew prior over the topic space. The approach allows for a more efficient inferenceand provides an intuitive interpretation of the data in terms of an informative signaltogether with structured noise. The factorized representation is shown to enhanceinference performance for image, text, and video classification.