This paper focuses on estimating probability distributions over the set of 3D rotations (SO(3)) using deep neural networks. Learning to regress models to the set of rotations is inherently difficult due to differences in topology between RN and SO(3). We overcome this issue by using a neural network to output the parameters for a matrix Fisher distribution since these parameters are homeomorphic to R9. By using a negative log likelihood loss for this distribution we get a loss which is convex with respect to the network outputs. By optimizing this loss we improve state-of-the-art on several challenging applicable datasets, namely Pascal3D+, ModelNet10-SO(3). Our code is available at https://github.com/Davmo049/Public_prob_orientation_estimation_with_matrix _fisher_distributions.