We uncover connections between maximum likelihood estimation in statistics and norm minimization over a group orbit in invariant theory. We present a dictionary that relates notions of stability from geometric invariant theory to the existence and uniqueness of a maximum likelihood estimate. Our dictionary holds for both discrete and continuous statistical models: we discuss log-linear models and Gaussian models, including matrix normal models and directed Gaussian graphical models. Our approach reveals promising consequences of the interplay between invariant theory and statistics. For instance, algorithms from statistics can be used in invariant theory, and vice versa.