For any integer m < n, where m can depend on n, we study the rate of convergence (Formula Presented) to its limiting Gaussian as n → ∞ for orthogonal, unitary and symplectic Haar distributed random matrices U of size n. In the unitary case, we prove that the total variation distance is less than (Formula Presented) times a constant. This result interpolates between the super-exponential bound obtained for fixed m and the 1/n bound coming from the Berry–Esseen theorem applicable when m ≥ n by a result of Rains. We obtain analogous results for the orthogonal and symplectic groups. In these cases, our total variation upper bound takes the form (Formula Presented) times a constant and the result holds provided n > 2m. For m = 1, we obtain complementary lower bounds and precise asymptotics for the L2-distances as n → ∞, which show how sharp our results are.