We show that the gapless boundary signatures-namely, chiral/helical hinge modes or localized zero modes- of three-dimensional higher-order topological insulators and superconductors with inversion symmetry can be gapped without symmetry breaking upon the introduction of non-Abelian surface topological order. In each case, the fractionalization pattern that appears on the surface is "anomalous" in the sense that it can be made consistent with symmetry only on the surface of a three-dimensional higher-order insulator/superconductor. Our results show that the interacting manifestation of higher-order topology is the appearance of "anomalous gapped boundaries" between distinct topological orders whose quasiparticles are related by inversion, possibly in conjunction with other protecting symmetries such as time-reversal symmetry and charge conservation.