Quantum scattering amplitudes for massive matter have received new attention in connection to classical calculations relevant to gravitational-wave physics. Amplitude methods and insights are now employed for precision computations of observables needed for describing the gravitational dynamics of bound massive objects such as black holes. An important direction is the inclusion of spin effects needed to accurately describe rotating (Kerr) black holes. Higher-spin amplitudes introduced by Arkani-Hamed, Huang and Huang at three points have by now a firm connection to the effective description of Kerr black-hole physics. The corresponding Compton higher-spin amplitudes remain however an elusive open problem. Here we draw from results of the higher-spin literature and show that physical insights can be used to uniquely fix the Compton amplitudes up to spin 5/2, by imposing a constraint on a three-point higher-spin current that is a necessary condition for the existence of an underlying unitary theory. We give the unique effective Lagrangians up to spin 5/2, and show that they reproduce the previously-known amplitudes. For the multi-graviton amplitudes analogous to the Compton amplitude, no further corrections to our Lagrangians are expected, and hence such amplitudes are uniquely predicted. As an essential tool, we introduce a modified version of the massive spinor-helicity formalism which allows us to conveniently obtain higher-spin states, propagators and compact expressions for the amplitudes.