Motivated by recent advances in the creation of few-body atomic Fermi gases with attractive interactions, we study theoretically the few-to-many-particle crossover of pair excitations, which for large particle numbers evolve into a mode that describes amplitude fluctuations of the superfluid order parameter (the "Higgs"mode). Our analysis is based on the hypothesis that salient aspects of the excitation spectrum are captured by interactions between time-reversed pair states in a harmonic oscillator potential. Microscopically, this assumption leads to a Richardson-type pairing model, which is integrable and thus allows a systematic quantitative study of the few-to-many-particle crossover with only minor numerical effort. We first establish a parity effect in the ground-state energy, i.e., a spectral convexity in the energy of open-shell configurations compared to their closed-shell neighbors, which is quantified by a so-called Matveev-Larkin parameter discussed for mesoscopic superconductors, which generalizes the pairing gap to mesoscopic ensembles and which behaves quantitatively differently in a few-body and a many-body regime. The crossover point for this quantity is widely tunable as a function of interaction strength. We then compute the excitation spectrum and demonstrate that the pair excitation energy shows a minimum that deepens with increasing particle number and shifts to smaller interaction strengths, consistent with the finite-size precursor of a quantum phase transition to a superfluid state. We extract a critical finite-size scaling exponent that characterizes the decrease of the gap with increasing particle number.