Unbound aggregate materials (UAMs) with large air voids are increasingly used to construct pavement base and subbase layers as part of the initiative to develop sponge cities and improve drainage performance. However, the motion of particles and the spatial distribution of kinetic energy during the particle rearrangement process induced by vibratory loading remain unclear. This study presented the results of laboratory vibratory plate compaction tests conducted on UAM specimens under various combinations of vibratory parameters and different levels of Gravel to Sand ratio (G/S). SmartRock (SR) sensors were embedded within the specimens to monitor real-time particle motion, while kinematic energy and its spatial distribution were analyzed from the acceleration time-history signals collected by the SR sensors. Based on high-precision industrial X-ray computed tomography (XCT) and three-dimensional (3D) reconstruction technology, the motion and migration characteristics of coarse particles were analyzed. A new compaction index was proposed based on particle motion and kinematic energy to evaluate the compaction quality of the specimens. The findings of this study reveal that vibratory compaction can be divided into three distinct stages. During the first stage, coarse particles primarily move vertically, while energy dissipation occurs mainly through the compression of air voids but does not form a dense skeleton structure. In the second stage, coarse particles translate horizontally while rotating vertically, resulting in a tendency of the particles to align horizontally with their long axes. During this stage, dissipated kinematic energy is primarily used to fill air voids, leading to the formation of a densely packed skeleton structure. Kinematic energy indices and particle movement in the middle of the specimens can be used to evaluate the compaction stage and quality. Additionally, the lateral particle motion within the specimens transitions from continuous ascent to gradual descent (i.e., culminating in minimal kinematic energy), thus indicating a relatively dense compaction state. Reducing the void ratio and increasing the contact area between particles within the size ranges of 4.75–9.5 mm and 2.36–4.75 mm can significantly increase the compaction density and improve the stability and deformation resistance of unbound pavement base/subbase layers. The results of this study provide valuable insights into the mechanisms of vibratory compaction and can be used to optimize compaction methods and improve pavement performance.