Failed spacecraft are generally regarded as space targets with flexible solar panels. Obtaining high-accuracy complex motion parameters of space targets is a prerequisite for maintenance or repair. However, due to the limitations of existing measurement equipment, low accuracy arises when measuring motion parameters directly. Thus, it is essential to obtain the estimation of motion parameters that are accurate enough from the measurement data with noise. In this paper, the estimation method for the complex motion parameters of a nonlinear dynamic system based on the vibration mode database is proposed for the space target with flexible solar panels. Firstly, a rigid-flexible coupling dynamic model is established, and the coupling measure between the central rigid body and the solar panels is thoroughly analyzed. Secondly, an adaptive particle filter for the motion parameters estimation of the nonlinear dynamic system is improved by optimizing the particle number using the Kullback-Leibler divergence-based sampling strategy. Simulation results show there exists a strong dynamic coupling between the attitude motion of the central rigid body and the flexible vibration of the solar panels. Compared with the extended Kalman filter and classical particle filter, the proposed method has higher accuracy in estimating complex motion parameters. In the future, the proposed method could be further extended to accommodate more complex spacecraft configurations and dynamic environments, such as those involving multi-body systems or external disturbances.