This Thesis is devoted to theoretical and experimental studies of resonant inelastic X-ray scattering (RIXS) of carbon monoxide and water molecules. Using state-of-the-art ab initio electronic structure calculations and a time-dependent wave packet formalism, we make a complete analysis of the experimental RIXS spectra of the two molecular systems. In the CO RIXS analysis, we are able to reproduce the RIXS experiment with an excellent accuracy. Interference between different RIXS channels corresponding to the scattering via orthogonal molecular orbitals in the core-excited state of CO is described. We show the complete breakdown of the Born-Oppenheimer approximation in the region where forbidden final Rydberg states are mixed with a valence allowed final state. Here we explain the formation of a spectral feature which was attributed to a single state in previous studies. Moreover, through an experimental-theoretical combination, we improve the minimum of the valence E’Π excited state potential, along with the coupling constant between two Rydberg states. We developed a new theoretical approach to describe triatomic molecules through the wave packet propagation formalism to study the water system, which reproduces with high accuracy the vibrational structure of the high-resolution experimental quasi-elastic RIXS spectra. We demonstrate that due to the vibrational mode coupling and anharmonicity of the ground and core-excited potential energy surfaces, different core-excited states in RIXS can be used as gates to probe different vibrational dynamics and to map the ground state potential. Isotopic substitution is investigated by theoretical simulations and important dynamical features are discussed, especially for the dissociative core-excited state, where a so-called “atomic” peak is formed. We show the strong potential of high-resolution RIXS experiments combined with high-level theoretical simulations for advanced studies of highly excited molecular states.