Accurate assessment of knee osteoarthritis (KOA) is limited by the current arthroscopy's reliance on subjective visual inspection. We propose a 3D spatio-spectral automatic grading framework combining deep learning with chromatic differential confocal matrix-based 3D topography and hyperspectral imaging (CDCM-THI). Our digital micromirror device-based system simultaneously captures high-resolution 3D micro-topography and hyperspectral cartilage signatures (lateral: 3.91 & micro;m, axial: 1.96 & micro;m, spectral: similar to 15 nm). To analyze this multi-modal data, we developed an improved ResNet50 late fusion network optimized with a consistency-enhanced loss function. Evaluated on ex vivo clinical samples, our framework achieved an average cross-validation accuracy of 94.83%, significantly outperforming single-modal baselines. This non-destructive approach bridges the gap between topographical assessment and broadband optical analysis, providing a powerful quantitative tool for future deep learning-assisted arthroscopic integration.