In this study, a portable confocal hyperspectral microscope is developed. In traditional confocal laser scanning microscopes, scan lens and tube lens are utilized to achieve a conjugate relationship between the galvanometer and the back focal plane of the objective, in order to achieve a better resolution. However, these lenses make it difficult to scale down the volume of the system. In our portable confocal hyperspectral microscope (PCHM), the objective is placed directly next to the galvomirror. Thus, scan lens and tube lens are not included in our system and the size of this system is greatly reduced. Furthermore, the resolution is also acceptable in many biomedical and food-safety applications. Through reducing the optical length of the system, the signal detection efficiency is enhanced. This is conducive to realizing both the fluorescence and Raman hyperspectral imaging. With a multimode fiber as a pinhole, an improved image contrast is also achieved. Fluorescent spectral images for HeLa cells/fingers and Raman spectral images of kumquat pericarp are present. The spectral resolution and spatial resolutions are about 0.4 nm and 2.19 mu m, respectively. These results demonstrate that this portable hyperspectral microscope can be used in in-vivo fluorescence imaging and in situ Raman spectral imaging.