We propose and demonstrate a new method to reduce the absorption recovery time of semiconductor saturable absorber mirrors operating at the 1060-nm wavelength range. The method is based on controlling the amount of nonradiative recombination centers within the absorbing region by incorporating an InGaP epitaxial layer with a relative large lattice mismatch to GaAs (~2.2 %). The defect density within the absorbing region can be controlled by the thickness of a GaAs buffer layer grown between the InGaP lattice mismatched layer and the InGaAs/GaAs quantum-wells. For thickness of the GaAs buffer of ∼110 nm and∼570 nm the absorption recovery time was∼5 ps and ∼10 ps, respectively. It is important to note that the fast recovery time was achieved without degrading the nonlinear optical properties of the saturable absorber mirror. The practicality of the structures was proved by demonstrating a reliable self-starting operation of a mode-locked Yb-doped fiber laser.