Digital Image Correlation (DIC) typically has poor accuracy when the speckle pattern is degraded, as in cases involving fractures, speckle melting or oxidation in high-temperature measurements, speckle slip, speckle obstruction due to surface roughness, pixel overexposure, or low-quality sensors with dead or defective pixels. We propose a pixel-removing DIC (PR-DIC) method that can accurately match the images with these challenging issues. The PR-DIC dynamically discards unreliable pixels within each subset and keeps only the good pixels for subset matching, enabling a better correlation. Numerical tests shows that PR-DIC has good matching accuracy and efficiency even under severe speckle pattern degradation, which obviously outperforms the classical DIC. A real test of joint root demonstrates reliable performance under practical fracture conditions when the images exhibit severe speckle degradation and crack-induced discontinuities.