Dust in the interstellar medium (ISM) is critical to the absorption and intensity of emission profiles used widely in astronomical observations, and necessary for star and planet formation. Supernovae (SNe) both produce and destroy ISM dust. In particular the destruction rate is difficult to assess. Theory and prior simulations of dust processing by SNe in a uniform ISM predict quite high rates of dust destruction, potentially higher than the supernova dust production rate in some cases. Here we show simulations of supernova-induced dust processing with realistic ISM dynamics including magnetic field effects and demonstrate how ISM inhomogeneity and magnetic fields inhibit dust destruction. Compared to the non-magnetic homogeneous case, the dust mass destroyed within 1 Myr per SNe is reduced by more than a factor of two, which can have a great impact on the ISM dust budget.