Cell-free massive multiple-input multiple-output (CF mMIMO) systems serve the user equipments (UEs) by geographically distributed access points (APs) by means of joint transmission and reception. To limit the power consumption due to fronthaul signaling and processing, each UE should only be served by a subset of the APs, but it is hard to identify that subset. Previous works have tackled this combinatorial problem heuristically. In this paper, we propose a sparse distributed processing design for CF mMIMO, where the AP-UE association and long-term signal processing coefficients are jointly optimized. We formulate two sparsity-inducing mean-squared error (MSE) minimization problems and solve them by using efficient proximal approaches with block-coordinate descent. For the downlink, more specifically, we develop a virtually optimized large-scale fading precoding (V-LSFP) scheme using uplink-downlink duality. The numerical results show that the proposed sparse processing schemes work well in both uplink and downlink. In particular, they achieve almost the same spectral efficiency as if all APs would serve all UEs, while the energy efficiency is 2-4 times higher thanks to the reduced processing and signaling.