XL-Calibur is a balloon-borne Compton polarimeter for X-rays in the ∼15–80 keV range. Using an X-ray mirror with a 12 m focal length for collecting photons onto a beryllium scattering rod surrounded by CZT detectors, a minimum-detectable polarization as low as ∼3% is expected during a 24-hour on-target observation of a 1 Crab source at 45° elevation. Systematic effects alter the reconstructed polarization as the mirror focal spot moves across the beryllium scatterer, due to pointing offsets, mechanical misalignment or deformation of the carbon-fiber truss supporting the mirror and the polarimeter. Unaddressed, this can give rise to a spurious polarization signal for an unpolarized flux, or a change in reconstructed polarization fraction and angle for a polarized flux. Using bench-marked Monte-Carlo simulations and an accurate mirror point-spread function characterized at synchrotron beam-lines, systematic effects are quantified, and mitigation strategies discussed. By recalculating the scattering site for a shifted beam, systematic errors can be reduced from several tens of percent to the few-percent level for any shift within the scattering element. The treatment of these systematic effects will be important for any polarimetric instrument where a focused X-ray beam is impinging on a scattering element surrounded by counting detectors.