The supernova remnant (SNR) 0540-69.3, twin of the Crab Nebula, offers an excellent opportunity to study the continuum emission from a young pulsar and pulsar wind nebula (PWN). We present observations taken with the Very Large Telescope instruments MUSE and X-shooter in the wavelength range 3000-25000 Å, which allow us to study spatial variations of the optical spectra, along with the first near-infrared (NIR) spectrum of the source. We model the optical spectra with a power law (PL) F ν ∝ ν −α and find clear spatial variations (including a torus-jet structure) in the spectral index across the PWN. Generally, we find spectral hardening toward the outer parts, from α ∼ 1.1 to ∼0.1, which may indicate particle reacceleration by the PWN shock at the inner edge of the ejecta or alternatively time variability of the pulsar wind. The optical-NIR spectrum of the PWN is best described by a broken PL, confirming that several breaks are needed to model the full spectral energy distribution of the PWN, and suggesting the presence of more than one particle population. Finally, subtracting the PWN contribution from the pulsar spectrum we find that the spectrum is best described with a broken-PL model with a flat and a positive spectral index, in contrast to the Crab pulsar that has a negative spectral index and no break in the optical. This might imply that pulsar differences propagate to the PWN spectra.