Discovery of pulsations from a number of ultra-luminous X-ray (ULX) sources proved that accretion onto neutron stars can produce luminosities exceeding the Eddington limit by several orders of magnitude. The conditions necessary to achieve such high luminosities as well as the exact geometry of the accretion flow in the neutron star vicinity are, however, a matter of debate. The pulse phase-resolved polarization measurements that became possible with the launch of the Imaging X-ray Polarimetry Explorer (IXPE) can be used to determine the pulsar geometry and its orientation relative to the orbital plane. They provide an avenue to test different theoretical models of ULX pulsars. In this paper we present the results of three IXPE observations of the first Galactic ULX pulsar Swift J0243.6+6124 during its 2023 outburst. We find strong variations in the polarization characteristics with the pulsar phase. The average polarization degree increases from about 5% to 15% as the flux dropped by a factor of three in the course of the outburst. The polarization angle (PA) as a function of the pulsar phase shows two peaks in the first two observations, but changes to a characteristic sawtooth pattern in the remaining data set. This is not consistent with a simple rotating vector model. Assuming the existence of an additional constant polarized component, we were able to fit the three observations with a common rotating vector model and obtain constraints on the pulsar geometry. In particular, we find the pulsar angular momentum inclination with respect to the line of sight of i(p)=15 degrees-40 degrees, the magnetic obliquity of theta(p) = 60 degrees-80 degrees, and the pulsar spin position angle of chi(p)approximate to-50 degrees, which significantly differs from the constant component PA of about 10 degrees. Combining these X-ray measurements with the optical PA, we find evidence for at least a 30 degrees misalignment between the pulsar angular momentum and the binary orbital axis.