We report the observation of a peculiar polarization behavior of BaFe12O19 in electric field where the linear polarization is detected at temperatures below 150 K whereas at higher temperatures a hysteresis-like polarization response is observed. At the same time, the performed neutron diffraction analysis shows no variations in crystal or magnetic structures with temperature. Based on the results of ab initio calculations we suggest the mechanism able to explain the experimentally observed behavior. We show that specific Fe atoms do not occupy the positions formally assigned to them by the conventional centrosymmetric P6(3)/mmc (#194) space group (z = 0.25; 0.75) as these positions correspond to local energy maxima. Instead, these Fe atoms are shifted along the z-axis to positions z = 0.259 (0.241) and z = 0.759 (0.741), which correspond to local energy minima. To an inversion center move between these minima Fe atoms need to overcome an energy barrier. This barrier is rather insignificant for smaller volumes but it becomes larger for expanded volumes due to coupling between the displacements of these Fe atoms. Additionally, our analysis suggests that the non-centrosymmetric and polar P6(3)mc (#186) space group could be appropriate for the description of the BaFe12O19 structure.