Channel estimation attacks can degrade the performance of the legitimate system and facilitate eavesdropping. It is known that pilot contamination can alter the legitimate transmit precoder design and strengthen the quality of the received signal at the eavesdropper, without being detected. In this paper, we devise a technique which employs random pilots chosen from a known set of phase-shift keying (PSK) symbols to detect pilot contamination. The scheme only requires two training periods without any prior channel knowledge. Our analysis demonstrates that using the proposed technique in a massive MIMO system, the detection probability of pilot contamination attacks can be made arbitrarily close to 1. Simulation results reveal that the proposed technique can significantly increase the detection probability and is robust to noise power as well as the eavesdropper's power.