In this work, fundamental control limitations for rotor angle stability are considered. Limitations are identified by characterizing open-loop transfer function zeros for input-output combinations of certain power system configurations. Of particular interest are non-minimum phase (NMP) zeros that limit the achievable performance of the closed-loop system. By studying a single-machine infinite bus power system model, analytic conditions for the presence of NMP zeros are derived. They are shown to be closely linked to the destabilizing effect of automatic voltage regulators (AVRs). Depending on the control loop, it is found that NMP zeros may persist in the system even if the closed-loop system is stabilized through feedback control. A simulation study shows that NMP zeros introduced by AVR limit the achievable performance and stabilization using feedback control.