Coherent detection together with multilevel modulation formats has the potential to significantly increase the capacity of existing optical communication systems at no extra cost in signal bandwidth. However, these modulation formats are more susceptible to the impact of different noise sources and distortions as the distance between its constellation points in the complex plane reduces with the modulation index. In this context, digital signal processing (DSP) plays a key role as it allows compensating for the impairments occurring during signal generation, transmission and/or detection relaxing the complexity of the overall system. The transition towards pluggable optical transceivers, offers flexibility for network design/upgrade but sets strict requirements on the power consumption of the DSP thus limiting its complexity. The DSP module complexity however, scales with the modulation order and, in this scenario, low complex yet high performance DSP algorithms are highly desired.

In this thesis, we mainly focus on the impact of laser phase noise arising from the transmitter and local oscillator (LO) lasers in coherent optical communication systems employing high order modulation formats. In these systems, the phase noise of the transmitting and LO lasers translate into phase noise in the received constellation impeding the proper recovery of the transmitted data. In order to increase the system phase noise tolerance, we firstly explore the possibility of re-arranging the constellation points in a circularly shaped mQAM (C-mQAM) constellation shape to exploit its inherent phase noise tolerance. Different low-complex carrier phase recovery (CPR) schemes applicable to these constellations are proposed along with a discussion on its performance and implementation complexity. Secondly, the design guidelines of high performance and low complex CPR schemes for conventional square mQAM constellations are presented. We identify the inherent limitation of the state-of-the-art blind phase search (BPS) carrier phase recovery algorithm which hinders its achievable performance and implementation complexity and present a low complex solution to overcome it. The design guidelines of multi-stage CPR schemes for high order modulation formats, where the BPS algorithm is employed at any of the stages, are also provided and discussed. Finally, the interplay between the received dispersed signal and the LO phase noise is analytically investigated to characterize the origin of the equalization enhanced phase noise phenomena.

The increase in capacity demand along with the advancement in digital signal processing (DSP) have recently revived the interest in coherent optical communications and led to its commercialization. However, design and development of robust DSP algorithms for example for carrier phase recovery (CPR) becomes complex as we opt for high order modulation formats such as 16QAM and beyond. Further, electrical-domain dispersion compensation (EDC), while providing many advantages, makes the system more susceptible to laser frequency noise (FN). For instance, in coherent optical links with post-reception EDC, while the transmitter frequency noise causes only phase impairment, the local oscillator (LO) FN in these systems results in a noise enhancement in both amplitude and phase. This noise is commonly known as equalization enhanced phase noise (EEPN). It results in asymmetric requirements for transmitter laser and LO laser. Further, the system design in the presence of lasers with non-white frequency noise becomes increasingly challenging for increased capacity-distance product.

The main contributions of this thesis are, firstly, an experimentally validated theory of coherent optical links with lasers having general non-white frequency noise spectrum and corresponding system/laser design criteria and mitigation technique. Secondly, low complexity and high phase noise tolerant CPR for high order modulation formats.

The general theory propounded in this thesis elucidates the origin of the laser frequency noise induced noise enhancement in coherent optical links with different DSP configurations. The thesis establishes the existence of multiple frequency noise regimes and shows that each regime results in different set of impairments. The influence of the impairments due to some regimes can ideally be reduced by optimizing the corresponding mitigation algorithms, while other regimes cause irretrievable impairments. Experimentally validated theoretical boundaries of these regimes and corresponding criteria applicable to system/laser design are provided. Further, an EEPN mitigation method and its two possible implementations are proposed and discussed.

The thesis also demonstrates an intrinsic limitation of the conventional Blind Phase Search (BPS) algorithm due to angular quantization and provides methods to overcome it. Finally, this thesis proposes and demonstrates single stage and multi-stage carrier phase recovery algorithms for compensation of phase impairments due to the two lasers for higher order circular and square modulations. The proposed methods outperform the state of art algorithms both in performance and in complexity.