Coherent detection employing multilevel modulation formats has become one of the most promising technologies for next generation high speed transmission systems due to the high power and spectral efficiencies. Using the powerful digital signal processing (DSP), coherent optical receivers allow the significant equalization of chromatic dispersion (CD), polarization mode dispersion (PMD), phase noise (PN) and nonlinear effects in the electrical domain. Recently, the realizations of these DSP algorithms for mitigating the channel distortions in the coherent transmission systems are the most attractive investigations.

The CD equalization can be performed by the digital filters developed in the time and the frequency domain, which can suppress the fiber dispersion effectively. The PMD compensation is usually performed in the time domain with the adaptive least mean square (LMS) and constant modulus algorithms (CMA) equalization. Feed-forward and feed-back carrier phase estimation (CPE) algorithms are employed to mitigate the phase noise (PN) from the transmitter (TX) and the local oscillator (LO) lasers. The fiber nonlinearities are compensated by using the digital backward propagation methods based on solving the nonlinear Schrödinger (NLS) equation and the Manakov equation.

In this dissertation, we present a comparative analysis of three digital filters for chromatic dispersion compensation, a comparative evaluation of different carrier phase estimation methods considering digital equalization enhanced phase noise (EEPN) and a brief discussion for PMD adaptive equalization. To implement these investigations, a 112-Gbit/s non-return-to-zero polarization division multiplexed quadrature phase shift keying (NRZ-PDM-QPSK) coherent transmission system with post-compensation of dispersion is realized in the VPI simulation platform. In the coherent transmission system, these CD equalizers have been compared by evaluating their applicability for different fiber lengths, their usability for dispersion perturbations and their computational complexity. The carrier phase estimation using the one-tap normalized LMS (NLMS) filter, the differential detection, the block-average (BA) algorithm and the Viterbi-Viterbi (VV) algorithm is evaluated, and the analytical predictions are compared to the numerical simulations. Meanwhile, the phase noise mitigation using the radio frequency (RF) pilot tone is also investigated in a 56-Gbit/s NRZ single polarization QPSK (NRZ-SP-QPSK) coherent transmission system with post-compensation of chromatic dispersion. Besides, a 56-Gbit/s NRZ-SP-QPSK coherent transmission system with CD pre-distortion is also implemented to analyze the influence of equalization enhanced phase noise in more detail.