Coherent detection employing multilevel modulation format has become one of the most promising technologies for next generation high speed transmission system due to the high power and spectral efficiencies. With the powerful digital signal processing (DSP), coherent optical receivers allow the significant equalization of chromatic dispersion (CD), polarization mode dispersion (PMD), phase noise (PN) and nonlinear effects in the electrical domain. Recently, the realizations of these DSP algorithms for mitigating the channel distortions in the transmission system are the most attractive investigations. 

The CD equalization can be performed by the digital filters developed in the time and the frequency domain, which can suppress the fiber dispersion effectively. The PMD compensation is usually performed in the time domain with the adaptive least mean square (LMS) and constant modulus algorithms (CMA) equalization. Feed-forward and feed-back carrier phase estimation algorithms are employed to mitigate the phase noise from the transmitter and local oscillator lasers. The fiber nonlinearities are compensated by using the digital backward propagation methods based on solving the nolinear Schrodinger (NLS) equation and the Manakov equation. 

In this dissertation, we present a comparative analysis of three digital filters for chromatic dispersion compensation, an analytical evaluation of carrier phase estimation with digital equalization enhanced phase noise and a brief discussion for PMD adaptive equalization. To implement these investigations, a 112-Gbit/s non-return-to-zero polarization division multiplexed quadrature phase shift keying (NRZ-PDM-QPSK) coherent transmission system is realized in the VPI simulation platform. With the coherent transmission system, these CD equalizers have been compared by evaluating their applicability for different fiber lengths, their usability for dispersion perturbations and their computational complexity. Meanwhile, the bit-error-rate (BER) floor in carrier phase estimation using a one-tap normalized LMS filter is evaluated analytically, and the numerical results are compared to a differential QPSK detection system.