User demand for faster access to more data is at a historic high and rising. One of the enabling technologies that makes the information age possible is fiber-optic communications, where light is used to carry information from one place to another over optical fiber. Since the technology was first shown to be feasible in the 1970s, it has been constantly evolving with each new generation of fiber-optic systems achieving higher data rates than its predecessor.

Today, the most promising approach for further increasing data rates is digital signal processing (DSP)-based coherent optical transmission with multi-level modulation. As multi-level modulation formats are very susceptible to noise and distortions, forward error correction (FEC) is typically used in such systems. However, FEC has traditionally been designed for additive white Gaussian noise (AWGN) channels, whereas fiber-optic systems also have other impairments. For example, there is relatively high phase noise (PN) from the transmitter and local oscillator (LO) lasers.

The contributions of this thesis are in two areas. First, we use a unified approach to analyze theoretical performance limits of coherent optical receivers and microwave receivers, in terms of signal-to-noise ratio (SNR) and bit error rate (BER). By using our general framework, we directly compare the performance of ten coherent optical receiver architectures and five microwave receiver architectures. In addition, we put previous publications into context, and identify areas of agreement and disagreement between them. Second, we propose straightforward methods to select codes for systems with PN. We focus on Bose-Chaudhuri-Hocquenghem (BCH) codes with simple implementations, which correct pre-FEC BERs around 10⁻³. Our methods are semi-analytical, and need only short pre-FEC simulations to estimate error statistics. We propose statistical models that can be parameterized based on those estimates. Codes can be selected analytically based on our models.

Since the invention of fiber-optic systems in the 1970s, user demand has driven innovation forward, and each new generation of products has achieved higher data rates than its predecessor. Today, the most promising approach for further increasing data rates is coherent transmission with multi-level modulation and digital signal processing (DSP). By using multi-level modulation, data rates can be increased without increasing the spectral bandwidth of the signal. Digital signal processing has a highly-predictable design flow, and solutions are likely to become more attractive in the future as technology scales. As multi-level modulation is very susceptible to noise and distortions, these systems typically include forward error correction (FEC), which fits well with the DSP structure.

In this thesis, we focus on two aspects of DSP-based coherent systems. First, we use a unified approach to analyze theoretical performance limits of coherent optical receivers and microwave receivers, in terms of signal-to-noise ratio (SNR) and bit error rate (BER). By using our general framework, we directly compare the performance of ten coherent optical receiver architectures and five microwave receiver architectures. In addition, we put previous publications into context, and identify areas of agreement and disagreement between them.

Second, we consider simple Bose-Chaudhuri-Hocquenghem (BCH) codes for such systems. While most of coding theory is based on the assumption of additive white Gaussian noise (AWGN) channels, fiber-optic systems have other channel impairments in addition to AWGN. For example, there is relatively high phase noise (PN) from the transmitter and local oscillator (LO) lasers. We present a family of straightforward methods for selecting BCH codes for systems with PN. These codes are highly predictable and systematic to construct. They have low-complexity implementations and no error floor. Our methods are based on simple statistical models that can be parameterized from pre-FEC simulations, thus requiring only modest simulation effort. They are suitable for correcting pre-FEC BERs of around 10^−3. We consider differential quadrature phase-shift keying (DQPSK) modulation and higher-order differential quadrature amplitude modulation (DQAM) with star-shaped constellations.

This thesis is an extension of our licentiate thesis, and improves upon the latter in two significant ways. First, the methods for code selection that were previously limited to DQPSK are now generalized to higher-order star-shaped DQAM formats, which can potentially deliver higher data rates. Second, we consider block interleavers which yield practical low-complexity implementations. These complement our earlier analysis of uniform interleavers, which provide general theoretical insight.