The rapid development of data-driven techniques brings us new applications, such asfifth-generation new radio (5G NR), high-definition video, Internet of things (IoT),etc., which has greatly facilitated our daily lives. Optical networks as one fundamen-tal infrastructure are evolving to simultaneously support these high dimensional dataservices, with a feature of flexible, dynamic, and heterogeneous. Optical performancemonitoring (OPM) is a key enabler to guarantee reliable network management andmaintenance, which improving network controllability and resource efficiency. Accu-rately telemetry key performance indicators (KPIs) such as bit error rate (BER) canextend monitoring functionality and secure network management. However, retrievingthe BER level metric is time-consuming and inconvenient for OPM. Low-complexityOPM strategies are highly desired for ubiquitous departments at optical network nodes.This thesis investigates machine learning (ML) based intelligent error vector mag-nitude (EVM) estimation schemes in digital coherent communications, where EVMis widely used as an alternative BER metric for multilevel modulated signals. Wepropose a prototype of EVM estimation, which enables monitoring signal quality froma short observation period. Three alternative ML algorithms are explored to facilitatethe implementation of this prototype, namely convolutional neural networks (CNNs),feedforward neural networks (FFNNs), and linear regression (LR). We show that CNNconjunction with graphical signal representations, i.e., constellation diagrams and am-plitude histograms (AHs), can achieve decent EVM estimation accuracy for signalsbefore and after carrier phase recovery (CPR), which outperforms the conventionalEVM calculation. Moreover, we show that an FFNN-based scheme can reduce poten-tial energy and keep the estimation accuracy by directly operating with AH vectors.Furthermore, the estimation capability is thoroughly studied when the system hasdifferent impairments. Lastly, we demonstrate that a simple LR-designed model canperform as well as FFNN when the information on modulation formats is known. SuchLR-based can be easily implemented with modulation formats identification modulein OPM, providing accurate signal quality information.