With advances in Deep Neural Networks (DNN), Automated Driving Systems (ADS) enable the vehicle to perceive their surroundings in dynamic driving scenarios and perform behaviors by collecting operational data from sensors such as LiDAR and cameras. Current DNN typically detect objects by analyzing and classifying unstructured data (e.g., image data), providing critical information for ADS planning and decision-making. However, advanced ADS, particularly those required to perform the Dynamic Driving Task (DDT) autonomously, are expected to understand driving scenarios across various Operational Design Domains (ODD). This capability requires the support for a continuous comprehension of driving scenarios according to operational data collected by sensors. This paper presents a framework that adopts Graph Neural Networks (GNN) to describe and reason about dynamic driving scenarios via analyzing graph-based data based on collected sensor inputs. We first construct the graph-based data using a meta-path, which defines various interactions among different traffic participants. Next, we propose a design of GNN to support both the classification of the node types of objects and predicting relationships between objects. As results, the performance of the proposed method shows significant improvements compared to the baseline method. Specifically, the accuracy of node classification increases from 0.77 to 0.85, while that of relationships prediction rises from 0.74 to 0.82. To further utilize graph-based data constructed from dynamic driving scenarios, the proposed framework supports reasoning about operational risks by analyzing the observed nodes and relationships in the graph-based data. As a result, the model achieves a MRR of 0.78 in operational risks reasoning. To evaluate the practicality of the proposed framework in real-world systems, we also conduct a real-time performance evaluation by measuring the average process time and the Worst Case Execution Time (WCET). Compared to the baseline models, the results demonstrate the proposed framework presents acceptable real-time performance in analyzing graph-based data

Cyber-Physical Systems (CPS) refer to intelligent systems that combine computational and physical capabilities to enable advanced functionalities, such as autonomous behaviors, human-machine interaction, and machine collaboration in complex environments. Addressing these functionalities often necessitates the adoption of Artificial Intelligence (AI) techniques which are extensively utilized for operational perception and decision-making. However, due to the inherently data-intensive and opaque nature of most AI-enabled components, combined with unforeseen environments, the integration of AI-enabled techniques into CPS presents significant engineering challenges in quality management. Self-management, as an embedded system feature, extends conventional CPS with capabilities for operation monitoring, planning and adaptation. It is often considered as a necessary mechanism for ensuring the quality and trustworthiness of AI-enabled components. However, implementing self-management in AI-enabled CPS presents several challenges: The concept of self-management varies depending on the audience and application, making its definition and implementation more complex. Additionally, the data-intensive nature of AI-enabled components requires extra effort to ensure consistent performance across operational domains, especially under unforeseen conditions. To cope with these challenges, this thesis proposes to integrate data-driven and knowledge-enabled methods for self-management through the following efforts: 1) Proposing conceptual frameworks that define the necessary and sufficient functionalities for self-management, with the support for situation awareness regarding the internal and environmental conditions; 2) Developing condition monitoring modules within the proposed conceptual frameworks for the situation-awareness to analyze system status; 3) Creating human-explainable data-driven methods for understanding operational conditions; 4) Designinglearning-based agents to dynamically and effectively address vulnerabilities inAI-enabled systems that could lead to system failures or compromise the operational safety. This thesis consolidates key concepts and introduces novel features for self-management in CPS by synthesizing insights from existing research and addressing their limitations. It provides a framework for designing learning based agents that leverage data synthesis to achieve self-management. Additionally, the thesis develops data-driven methods integrated with knowledge enabled models to enhance situation awareness and trustworthiness, effectively addressing the complexity and opacity of AI-enabled computing processes.

Cyberfysiska system (CPS) är intelligenta system som kombinerar beräknings-mässiga och fysiska förmågor för att möjliggöra avancerade funktioner, som autonoma beteenden, interaktion mellan människa och maskin och maskinsamarbete i komplexa miljöer. För att hantera dessa funktionaliteter används ofta artificiell intelligens (AI), särskilt inom operativ perception och beslutsfattande. Men på grund av att de flesta AI-komponenter är dataintensiva och svårgenomtränglig, i kombination medoförut-sedda miljöer, innebär integrationen av AI-komponenter i CPS betydande tekniska utmaningar när det gäller kvalitetshantering.

Självförvaltning, som en inbyggd systemfunktion, utökar konventionella CPS med funktionaliteter för driftsövervakning, planering och anpassning. Den anses ofta vara en nödvändig mekanism för att säkerställa kvaliteten och pålitligheten hos AI-komponenter.

Att implementera självförvalt-ning i AI-aktiverade CPS innebär dock flera utmaningar: Begreppet tolkas olika beroende på målgrupp och tillämpning, vilket gör definitionen och implementeringen mer komplex. Dessutom kräver AI-komponenter ofta extra ansträngningar för att säkerställa konsekvent prestanda över olika operativa domäner, särskilt under oförutsedda förhållanden.

För att adressera dessa utmaningar föreslår denna avhandling en integrering av datadrivna och kunskapsbaserade metoder för självförvaltning genom följande insatser: 1) Utveckling av konceptuella ramverk som definierar de nödvändiga och tillräckliga funktionerna för autonom hantering, med stöd för situationsmedvetenhet om både interna och externa förhållandena; 2) Implementering av tillståndsövervakningsmoduler inom de föreslagna konceptuella ramverken för att analysera systemstatus och förbättra situationsmedvetenheten; 3) Skapande av förklarbara datadrivna metoder som möjliggör en bättre förståelse av operativa förhållanden; 4) Design av inlärningsbaserade agenter för att dynamiskt och effektivt hantera sårbarheter i AI-baserade system, förebygga systemfel och säkerställa driftsäkerhet.

Denna avhandling samlar nyckelbegrepp och introducerar nya funktioner för autonom hantering i cyberfysiska system genom att sammanställa insikter från befintliga studier och adressera deras begränsningar. Den presenterar ett ramverk för att utforma inlärningsbaserade agenter som utnyttjar datasyntes för att uppnå autonom hantering. Vidare utvecklas datadrivna metoder som kombineras med kunskapsbaserade modeller för att förbättra situationsmedvetenhet och pålitlighet, samtidigt som de komplexa och svårgenomträngliga databehandlingsprocesserna i AI-drivna komponenter hanteras.

This paper presents a reconfigurable near-sensor anomaly detection processor to real-time monitor the potential anomalous behaviors of amputees with limb prostheses. The processor is low-power, low-latency, and suitable for equipment on the prostheses and comprises a reconfigurable Variational Autoencoder (VAE), a scalable Self-Organizing Map (SOM) Array, and a window-size-adjustable Markov Chain, which can implement an integrated miniaturized anomaly detection system. With the reconfigurable VAE, the proposed processor can support up to 64 sensor sampling channels programmable by global configuration, which can meet the anomaly detection requirements in different scenarios. A scalable SOM array allows for the selection of different sizes based on the complexity of the data. Unlike traditional time accumulation-based anomaly detection methods, the Markov Chain is utilized to detect time-series-based anomalous data. The processor is designed and fabricated in a UMC 40-nm LP technology with a core area of 1.49 mm2 and a power consumption of 1.81 mW. It achieves real-time detection performance with 0.933 average F1 Score for the FSP dataset within 24.22 s, and 0.956 average F1 Score for the SFDLA-12 dataset within 30.48 s, respectively. The energy dissipation of detection for each input feature is 43.84 nJ with the FSP dataset, and 55.17 nJ with the SFDLA-12 dataset. Compared with ARM Cortex-M4 and ARM Cortex-M33 microcontrollers, the processor achieves energy and area efficiency improvements ranging from 257×, 193× and 11×, 8×, respectively. IEEE

Human Activity Recognition (HAR) refers to a field that aims to identify human activitiesby adopting multiple techniques. In this field, different applications, such as smart homes andassistive robots, are introduced to support individuals in their Activities of Daily Living (ADL)by analyzing data collected from various sensors. Apart from wearable sensors, the adoption ofcamera frames to analyze and classify ADL has emerged as a promising trend for achieving theidentification and classification of ADL. To accomplish this, the existing approaches typically rely onobject classification with pose estimation using the image frames collected from cameras. Given theexistence of inherent correlations between human–object interactions and ADL, further efforts areoften needed to leverage these correlations for more effective and well justified decisions. To this end,this work proposes a framework where Graph Neural Networks (GNN) are adopted to explicitlyanalyze human–object interactions for more effectively recognizing daily activities. By automaticallyencoding the correlations among various interactions detected through some collected relational data,the framework infers the existence of different activities alongside their corresponding environmentalobjects. As a case study, we use the Toyota Smart Home dataset to evaluate the proposed framework.Compared with conventional feed-forward neural networks, the results demonstrate significantlysuperior performance in identifying ADL, allowing for the classification of different daily activitieswith an accuracy of 0.88. Furthermore, the incorporation of encoded information from relational dataenhances object-inference performance compared to the GNN without joint prediction, increasingaccuracy from 0.71 to 0.77. 

Prognostics and health management (PHM) is an engineering discipline that aims to maintain system behaviour and function and ensure mission success, safety and effectiveness. Addressing the challenges in prognostics and health management for modern intelligent systems, especially automated driving systems, is complex due to the contextual nature of faults. This complexity necessitates a thorough understanding of spatial, and temporal conditions, and relationships within operational scenarios and life-cycle stages. This paper introduces a framework designed to automatically recognize driving scenarios in automated driving systems using graph neural networks (GNNs). The framework extracts relational data from image frames, constructing graph-based models and transforming unstructured sensory data into structured data with diverse node types and relationships. A specific graph neural network processes the graph model to reveal and detect operational conditions and relationships. The proposed framework is evaluated using the KITTI dataset, demonstrating superior performance compared to conventional feed-forward networks such as MLP, particularly in handling relational data.

Anomaly detection is essential in Industrial Cyber-Physical Systems (ICPS) for monitoring both system and environmental conditions. However, effective anomaly detection remains a continuous challenge due to the complexity and diversity of operational features in both spatial and temporal domains of such systems. Current data-driven approaches utilize Artificial Intelligence (AI)-enabled methods, such as Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN), to process and analyze the collected operational data. From the industrial perspective, a key challenge lies in enhancing the effectiveness and explainability, as these methods often exhibit opaque and sophisticated network structures when dealing with complex operational data. Meanwhile, recent advancements in Graph Neural Networks (GNN) demonstrate its effectiveness of supporting the analysis of complex relationships and dependencies. The adoption of GNN poses however inherent challenges when the data are unstructured. This paper designs a novel framework that integrates Self-Organizing Maps (SOM) and GNN for the analysis of complex operational relationships and dependencies. In particular, the adoption of SOM allows the generation graph-structured data from unstructured raw datasets and thereby enhances a GNN’s ability to differentiate normal and anomalous conditions effectively. As case studies, we select multiple public datasets to compare the performance of the proposed framework with other benchmark methods. The results show that the proposed methods present promising results. Additionally, compared to other baseline methods that use GNN-based structures to detect anomalies, the proposed framework achieves the highest F1-Score.

Artificial Intelligence (AI) techniques through Learning-Enabled Components (LEC) are widely employed in Automated Driving Systems (ADS) to support operation perception and other driving tasks relating to planning and control. Therefore, the risk management plays a critical role in assuring the operational safety of ADS. However, the probabilistic and nondeterministic nature of LEC challenges the safety analysis. Especially, the impacts of their functional faults and incompatible external conditions are often difficult to identify. To address this issue, this article presents a simulation-aided approach as follows: 1) A simulation-aided operational data generation service with the operational parameters extracted from the corresponding system models and specifications; 2) A Fault Injection (FI) service aimed at high-dimensional sensor data to evaluate the robustness and residual risks of LEC. 3) A Variational Bayesian (VB) method for encoding the collected operational data and supporting an effective estimation of the likelihood of operational conditions. As a case study, the paper presents the results of one experiment, where the behaviour of an Autonomous Emergency Braking (AEB) system is simulated under various weather conditions based on the CARLA driving simulator. A set of fault types of cameras, including solid occlusion, water drop, salt and pepper, are modelled and injected into the perception module of the AEB system in different weather conditions. The results indicate that our framework enables to identify the critical faults under various operational conditions. To approximate the critical faults in undefined weather, we also propose Variational Autoencoder (VAE) to encode the pixel-level data and estimate the likelihood.

Anomaly detection plays a critical role in condition monitors to support the trustworthiness of Cyber-Physical Systems (CPS). Detecting multivariate anomalous data in such systems is challenging due to the lack of a complete comprehension of anomalous behaviors and features. This paper proposes a framework to address time series multivariate anomaly detection problems by combining the Self-Organizing Map (SOM) with Deep Reinforcement Learning (DRL). By clustering the multivariate data, SOM creates an environment to enable the DRL agents interacting with the collected system  operational data in terms of a tabular dataset. In this environment, Markov chains reveal the likely anomalous features to support the DRL agent exploring and exploiting the state-action space to maximize anomaly detection performance. We use a time series dataset, Skoltech Anomaly Benchmark (SKAB), to evaluate our framework. Compared with the best results by some currently applied methods, our framework improves the F1 score by 9%, from 0.67 to 0.73. 

Anomaly detection plays a critical role in condition monitors to support the trustworthiness of Cyber- Physical Systems (CPS). Detecting multivariate anomalous data in such systems is challenging due to the lack of a complete comprehension of anomalous behaviors and features. This paper proposes a framework to address time series multivariate anomaly detection problems by combining the Self-Organizing Map (SOM) with Deep Reinforcement Learning (DRL). By clustering the multivariate data, SOM creates an environment to enable the DRL agents interacting with the collected system operational data in terms of a tabular dataset. In this environment, Markov chains reveal the likely anomalous features to support the DRL agent exploring and exploiting the state-action space to maximize anomaly detection performance. We use a time series dataset, Skoltech Anomaly Benchmark (SKAB), to evaluate our framework. Compared with the best results by some currently applied methods, our framework improves the F1 score by 9% from 0.67 to 0.73.

Rare critical scenarios are crucial to verify the performance of autonomous driving in different situations. Critical scenario generation requires the ability of sampling critical combinations from an infinite parameter space in the logical scenario. Existing solutions aim to explore the correlation of action parameters in the initial scenario rather than action sequences. How to model action sequences so that one can further consider the effects of different action parameters is the bottleneck of the problem. In this paper, we solve the problem by proposing the ECSAS framework. Specifically, we first propose a description language, BTScenario, allowing us to model action sequences of scenarios. We then use reinforcement learning to search for combinations of critical action parameters. Several optimizations are proposed to increase efficiency, including action mask and replay buffer. Experimental results show that our model with strong collision ability and effectively outperforms the existing methods on various nontrivial scenarios.