With advances in Deep Neural Networks (DNN), Automated Driving Systems (ADS) enable the vehicle to perceive their surroundings in dynamic driving scenarios and perform behaviors by collecting operational data from sensors such as LiDAR and cameras. Current DNN typically detect objects by analyzing and classifying unstructured data (e.g., image data), providing critical information for ADS planning and decision-making. However, advanced ADS, particularly those required to perform the Dynamic Driving Task (DDT) autonomously, are expected to understand driving scenarios across various Operational Design Domains (ODD). This capability requires the support for a continuous comprehension of driving scenarios according to operational data collected by sensors. This paper presents a framework that adopts Graph Neural Networks (GNN) to describe and reason about dynamic driving scenarios via analyzing graph-based data based on collected sensor inputs. We first construct the graph-based data using a meta-path, which defines various interactions among different traffic participants. Next, we propose a design of GNN to support both the classification of the node types of objects and predicting relationships between objects. As results, the performance of the proposed method shows significant improvements compared to the baseline method. Specifically, the accuracy of node classification increases from 0.77 to 0.85, while that of relationships prediction rises from 0.74 to 0.82. To further utilize graph-based data constructed from dynamic driving scenarios, the proposed framework supports reasoning about operational risks by analyzing the observed nodes and relationships in the graph-based data. As a result, the model achieves a MRR of 0.78 in operational risks reasoning. To evaluate the practicality of the proposed framework in real-world systems, we also conduct a real-time performance evaluation by measuring the average process time and the Worst Case Execution Time (WCET). Compared to the baseline models, the results demonstrate the proposed framework presents acceptable real-time performance in analyzing graph-based data

With the advance of Artificial Intelligence, Deep Neural Networks are widely employed in various sensor-based systems to analyze operational conditions. However, due to the inherently nondeterministic and probabilistic natures of neural networks, the assurance of overall system performance could become a challenging task. In particular, soft errors could weaken the robustness of such networks and thereby threaten the system's safety. Conventional fault-tolerant techniques by means of hardware redundancy and software correction mechanisms often involve a tricky trade-off between effectiveness and scalability in addressing the extensive design space of Deep Neural Networks. In this work, we propose a Reinforcement-Learning-based approach to protect neural networks from soft errors by addressing and identifying the vulnerable bits. The approach consists of three key steps: (1) analyzing layer-wise resiliency of Deep Neural Networks by a fault injection simulation; (2) generating layer-wise bit masks by a Reinforcement-Learning-based agent to reveal the vulnerable bits and to protect against them; and (3) synthesizing and deploying bit masks across the network with guaranteed operation efficiency by adopting transfer learning. As a case study, we select several existing neural networks to test and validate the design. The performance of the proposed approach is compared with the performance of other baseline methods, including Hamming code and the Most Significant Bits protection schemes. The results indicate that the proposed method exhibits a significant improvement. Specifically, we observe that the proposed method achieves a significant performance gain of at least 10% to 15% over on the test network. The results indicate that the proposed method dynamically and efficiently protects the vulnerable bits compared with the baseline methods.

With globalization and outsourcing trends, modern industrial companies often rely on an extensive network of suppliers to construct sophisticated products. To maintain effective production planning and scheduling, integrating and managing the extensive information from supply chain has become increasingly critical. In particular, industrial companies, particularly those aiming to achieve Industry 4.0, enable to collect and analyze data related to their supply chains. Due to the vast amount of collected data, there is a continuous challenge in integrating and analyzing dependencies within the supplier network. While the development of Artificial Intelligence (AI) offers a promising solution for extracting and analyzing features from data, the inherently opaque and training-intensive nature of AI-enabled methods still present obstacles to effectively and efficiently analyzing information. To cope with this issue, this paper presents a knowledge-enhanced framework to support supply chain information integration and analysis by combining Knowledge Base (KB) and Graph Neural Networks (GNN). Specifically, constructing a KB enables the integration of extensive collected data with domain knowledge to generate structured and relational information. These knowledge-enhanced data support the training of GNN to encode information about supply chains. The resulting embeddings enable multiple inference tasks for analyzing graph-based data, supporting supply chain management. The case studies cover the usage of encoded embeddings for node classification, link prediction, and scenario classification. The proposed GNN outperforms baseline methods, demonstrating a promising solution for analyzing graph-based data in the context of supply chain management.

Global warming is a critical issue today, largely due to the widespread use of fossil fuels in everyday life. One promising solution to reduce reliance on conventional energy sources is to promote the use of renewable power. In particular, to encourage the use of renewable energy in industrial sectors which involve development and manufacture of the industrial artifacts, there is continuous demand for tracing energy sources within the production processes. However, given a sophisticated industrial product that involves diverse and extensive components and their suppliers, the traceability analysis across its production is a critical challenge for ensuring the full utilization of renewable energy. To alleviate this issue, this paper presents a functional framework to support tracing the usage of renewable energy by integrating the Large Language Models (LLMs) and logic programming across supply chain networks. Specifically, the proposed framework contains the following components: (1) adopting graph-based models to process and manage the extensive information within supply chain networks; (2) using the Retrieval-Augmented Generation (RAG) techniques to support the LLM for processing the information related to supply chain networks and generating relevant responses with structured representations; and (3) presenting a logic programming-based solution to support the traceability analysis of renewable energy regarding the responses from the LLM. As a case study, we use a public dataset to evaluate the proposed framework by comparing it to the RAG-based LLM and its variant. Compared to baseline methods solely relying on LLMs, the experiments show that the proposed framework achieves significant improvement.

Prognostics and Health Management (PHM) services play a critical role in maintaining system performance by monitoring and managing the status of industrial Cyber-Physical Systems (CPS). These services typically involve analyzing multi-dimensional data from various sensors to extract spatial and temporal features. The advancement of data-driven methods, such as Machine Learning (ML) and Deep Learning (DL), provides an end-to-end solution for straightforwardly manipulating sensory data and learning patterns. In addition to these methods, the rise of Large Language Models (LLM) also facilitates the analysis and comprehension of multi-dimensional data. Compared to ML/DL models, LLM demonstrate powerful capabilities due to their extensive training data and cross-domain knowledge. However, continuous challenges remain in seamlessly detecting anomalies by integrating domain knowledge with these methods to effectively and efficiently monitor system status for health management (e.g., fault propagation via causality analysis). To alleviate this issue, this paper presents a functional framework to support the health management in CPS by leveraging the integration of LLM and domain knowledge. As case studies, this work uses the Tennessee Eastman (TE) process and an Ultra-Processed (UP) food manufacturing plant to evaluate the performance of integrating LLM with domain knowledge by identifying and analyzing the causality of collected data across sensors. Compared to baseline methods that rely solely on the LLM, the results show an increase in F1-score from 0.15 to 0.62 for the TE process, and from 0.05 to 0.64 for the UP process.

Modern companies often rely on integrating an extensive network of suppliers to organize and produce industrial artifacts. Within this process, it is critical to maintain sustainability and flexibility by analyzing and managing information from the supply chain. In particular, there is a continuous demand to automatically analyze and infer information from extensive datasets structured in various forms, such as natural language and domain-specific models. The advancement of Large Language Models (LLM) presents a promising solution to address this challenge. By leveraging prompts that contain the necessary information provided by humans, LLM can generate insightful responses through analysis and reasoning over the provided content. However, the quality of these responses is still affected by the inherent opaqueness of LLM, stemming from their complex architectures, thus weakening their trustworthiness and limiting their applicability across different fields. To address this issue, this work presents a framework to leverage the graph-based LLM to support the analysis of supply chain information by combining the LLM and domain knowledge. Specifically, this work proposes an integration of LLM and domain knowledge to support an analysis of the supply chain as follows: (1) constructing a graph-based knowledge base to describe and model the domain knowledge; (2) creating prompts to support the retrieval of the graph-based models and guide the generation of LLM; (3) generating responses via LLM to support the analysis and reason about information across the supply chain. We demonstrate the proposed framework in the tasks of entity classification, link prediction, and reasoning across entities. Compared to the average performance of the best methods in the comparative studies, the proposed framework achieves a significant improvement of 59%, increasing the ROUGE-1 F1 score from 0.42 to 0.67.

Cyber-Physical Systems (CPS) refer to intelligent systems that combine computational and physical capabilities to enable advanced functionalities, such as autonomous behaviors, human-machine interaction, and machine collaboration in complex environments. Addressing these functionalities often necessitates the adoption of Artificial Intelligence (AI) techniques which are extensively utilized for operational perception and decision-making. However, due to the inherently data-intensive and opaque nature of most AI-enabled components, combined with unforeseen environments, the integration of AI-enabled techniques into CPS presents significant engineering challenges in quality management. Self-management, as an embedded system feature, extends conventional CPS with capabilities for operation monitoring, planning and adaptation. It is often considered as a necessary mechanism for ensuring the quality and trustworthiness of AI-enabled components. However, implementing self-management in AI-enabled CPS presents several challenges: The concept of self-management varies depending on the audience and application, making its definition and implementation more complex. Additionally, the data-intensive nature of AI-enabled components requires extra effort to ensure consistent performance across operational domains, especially under unforeseen conditions. To cope with these challenges, this thesis proposes to integrate data-driven and knowledge-enabled methods for self-management through the following efforts: 1) Proposing conceptual frameworks that define the necessary and sufficient functionalities for self-management, with the support for situation awareness regarding the internal and environmental conditions; 2) Developing condition monitoring modules within the proposed conceptual frameworks for the situation-awareness to analyze system status; 3) Creating human-explainable data-driven methods for understanding operational conditions; 4) Designinglearning-based agents to dynamically and effectively address vulnerabilities inAI-enabled systems that could lead to system failures or compromise the operational safety. This thesis consolidates key concepts and introduces novel features for self-management in CPS by synthesizing insights from existing research and addressing their limitations. It provides a framework for designing learning based agents that leverage data synthesis to achieve self-management. Additionally, the thesis develops data-driven methods integrated with knowledge enabled models to enhance situation awareness and trustworthiness, effectively addressing the complexity and opacity of AI-enabled computing processes.

Cyberfysiska system (CPS) är intelligenta system som kombinerar beräknings-mässiga och fysiska förmågor för att möjliggöra avancerade funktioner, som autonoma beteenden, interaktion mellan människa och maskin och maskinsamarbete i komplexa miljöer. För att hantera dessa funktionaliteter används ofta artificiell intelligens (AI), särskilt inom operativ perception och beslutsfattande. Men på grund av att de flesta AI-komponenter är dataintensiva och svårgenomtränglig, i kombination medoförut-sedda miljöer, innebär integrationen av AI-komponenter i CPS betydande tekniska utmaningar när det gäller kvalitetshantering.

Självförvaltning, som en inbyggd systemfunktion, utökar konventionella CPS med funktionaliteter för driftsövervakning, planering och anpassning. Den anses ofta vara en nödvändig mekanism för att säkerställa kvaliteten och pålitligheten hos AI-komponenter.

Att implementera självförvalt-ning i AI-aktiverade CPS innebär dock flera utmaningar: Begreppet tolkas olika beroende på målgrupp och tillämpning, vilket gör definitionen och implementeringen mer komplex. Dessutom kräver AI-komponenter ofta extra ansträngningar för att säkerställa konsekvent prestanda över olika operativa domäner, särskilt under oförutsedda förhållanden.

För att adressera dessa utmaningar föreslår denna avhandling en integrering av datadrivna och kunskapsbaserade metoder för självförvaltning genom följande insatser: 1) Utveckling av konceptuella ramverk som definierar de nödvändiga och tillräckliga funktionerna för autonom hantering, med stöd för situationsmedvetenhet om både interna och externa förhållandena; 2) Implementering av tillståndsövervakningsmoduler inom de föreslagna konceptuella ramverken för att analysera systemstatus och förbättra situationsmedvetenheten; 3) Skapande av förklarbara datadrivna metoder som möjliggör en bättre förståelse av operativa förhållanden; 4) Design av inlärningsbaserade agenter för att dynamiskt och effektivt hantera sårbarheter i AI-baserade system, förebygga systemfel och säkerställa driftsäkerhet.

Denna avhandling samlar nyckelbegrepp och introducerar nya funktioner för autonom hantering i cyberfysiska system genom att sammanställa insikter från befintliga studier och adressera deras begränsningar. Den presenterar ett ramverk för att utforma inlärningsbaserade agenter som utnyttjar datasyntes för att uppnå autonom hantering. Vidare utvecklas datadrivna metoder som kombineras med kunskapsbaserade modeller för att förbättra situationsmedvetenhet och pålitlighet, samtidigt som de komplexa och svårgenomträngliga databehandlingsprocesserna i AI-drivna komponenter hanteras.

This paper presents a reconfigurable near-sensor anomaly detection processor to real-time monitor the potential anomalous behaviors of amputees with limb prostheses. The processor is low-power, low-latency, and suitable for equipment on the prostheses and comprises a reconfigurable Variational Autoencoder (VAE), a scalable Self-Organizing Map (SOM) Array, and a window-size-adjustable Markov Chain, which can implement an integrated miniaturized anomaly detection system. With the reconfigurable VAE, the proposed processor can support up to 64 sensor sampling channels programmable by global configuration, which can meet the anomaly detection requirements in different scenarios. A scalable SOM array allows for the selection of different sizes based on the complexity of the data. Unlike traditional time accumulation-based anomaly detection methods, the Markov Chain is utilized to detect time-series-based anomalous data. The processor is designed and fabricated in a UMC 40-nm LP technology with a core area of 1.49 mm2 and a power consumption of 1.81 mW. It achieves real-time detection performance with 0.933 average F1 Score for the FSP dataset within 24.22 s, and 0.956 average F1 Score for the SFDLA-12 dataset within 30.48 s, respectively. The energy dissipation of detection for each input feature is 43.84 nJ with the FSP dataset, and 55.17 nJ with the SFDLA-12 dataset. Compared with ARM Cortex-M4 and ARM Cortex-M33 microcontrollers, the processor achieves energy and area efficiency improvements ranging from 257×, 193× and 11×, 8×, respectively. IEEE

Human Activity Recognition (HAR) refers to a field that aims to identify human activitiesby adopting multiple techniques. In this field, different applications, such as smart homes andassistive robots, are introduced to support individuals in their Activities of Daily Living (ADL)by analyzing data collected from various sensors. Apart from wearable sensors, the adoption ofcamera frames to analyze and classify ADL has emerged as a promising trend for achieving theidentification and classification of ADL. To accomplish this, the existing approaches typically rely onobject classification with pose estimation using the image frames collected from cameras. Given theexistence of inherent correlations between human–object interactions and ADL, further efforts areoften needed to leverage these correlations for more effective and well justified decisions. To this end,this work proposes a framework where Graph Neural Networks (GNN) are adopted to explicitlyanalyze human–object interactions for more effectively recognizing daily activities. By automaticallyencoding the correlations among various interactions detected through some collected relational data,the framework infers the existence of different activities alongside their corresponding environmentalobjects. As a case study, we use the Toyota Smart Home dataset to evaluate the proposed framework.Compared with conventional feed-forward neural networks, the results demonstrate significantlysuperior performance in identifying ADL, allowing for the classification of different daily activitieswith an accuracy of 0.88. Furthermore, the incorporation of encoded information from relational dataenhances object-inference performance compared to the GNN without joint prediction, increasingaccuracy from 0.71 to 0.77. 

Prognostics and health management (PHM) is an engineering discipline that aims to maintain system behaviour and function and ensure mission success, safety and effectiveness. Addressing the challenges in prognostics and health management for modern intelligent systems, especially automated driving systems, is complex due to the contextual nature of faults. This complexity necessitates a thorough understanding of spatial, and temporal conditions, and relationships within operational scenarios and life-cycle stages. This paper introduces a framework designed to automatically recognize driving scenarios in automated driving systems using graph neural networks (GNNs). The framework extracts relational data from image frames, constructing graph-based models and transforming unstructured sensory data into structured data with diverse node types and relationships. A specific graph neural network processes the graph model to reveal and detect operational conditions and relationships. The proposed framework is evaluated using the KITTI dataset, demonstrating superior performance compared to conventional feed-forward networks such as MLP, particularly in handling relational data.