In the context of human-centric smart manufacturing, human-robot collaboration (HRC) systems leverage the strengths of both humans and machines to achieve more flexible and efficient manufacturing. In particular, estimating and monitoring human motion status determines when and how the robots cooperate. However, the presence of occlusion in industrial settings seriously affects the performance of human pose estimation (HPE). Using more sensors can alleviate the occlusion issue, but it may cause additional computational costs and lower workers' comfort. To address this issue, this work proposes a visual-inertial fusion-based method for HPE in HRC, aiming to achieve accurate and robust estimation while minimizing the influence on human motion. A part-specific cross-modal fusion mechanism is designed to integrate spatial information provided by a monocular camera and six Inertial Measurement Units (IMUs). A multi-scale temporal module is developed to model the motion dependence between frames at different granularities. Our approach achieves 34.9 mm Mean Per Joint Positional Error (MPJPE) on the TotalCapture dataset and 53.9 mm on the 3DPW dataset, outperforming state-of-the-art visual-inertial fusion-based methods. Tests on a synthetic-occlusion dataset further validate the occlusion robustness of our network. Quantitative and qualitative experiments on a real assembly case verified the superiority and potential of our approach in HRC. It is expected that this work can be a reference for human motion perception in occluded HRC scenarios.

Digital Twin (DT) of a manufacturing system mainly involving materials and machines has been widely explored in the past decades to facilitate the mass customization of modern products. Recently, the new vision of Industry 5.0 has brought human operators back to the core part of work cells. To this end, designing human-centric DT systems is vital for an ergonomic and symbiotic working environment. However, one major challenge is the construction and utilization of high-fidelity digital human models. In the literature, preset universal human avatar models such as skeletons are mostly employed to represent the human operators, which overlooks the individual differences of physical traits. Besides, the fundamental utilization features such as motion tracking and procedure recognition still do not well address the practical issues such as occlusions and incomplete observations. To deal with the challenge, this paper proposes a systematic design framework to quickly and precisely build and utilize the human-centric DT systems. The mesh-based customized human operator models with rendered appearances are first generated within one minute from a short motion video. Then transformer-based deep learning networks are developed to realize the motion-related operator status synchronization in complex conditions. Extensive experiments on multiple real-world human–robot collaborative work cells show the superior performance of the proposed framework over the state-of-the-art.

At present, the tool remaining useful life prediction technology is important to the effectiveness of machining, because tool life prediction plays the role of safety maintenance, cost optimization and quality assurance. However, this the technology faces many challenges in practical applications. The main problems include that when the spatial distribution of data features is too different, the model is difficult to adapt to multi-scene data and the feature extraction of data time series is not obvious. Therefore, this paper proposes a method for predicting the remaining useful life of milling cutters by using multi-scale spatial data feature visualization and domain separation prediction network (MTF-SE-DSPNs). Firstly, the one-dimensional time series data are globally normalized by this method, and then the processed data are transformed into images by MTF, which enhances the time series features expression ability of data. At the same time, the convolutional neural network based on DenseNet architecture is used and SElayer is added to adjust the feature weight to mine the sensitive information in the signal. To improve the prediction ability of the model, the time decay factor ξT is introduced to optimize the reconstruction loss, so that it can dynamically measure the relative importance of source domain and target domain data and improve the robustness of feature information reconstruction. Finally, the effectiveness of the method is validated by milling experiments under the same and different working conditions. The experimental results are compared with the other models, which proves the significant advantages of the model in various tasks.

Industrial robots (IRs) serve as critical equipment in advanced manufacturing systems. Building high-fidelity digital twin models of IRs is essential for various applications like precision simulation, and intelligent operation and maintenance. Despite technological potentials of digital twins, existing modeling methods for industrial robot digital twins (IRDTs) predominantly focus on isolated domains. This fails to address inherent multi-domain complexities of IRs that arise from their integrated mechanical-electrical-control characteristic. To bridge this gap, first, this study proposes a multi-level multi-domain (MLMD) digital twin modeling framework and method. The framework systematically integrates physical space, digital space, and their bidirectional interactions, while explicitly defining hierarchical structures and cross-domain mechanisms. Subsequently, a four-step method is established, which encompasses component analysis, parameter extraction, MLMD IRDT modeling based on function blocks (FBs), and model validation. Then, implementation details are illustrated through an SD3/500 IR case study, where domain-specific modeling techniques and cross-domain integration mechanisms are systematically analyzed. Finally, effectiveness and feasibility of the proposed method is validated through experiments.

In order to address the problems with current robotic automated assembly such as limitations of model-based methods in unstructured assembly scenarios, low training efficiency of learning-based methods, and limited performance of policy generalization, this paper proposes two modeling methodologies based on deep reinforcement learning under the overall framework of phased assembly for complex robotic assembly tasks, i.e., separated-phased policy modeling (SPM) and integrated policy modeling (IPM). Regarding policy learning with deep reinforcement learning, we present a refined SAC algorithm that merges a policy-lead mechanism and an LSTM network (i.e., PL-LSTM-SAC). A comprehensive testbed based on the assembly of a triple-task planetary gear train is designed to validate the framework and the proposed approach. Experimental results indicate that the trained assembly policies for each task are effective under both policy modeling methodologies, but SPM has higher stability and policy convergence efficiency than IPM. Physical tests indicate the sim-to-real transferability of the trained policies with both SPM and IPM and an average success rate of 92.0 % is achieved. The PL-LSTM-SAC algorithm proposed can significantly accelerate training speed and enhance compliance and overall performance of assembly actions by a 13.9 % reduction in the average contact force during assembly processes.

In the process of metal cutting, realizing effective monitoring of tool wear is of great significance to ensure the quality of parts machining. To address the tool wear monitoring (TWM) problem, a tool wear monitoring method based on data-driven and physical output is proposed. The method divides two Physical models (PM) into multiple stages according to the tool wear in real machining scenarios, making the coefficients of PM variable. Meanwhile, by analyzing the monitoring capabilities of different PMs at each stage and fusing them, the PM's ability to deal with complex nonlinear relationships, which is difficult to handle, is improved, and the flexibility of the model is improved; The pre-processed signal data features were extracted, and the original features were fused and downscaled using Stacked Sparse Auto-Encoder (SSAE) networker to build a data-driven model (DDM). At the same time, the DDM is used as a guidance layer to guide the fused PM for the prediction of wear amount at each stage of the tool, which enhances the interpretability of the monitoring model. The experimental results show that the proposed method can realize the accurate monitoring of tool wear, which has a certain reference value for the flexible tool change in the actual metal-cutting process.

During cutting operations, tool condition monitoring (TCM) is essential for maintaining safety and cost optimization, especially in the accelerated tool wear phase. Due to the safety constraints of the actual production environment and the tool's properties, the data for each wear stage is usually unbalanced, and these unbalances lead to difficulties in failure monitoring. To this end, a novel TCM method based on data augmentation is proposed, which uses generative adversarial networks (GANs) to generate valuable artificial samples for a few classes to balance the data distribution. Unlike the traditional GANs, the proposed Conditional Wasserstein GAN-Gradient Penalty (CWGAN-GP) avoids pattern collapse and training instability and simultaneously generates more realistic data and signal samples with labels for different wear states. To evaluate the quality of the generated data, an evaluation index is proposed to evaluate the generated data while further screening the samples to achieve effective oversampling. Finally, the continuous wavelet transform (CWT) is combined with the convolutional neural network (CNN) architecture of Inception-ResNet-v2 for TCM, and it is demonstrated that data augmentation can effectively improve the performance of training classification models for unbalanced data by comparing three classification methods with two data augmentation experiments, and the proposed method has a better monitoring performance.

Augmented Reality (AR) innovates product assembly by overlaying intuitive visual instructions onto the physical workspace, improving operational efficiency. However, existing AR-guided assembly methods has the following drawbacks. Most studies lack consideration of the dynamic adaptation of operators’ cognitive abilities. Excessive or insufficient information push may cause cognitive overload or information underutilization, which has instead a negative impact on assembly work. In addition, due to lack of automatic perceiving assembly state, frequent human-interface interactions interferences the workflow continuity, which is particularly disadvantageous for manual assembly work. To address the problems, we propose an intelligent AR-guided adaptive information push (AR-AIP) method based on context-awareness, which personalizes information push according to individual cognitive abilities. Firstly, the method comprehensively assesses human cognitive abilities by separately considering factors such as fatigue state and skill level. Secondly, it quantifies the information load associated with various visual representations of AR content, aiming to identify the most suitable combination that aligns with each user's cognitive abilities. Thirdly, the perception of the assembly process and its state is fundamental for achieving adaptive content push. This study, therefore, employs context awareness by integrating the recognition of assembly actions and the detection of parts, facilitating the automatic inference of the assembly process and its state. The AR-AIP aims at three “rights”, i.e., push the right information in the right AR representation forms to the operator at the right time, proactively and automatically, with minimal manual interactions. This approach is designed to alleviate the cognitive burden on operators, thereby enhancing assembly quality and efficiency. A comparative study on electronic equipment assembly shows that compared to traditional AR methods, AR-AIP significantly improves task completion time, reduces errors, lowers cognitive load, enhances user experience while minimizing skill and fatigue variations. The research findings offer insights for designing AR visual content to assist assembly tasks, providing a new approach for proactive perception and understanding of real-time assembly states.

Dynamic order picking has usually demonstrated significant impacts on production efficiency in warehouse management. In the context of an automotive-part warehouse, this paper addresses a dynamic multi-tour order-picking problem based on a novel attention-aware deep reinforcement learning-based (ADRL) method. The multi-tour represents that one order-picking task must be split into multiple tours due to the cart capacity and the operator’s workload constraints. First, the multi-tour order-picking problem is formulated as a mathematical model, and then reformulated as a Markov decision process. Second, a novel DRL-based method is proposed to solve it effectively. Compared to the existing DRL-based methods, this approach employs multi-head attention to perceive warehouse situations. Additionally, three improvements are proposed to further strengthen the solution quality and generalization, including (1) the extra location representation to align the batch length during training, (2) the dynamic decoding to integrate real-time information of the warehouse environment during inference, and (3) the proximal policy optimization with entropy bonus to facilitate action exploration during training. Finally, comparison experiments based on thousands of order-picking instances from the Swedish warehouse validated that the proposed ADRL could outperform the other twelve DRL-based methods at most by 40.6%, considering the optimization objective. Furthermore, the performance gap between ADRL and seven evolutionary algorithms is controlled within 3%, while ADRL can be hundreds or thousands of times faster than these EAs regarding the solving speed.

The proliferation of data silos poses a significant impediment to the advancement of machine learning applications. The traditional approach of centralized data learning is becoming increasingly impractical in certain domains, primarily due to escalating concerns over data privacy and security. Particularly in the manufacturing sector, the integration of Federated Learning (FL) presents a promising avenue for safeguarding collaborative data mining efforts across a network of distributed manufacturers. This paper offers an in-depth review of research about FL in the realms of smart manufacturing and product lifecycle management. We elucidate the imperative need for FL applications from a socio-technical systems perspective, underscoring the interplay between societal and technological factors. Subsequently, we delve into the categorization of FL methodologies and their pivotal enablers, contextualized within the framework of manufacturing engineering. This paper further presents a comprehensive overview of FL applications, complemented by an analysis of the key performance metrics that are germane to the manufacturing industry. In conclusion, we engage in a discourse on the technical challenges, societal barriers, and prospective research trajectories for FL. Our discussion is anchored towards the emerging paradigm of Industry 5.0, which envisions a future where resilient, human-centric, and sustainable manufacturing systems are seamlessly integrated with cutting-edge digital technologies.