Integrating Large Language Model for Natural Language-Based Instruction toward Robust Human-Robot CollaborationShow others and affiliations
2024 (English)Conference paper, Published paper (Refereed)
Abstract [en]
Human-Robot Collaboration (HRC) aims to create environments where robots can understand workspace dynamics and actively assist humans in operations, with the human intention recognition being fundamental to efficient and safe task fulfillment. Language-based control and communication is a natural and convenient way to convey human intentions. However, traditional language models require instructions to be articulated following a rigid, predefined syntax, which can be unnatural, inefficient, and prone to errors. This paper investigates the reasoning abilities that emerged from the recent advancement of Large Language Models (LLMs) to overcome these limitations, allowing for human instructions to be used to enhance human-robot communication. For this purpose, a generic GPT 3.5 model has been fine-tuned to interpret and translate varied human instructions into essential attributes, such as task relevancy and tools and/or parts required for the task. These attributes are then fused with perceived on-going robot action to generate a sequence of relevant actions. The developed technique is evaluated in a case study where robots initially misinterpreted human actions and picked up wrong tools and parts for assembly. It is shown that the fine-tuned LLM can effectively identify corrective actions across a diverse range of instructional human inputs, thereby enhancing the robustness of human-robot collaborative assembly for smart manufacturing.
Place, publisher, year, edition, pages
Elsevier BV , 2024. p. 313-318
Keywords [en]
Error correction, Human-robot collaboration, Large language model, Natural language processing
National Category
Production Engineering, Human Work Science and Ergonomics Robotics and automation
Identifiers
URN: urn:nbn:se:kth:diva-358215DOI: 10.1016/j.procir.2024.10.093Scopus ID: 2-s2.0-85214970201OAI: oai:DiVA.org:kth-358215DiVA, id: diva2:1924849
Conference
18th IFAC Workshop on Time Delay Systems, TDS 2024, Udine, Italy, October 2-5, 2023
Note
QC 20250114
2025-01-072025-01-072025-02-05Bibliographically approved