Human–robot collaboration (HRC) relies on accurate and timely recognition of human intentions to ensure seamless interactions. Among common HRC tasks, human-to-robot object handovers have been studied extensively for planning the robot's actions during object reception, assuming the human intention for object handover. However, distinguishing handover intentions from other actions has received limited attention. Most research on handovers has focused on visually detecting motion trajectories, which often results in delays or false detections when trajectories overlap. This paper investigates whether human intentions for object handovers are reflected in non-movement-based physiological signals. We conduct a multimodal analysis comparing three data modalities: electroencephalogram (EEG), gaze, and hand-motion signals. Our study aims to distinguish between handover-intended human motions and non-handover motions in an HRC setting, evaluating each modality's performance in predicting and classifying these actions before and after human movement initiation. We develop and evaluate human intention detectors based on these modalities, comparing their accuracy and timing in identifying handover intentions. To the best of our knowledge, this is the first study to systematically develop and test intention detectors across multiple modalities within the same experimental context of human–robot handovers. Our analysis reveals that handover intention can be detected from all three modalities. Nevertheless, gaze signals are the earliest as well as the most accurate to classify the motion as intended for handover or non-handover.

My research focuses on adapting robotic failure explanations to enhance human-robot collaboration (HRC). I examine how different explanation types and explanation progression strategies impact failure resolution and user satisfaction by conducting a user study with multiple interaction rounds featuring repeated robotic failures and varying explanations. I also created a novel multimodal dataset of human responses to these failures and explanations. By analyzing human behavioral responses, I developed a predictor to anticipate user confusion following a specific robotic explanation at a robotic failure. This predictor enables an adaptive mechanism to dynamically adjust explanations based on user needs, fostering efficient and natural collaboration. This research aims to significantly improve overall user experience in HRC, making collaborations with robots smoother and more intuitive even when failures occur.

As robots become more capable with technology, their presence in human environments is expected to increase, leading to more physical and social interactions between humans and robots. In these shared spaces, handovers—the act of transferring an object from one person to another—constitute a significant part of daily human interactions. This thesis focuses on enhancing human-robot interaction by drawing inspiration from human-to-human handovers.

In this thesis, we investigate forces in human handovers to formulate adaptive robot grip release strategies, specifically addressing when a robot should release an object as a human recipient begins to take it during a handover. We developed a data-driven grip release strategy based on a dataset of recorded human-human handovers, which has been experimentally validated in human-robot interactions. To refine this strategy for different object weights, we recorded additional handovers involving various weights, resulting in publicly available datasets and a weight-adpative grip release strategy. Further, this thesis also examines how object weight affects human motion during handovers, enabling robots to observe changes in human motion to estimate object weights and adapt their motions to convey changes in object weights during handovers.

Additionally, we investigate the use of non-touch modalities, such as EEG brain signals and gaze tracking, to discern human intentions during handovers, specifically differentiating between motions intended for handovers and those that are not. 

Lastly, we also explore how human-robot handovers can be used to resolve robotic failures by providing explanations for these failures and adapting the explanations based on human behavioral responses.

När robotar blir mer kapabla med teknik förväntas deras närvaro i mänskliga miljöer öka, vilket leder till mer fysisk och social interaktion mellan människor och robotar. I dessa delade utrymmen utgör överlämningar – handlingen att överföra ett objekt från en person till en annan – en betydande del av den dagliga mänskliga interaktionen. Den här avhandlingen fokuserar på att förbättra interaktionen mellan människa och robot genom att hämta inspiration från överlämningar från människa till människa.

I det här examensarbetet undersöker vi krafter i mänskliga överlämningar för att formulera adaptiva robotgrepp-release-strategier, som specifikt tar upp när en robot ska släppa ett föremål när en mänsklig mottagare börjar ta det under en överlämning. Vi utvecklade en datadriven strategi för frigörande av grepp baserad på en datauppsättning av inspelade människa-människa överlämningar, som har experimentellt validerats i interaktioner mellan människa och robot. För att förfina denna strategi för olika objektvikter, spelade vi in ytterligare överlämningar som involverade olika vikter, vilket resulterade i allmänt tillgängliga datauppsättningar och en viktadpativ strategi för grepp-släpp. Vidare undersöker denna avhandling också hur objektvikt påverkar mänsklig rörelse under överlämningar, vilket gör det möjligt för robotar att observera förändringar i mänsklig rörelse för att uppskatta objektvikter och anpassa sina rörelser för att förmedla förändringar i objektvikter under överlämningar.

Dessutom undersöker vi användningen av icke-touch-modaliteter, såsom EEG-hjärnsignaler och blickspårning, för att urskilja mänskliga avsikter under överlämningar, specifikt skilja mellan rörelser avsedda för överlämningar och de som inte är det.

Slutligen undersöker vi också hur mänsklig-robot-överlämningar kan användas för att lösa robotfel genom att tillhandahålla förklaringar till dessa fel och anpassa förklaringarna baserat på mänskliga beteendesvar.

This work presents REFLEX: Robotic Explanations to FaiLures and Human EXpressions, a comprehensive multimodal dataset capturing human reactions to robot failures and subsequent explanations in collaborative settings. It aims to facilitate research into human-robot interaction dynamics, addressing the need to study reactions to both initial failures and explanations, as well as the evolution of these reactions in long-term interactions. By providing rich, annotated data on human responses to different types of failures, explanation levels, and explanation varying strategies, the dataset contributes to the development of more robust, adaptive, and satisfying robotic systems capable of maintaining positive relationships with human collaborators, even during challenges like repeated failures

This work presents REFLEX: Robotic Explanations to FaiLures and Human EXpressions, a comprehensive multimodal dataset capturing human reactions to robot failures and subsequent explanations in collaborative settings. It aims to facilitate research into human-robot interaction dynamics, addressing the need to study reactions to both initial failures and explanations, as well as the evolution of these reactions in long-term interactions. By providing rich, annotated data on human responses to different types of failures, explanation levels, and explanation varying strategies, the dataset contributes to the development of more robust, adaptive, and satisfying robotic systems capable of maintaining positive relationships with human collaborators, even during challenges like repeated failures.

To facilitate human-robot interaction (HRI), we aim for robot behavior that is efficient, transparent, and closely resembles human actions. Signal Temporal Logic (STL) is a formal language that enables the specification and verification of complex temporal properties in robotic systems, helping to ensure their correctness. STL can be used to generate explainable robot behaviour, the degree of satisfaction of which can be quantified by checking its STL robustness. In this letter, we use data-driven STL inference techniques to model human behavior in human-human interactions, on a handover dataset. We then use the learned model to generate robot behavior in human-robot interactions. We present a handover planner based on inferred STL specifications to command robotic motion in human-robot handovers. We also validate our method in a human-to-robot handover experiment.

Handovers are basic yet sophisticated motor tasks performed seamlessly by humans. They are among the most common activities in our daily lives and social environments. This makes mastering the art of handovers critical for a social and collaborative robot. In this work, we present an experimental study that involved human-human handovers by 13 pairs, i.e., 26 participants. We record and explore multiple features of handovers amongst humans aimed at inspiring handovers amongst humans and robots. With this work, we further create and publish a novel data set of 8672 handovers, which includes human motion tracking and the handover-forces. We further analyze the effect of object weight and the role of visual sensory input in human-human handovers, as well as possible design implications for robots. As a proof of concept, the data set was used for creating a human-inspired data-driven strategy for robotic grip release in handovers, which was demonstrated to result in better robot to human handovers.

Human-robot collaboration (HRC) relies on smooth and safe interactions. In this paper, we focus on the human-to-robot handover scenario, where the robot acts as a taker. We investigate the feasibility of detecting the intention of a human-to-robot handover action through the analysis of electroencephalogram (EEG) signals. Our study confirms that temporal patterns in EEG signals provide information about motor planning and can be leveraged to predict the likelihood of an individual executing a motor task with an average accuracy of 94.7%. We also suggest the effectiveness of the time-frequency features of EEG signals in the final second prior to the movement for distinguishing between handover action and other actions. Furthermore, we classify human intentions for different tasks based on time-frequency representations of pre-movement EEG signals and achieve an average accuracy of 63.5% for contrasting every two tasks against each other. The result encourages the possibility of using EEG signals to detect human handover intention in HRC tasks.

Despite significant improvements in robot capabilities, they are likely to fail in human-robot collaborative tasks due to high unpredictability in human environments and varying human expectations. In this work, we explore the role of explanation of failures by a robot in a human-robot collaborative task. We present a user study incorporating common failures in collaborative tasks with human assistance to resolve the failure. In the study, a robot and a human work together to fill a shelf with objects. Upon encountering a failure, the robot explains the failure and the resolution to overcome the failure, either through handovers or humans completing the task. The study is conducted using different levels of robotic explanation based on the failure action, failure cause, and action history, and different strategies in providing the explanation over the course of repeated interaction. Our results show that the success in resolving the failures is not only a function of the level of explanation but also the type of failures. Furthermore, while novice users rate the robot higher overall in terms of their satisfaction with the explanation, their satisfaction is not only a function of the robot's explanation level at a certain round but also the prior information they received from the robot.

To facilitate human-robot interaction and gain human trust, a robot should recognize and adapt to changes in human behavior. This work documents different human behaviors observed while taking objects from an interactive robot in an experimental study, categorized across two dimensions: pull force applied and handedness. We also present the changes observed in human behavior upon repeated interaction with the robot to take various objects.