Preference-based reinforcement learning (PbRL) is a suitable approach for style adaptation of pre-trained robotic behavior: adapting the robot's policy to follow human user preferences while still being able to perform the original task. However, collecting preferences for the adaptation process in robotics is often challenging and time-consuming. In this work we explore the adaptation of pre-trained robots in the low-preference-data regime. We show that, in this regime, recent adaptation approaches suffer from catastrophic reward forgetting (CRF), where the updated reward model overfits to the new preferences, leading the agent to become unable to perform the original task. To mitigate CRF, we propose to enhance the original reward model with a small number of parameters (low-rank matrices) responsible for modeling the preference adaptation. Our evaluation shows that our method can efficiently and effectively adjust robotic behavior to human preferences across simulation benchmark tasks and multiple real-world robotic tasks. We provide videos of our results and source code at https://sites.google.com/view/preflora/

Reinforcement learning policies are becoming increasingly prevalent in robotics and AI-human interactions due to their effectiveness in tackling complex and challenging domains. Many of these policies—also referred to as AI agents—are trained using human feedback through techniques collectively known as Reinforcement Learning from Human Feedback (RLHF). This thesis addresses three key challenges—safety, alignment, and efficiency—that arise when deploying these policies in real-world applications involving actual human users. To this end, it proposes several novel methods. Ensuring the safety of human-robot interaction is a fundamental requirement for their deployment. While most prior research has explored safety within discrete state and action spaces, we investigate novel approaches for synthesizing safety shields from human feedback, enabling safer policy execution in various challenging settings, including continuous state and action spaces, such as social navigation. To better align policies with human feedback, contemporary works predominantly rely on single-reward settings. However, we argue for the necessity of a multi-objective paradigm, as most human goals cannot be captured by a single valued reward function. Moreover, most robotic tasks have baseline predefined goals related to task success, such as reaching a navigation waypoint. Accordingly, we first introduce a method to align policies with multiple objectives using pairwise preferences. Additionally, we propose a novel multi-modal approach that leverages zero-shot reasoning with large language models alongside pairwise preferences to adapt multi-objective goals for these policies. The final challenge addressed in this thesis is improving the sample efficiency and reusability of these policies, which is crucial when adapting policies based on real human feedback. Since requesting human feedback is both costly and burdensome—potentially degrading the quality of human-agent interactions—we propose two distinct methods to mitigate these issues. First, to enhance the efficiency of RLHF, we introduce an active learning method that combines unsupervised learning techniques with uncertainty estimation to prioritize the most informative queries for human feedback. Second, to improve the reusability of reward functions derived from human feedback and reduce the need for redundant queries in similar tasks, we investigate low-rank adaptation techniques for adapting pre-trained reward functions to new tasks.

Reinforcement learning-policyer blir allt vanligare inom robotik och AI-mänsklig interaktion tack vare deras effektivitet i att hantera komplexa och utmanande domäner. Många av dessa policyer – även kallade AI-agenter – tränas med hjälp av mänsklig återkoppling genom tekniker som kollektivt benämns Reinforcement Learning from Human Feedback (RLHF). Denna avhandling tar upp tre centrala utmaningar – säkerhet, anpassning och effektivitet – som uppstår vid implementering av dessa policyer i verkliga tillämpningar som involverar faktiska mänskliga användare. För detta ändamål föreslås flera nya metoder. Att säkerställa säkerheten i människa-robot-interaktion är en grundläggande förutsättning för deras implementering. Medan tidigare forskning främst har undersökt säkerhet inom diskreta tillstånds- och aktionsrum, undersöker vi nya metoder för att syntetisera säkerhetssköldar utifrån mänsklig återkoppling, vilket möjliggör säkrare policyutförande i olika utmanande miljöer, inklusive kontinuerliga tillstånds- och aktionsrum, såsom social navigation. För att bättre anpassa policyer till mänsklig återkoppling förlitar sig moderna arbeten huvudsakligen på inställningar med enstaka belöningar. Vi argumenterar dock för behovet av ett multiobjektivparadigm, eftersom de flesta mänskliga mål inte kan fångas av en belöningsfunktion med ett enda värde. Dessutom har de flesta robotuppgifter fördefinierade basmål kopplade till uppgiftsframgång, såsom att nå en navigationspunkt. Följaktligen introducerar vi först en metod för att anpassa policyer till flera mål genom parvisa preferenser. Dessutom föreslår vi en ny multimodal metod som utnyttjar zeroshot-reasoning med stora språkmodeller tillsammans med parvisa preferenser för att anpassa multiobjektiva mål för dessa policyer. Den sista utmaningen som behandlas i denna avhandling är att förbättra sampeleffektiviteten och återanvändbarheten hos dessa policyer, vilket är avgörande vid anpassning av policyer baserat på verklig mänsklig återkoppling. Eftersom insamling av mänsklig återkoppling både är kostsamt och betungande – och potentiellt försämrar kvaliteten på människa-agent-interaktioner – föreslår vi två olika metoder för att minska dessa problem. För det första introducerar vi en aktiv inlärningsmetod för att förbättra effektiviteten av RLHF genom att kombinera osuperviserade inlärningstekniker med osäkerhetsuppskattning för att prioritera de mest informativa förfrågningarna om mänsklig återkoppling. För det andra undersöker vi low-rank-anpassningstekniker för att anpassa förtränade belöningsfunktioner till nya uppgifter, vilket förbättrar återanvändbarheten av belöningsfunktioner från mänsklig återkoppling och minskar behovet av redundanta förfrågningar i liknande uppgifter.

Many solutions to address the challenge of robot learning have been devised, namely through exploring novel ways for humans to communicate complex goals and tasks in reinforcement learning (RL) setups. One way that experienced recent research interest directly addresses the problem by considering human feedback as preferences between pairs of trajectories (sequences of state-action pairs). However, when simply attributing a single preference to a pair of trajectories that contain many agglomerated steps, key pieces of information are lost in the process. We amplify the initial definition of preferences to account for highlights: state-action pairs of relatively high information (high/low reward) within a preferred trajectory. To include the additional information, we design novel regularization methods within a preference learning framework. To this extent, we present our method which is able to greatly reduce the necessary amount of preferences, by permitting the highlighting of favoured trajectories, in order to reduce the entropy of the credit assignment. We show the effectiveness of our work in both simulation and a user study, which analyzes the feedback given and its implications. We also use the total collected feedback to train a robot policy for socially compliant trajectories in a simulated social navigation environment. We release code and video examples at https://sites.google.com/view/rl-polite

Preference-based reinforcement learning (RL) has emerged as a new field in robot learning, where humans play a pivotal role in shaping robot behavior by expressing preferences on different sequences of state-action pairs. However, formulating realistic policies for robots demands responses from humans to an extensive array of queries. In this work, we approach the sample-efficiency challenge by expanding the information collected per query to contain both preferences and optional text prompting. To accomplish this, we leverage the zero-shot capabilities of a large language model (LLM) to reason from the text provided by humans. To accommodate the additional query information, we reformulate the reward learning objectives to contain flexible highlights - state-action pairs that contain relatively high information and are related to the features processed in a zero-shot fashion from a pretrained LLM. In both a simulated scenario and a user study, we reveal the effectiveness of our work by analyzing the feedback and its implications. Additionally, the collective feedback collected serves to train a robot on socially compliant trajectories in a simulated social navigation landscape. We provide video examples of the trained policies at https://sites.google.com/view/rl-predilect.

Preference-based reinforcement learning (RL) poses as a recent research direction in robot learning, by allowing humans to teach robots through preferences on pairs of desired behaviours. Nonetheless, to obtain realistic robot policies, an arbitrarily large number of queries is required to be answered by humans. In this work, we approach the sample-efficiency challenge by presenting a technique which synthesizes queries, in a semi-supervised learning perspective. To achieve this, we leverage latent variational autoencoder (VAE) representations of trajectory segments (sequences of state-action pairs). Our approach manages to produce queries which are closely aligned with those labeled by humans, while avoiding excessive uncertainty according to the human preference predictions as determined by reward estimations. Additionally, by introducing variation without deviating from the original human’s intents, more robust reward function representations are achieved. We compare our approach to recent state-of-the-art preference-based RL semi-supervised learning techniques. Our experimental findings reveal that we can enhance the generalization of the estimated reward function without requiring additional human intervention. Lastly, to confirm the practical applicability of our approach, we conduct experiments involving actual human users in a simulated social navigation setting. Videos of the experiments can be found at https://sites.google.com/view/rl-sequel

A crucial part of traditional reinforcement learning (RL) is the initial exploration phase, in which trying available actions randomly is a critical element. As random behavior might be detrimental to a social interaction, this work proposes a novel paradigm for learning social robot behavior-the use of shielding to ensure socially appropriate behavior during exploration and learning. We explore how a data-driven approach for shielding could be used to generate listening behavior. In a video-based user study (N=110), we compare shielded exploration to two other exploration methods. We show that the shielded exploration is perceived as more comforting and appropriate than a straightforward random approach. Based on our findings, we discuss the potential for future work using shielded and socially guided approaches for learning idiosyncratic social robot behaviors through RL.

A crucial part of traditional reinforcement learning (RL) is the initial exploration phase, in which trying available actions randomly is a critical element. As random behavior might be detrimental to a social interaction, this work proposes a novel paradigm for learning social robot behavior--the use of shielding to ensure socially appropriate behavior during exploration and learning. We explore how a data-driven approach for shielding could be used to generate listening behavior. In a video-based user study (N=110), we compare shielded exploration to two other exploration methods. We show that the shielded exploration is perceived as more comforting and appropriate than a straightforward random approach. Based on our findings, we discuss the potential for future work using shielded and socially guided approaches for learning idiosyncratic social robot behaviors through RL.   

Practical implementations of deep reinforcement learning (deep RL) have been challenging due to an amplitude of factors, such as designing reward functions that cover every possible interaction. To address the heavy burden of robot reward engineering, we aim to leverage subjective human preferences gathered in the context of human-robot interaction, while taking advantage of a baseline reward function when available. By considering baseline objectives to be designed beforehand, we are able to narrow down the policy space, solely requesting human attention when their input matters the most. To allow for control over the optimization of different objectives, our approach contemplates a multi-objective setting. We achieve human-compliant policies by sequentially training an optimal policy from a baseline specification and collecting queries on pairs of trajectories. These policies are obtained by training a reward estimator to generate Pareto optimal policies that include human preferred behaviours. Our approach ensures sample efficiency and we conducted a user study to collect real human preferences, which we utilized to obtain a policy on a social navigation environment.

Human-in-the-loop reinforcement learning (RL) methods actively integrate human knowledge to create reward functions for various robotic tasks. Learning from preferences shows promise as alleviates the requirement of demonstrations by querying humans on state-action sequences. However, the limited granularity of sequence-based approaches complicates temporal credit assignment. The amount of human querying is contingent on query quality, as redundant queries result in excessive human involvement. This paper addresses the often-overlooked aspect of query selection, which is closely related to active learning (AL). We propose a novel query selection approach that leverages variational autoencoder (VAE) representations of state sequences. In this manner, we formulate queries that are diverse in nature while simultaneously taking into account reward model estimations. We compare our approach to the current state-of-the-art query selection methods in preference-based RL, and find ours to be either on-par or more sample efficient through extensive benchmarking on simulated environments relevant to robotics. Lastly, we conduct an online study to verify the effectiveness of our query selection approach with real human feedback and examine several metrics related to human effort.

Human-in-the-loop reinforcement learning (RL) methods actively integrate human knowledge to create reward functions for various robotic tasks. Learning from preferences shows promise as alleviates the requirement of demonstrations by querying humans on state-action sequences. However, the limited granularity of sequence-based approaches complicates temporal credit assignment. The amount of human querying is contingent on query quality, as redundant queries result in excessive human involvement. This paper addresses the often-overlooked aspect of query selection, which is closely related to active learning (AL). We propose a novel query selection approach that leverages variational autoencoder (VAE) representations of state sequences. In this manner, we formulate queries that are diverse in nature while simultaneously taking into account reward model estimations. We compare our approach to the current state-of-the-art query selection methods in preference-based RL, and find ours to be either on-par or more sample efficient through extensive benchmarking on simulated environments relevant to robotics. Lastly, we conduct an online study to verify the effectiveness of our query selection approach with real human feedback and examine several metrics related to human effort.