Preference-based reinforcement learning (PbRL) is a suitable approach for style adaptation of pre-trained robotic behavior: adapting the robot's policy to follow human user preferences while still being able to perform the original task. However, collecting preferences for the adaptation process in robotics is often challenging and time-consuming. In this work we explore the adaptation of pre-trained robots in the low-preference-data regime. We show that, in this regime, recent adaptation approaches suffer from catastrophic reward forgetting (CRF), where the updated reward model overfits to the new preferences, leading the agent to become unable to perform the original task. To mitigate CRF, we propose to enhance the original reward model with a small number of parameters (low-rank matrices) responsible for modeling the preference adaptation. Our evaluation shows that our method can efficiently and effectively adjust robotic behavior to human preferences across simulation benchmark tasks and multiple real-world robotic tasks. We provide videos of our results and source code at https://sites.google.com/view/preflora/

With the rapid advancement of AI, the technology has moved out of the industrial and lab setting and into the hands of everyday people. Once AI and robot agents are placed in everyday households they need to be able to take into account human needs. With methods like Reinforcement Learning from Human Feedback (RLHF), the agent can learn desirable behavior by learning a reward function or optimizing a policy directly based on their feedback. Unlike vision models and large language models (LLM) which benefit from internet-scale data, RLHF is limited by the amount of feedback provided since it requires additional human effort. In this thesis, we look into how we can decrease the amount of feedback humans provide to reduce their burden when estimating a reward function without degrading the estimate. We investigate the fundamental trade-off between the informativeness and efficiency of feedback from a preference-based learning perspective. In this regard, we introduce multiple methods that can be categorized into two groups, implicit methods that increase the quality of the feedback without additional human effort, and explicit methods that aim to drastically increase the information content by using additional feedback types. To implicitly improve the efficiency of preference feedback, we look into how we can utilize Active Learning (AL) to improve the diversity of samples by strategically picking from different clusters in a learned representation through a Variational Autoencoder (VAE). Furthermore, we make use of the unique relationship between preference pairs to perform data synthesis by interpolation on the latent space of the VAE. While the implicit methods have the benefit of requiring no extra effort, they still suffer from the limited amount of information that preferences alone can provide. One limitation of preferences on trajectories is that there is no discounting which means that if a trajectory is preferred, the assumption is that the whole trajectory is preferred leading to casual confusion. Therefore, we introduce a new form of feedback called highlights that lets the user show on the trajectory, which part was good and which part was bad. Furthermore, leveraging LLMs we create a method for letting humans explain their preferences via natural language to deduce which parts were preferred. Overall, this thesis takes a step away from the assumption of internet-scale data and shows how we can achieve alignment from less human feedback.

Med den snabba utvecklingen av AI har teknologin lämnat den industriella och laboratoriebaserade miljön och hamnat i händerna på vanliga människor. När AI- och robotagenter placeras i vardagliga hushåll måste de kunna ta hänsyn till mänskliga behov. Med metoder som Reinforcement Learning from Human Feedback (RLHF) kan en agent lära sig önskvärt beteende genom att antingen lära sig en belöningsfunktion eller optimera en policy direkt baserat på mänsklig feedback. Till skillnad från visionsmodeller och stora språkmodeller (LLM), som gynnas av internet-skaliga datamängder, är RLHF begränsat av mängden feedback som ges, eftersom det kräver ytterligare mänsklig insats.I denna avhandling undersöker vi hur man kan minska mängden feedback som människor behöver ge för att minska deras börda vid estimering av en belöningsfunktion, utan att försämra uppskattningen. Vi undersöker den fundamentala avvägningen mellan informationsinnehållet och effektiviteten i feedback från ett preferensbaserat inlärningsperspektiv. I detta avseende introducerar vi flera metoder som kan kategoriseras i två grupper: implicita metoder, som förbättrar kvaliteten på feedback utan extra mänsklig ansträngning, och explicita metoder, som syftar till att drastiskt öka informationsinnehållet genom att använda ytterligare typer av feedback.För att implicit förbättra effektiviteten av preferensfeedback undersöker vi hur Active Learning (AL) kan användas för att förbättra mångfalden av urval genom att strategiskt välja från olika kluster i en inlärd representation med hjälp av en Variational Autoencoder (VAE). Vidare utnyttjar vi den unika relationen mellan preferenspar för att utföra datasyntes genom interpolation i VAE:s latenta utrymme.Även om de implicita metoderna har fördelen att de inte kräver extra ansträngning, lider de fortfarande av den begränsade mängd information som preferenser ensamma kan ge. En begränsning med preferenser på trajektorier är att det saknas diskontering, vilket innebär att om en trajektori föredras, antas det att hela trajektorin föredras, vilket kan leda till kausal förvirring. Därför introducerar vi en ny form av feedback, kallad highlights, där användaren kan markera på trajektorier vilka delar som var bra och vilka som var dåliga. Vidare utnyttjar vi LLM:er för att skapa en metod där människor kan förklara sina preferenser genom naturligt språk för att dra slutsatser om vilka delar som föredrogs.Sammanfattningsvis tar denna avhandling ett steg bort från antagandet om internet-skaliga datamängder och visar hur vi kan uppnå anpassning med mindre mänsklig feedback.

Many solutions to address the challenge of robot learning have been devised, namely through exploring novel ways for humans to communicate complex goals and tasks in reinforcement learning (RL) setups. One way that experienced recent research interest directly addresses the problem by considering human feedback as preferences between pairs of trajectories (sequences of state-action pairs). However, when simply attributing a single preference to a pair of trajectories that contain many agglomerated steps, key pieces of information are lost in the process. We amplify the initial definition of preferences to account for highlights: state-action pairs of relatively high information (high/low reward) within a preferred trajectory. To include the additional information, we design novel regularization methods within a preference learning framework. To this extent, we present our method which is able to greatly reduce the necessary amount of preferences, by permitting the highlighting of favoured trajectories, in order to reduce the entropy of the credit assignment. We show the effectiveness of our work in both simulation and a user study, which analyzes the feedback given and its implications. We also use the total collected feedback to train a robot policy for socially compliant trajectories in a simulated social navigation environment. We release code and video examples at https://sites.google.com/view/rl-polite

Preference-based reinforcement learning (RL) has emerged as a new field in robot learning, where humans play a pivotal role in shaping robot behavior by expressing preferences on different sequences of state-action pairs. However, formulating realistic policies for robots demands responses from humans to an extensive array of queries. In this work, we approach the sample-efficiency challenge by expanding the information collected per query to contain both preferences and optional text prompting. To accomplish this, we leverage the zero-shot capabilities of a large language model (LLM) to reason from the text provided by humans. To accommodate the additional query information, we reformulate the reward learning objectives to contain flexible highlights - state-action pairs that contain relatively high information and are related to the features processed in a zero-shot fashion from a pretrained LLM. In both a simulated scenario and a user study, we reveal the effectiveness of our work by analyzing the feedback and its implications. Additionally, the collective feedback collected serves to train a robot on socially compliant trajectories in a simulated social navigation landscape. We provide video examples of the trained policies at https://sites.google.com/view/rl-predilect.

Preference-based reinforcement learning (RL) poses as a recent research direction in robot learning, by allowing humans to teach robots through preferences on pairs of desired behaviours. Nonetheless, to obtain realistic robot policies, an arbitrarily large number of queries is required to be answered by humans. In this work, we approach the sample-efficiency challenge by presenting a technique which synthesizes queries, in a semi-supervised learning perspective. To achieve this, we leverage latent variational autoencoder (VAE) representations of trajectory segments (sequences of state-action pairs). Our approach manages to produce queries which are closely aligned with those labeled by humans, while avoiding excessive uncertainty according to the human preference predictions as determined by reward estimations. Additionally, by introducing variation without deviating from the original human’s intents, more robust reward function representations are achieved. We compare our approach to recent state-of-the-art preference-based RL semi-supervised learning techniques. Our experimental findings reveal that we can enhance the generalization of the estimated reward function without requiring additional human intervention. Lastly, to confirm the practical applicability of our approach, we conduct experiments involving actual human users in a simulated social navigation setting. Videos of the experiments can be found at https://sites.google.com/view/rl-sequel

Multisensory integration influences emotional perception, as the McGurk effect demonstrates for the communication between humans. Human physiology implicitly links the production of visual features with other modes like the audio channel: Face muscles responsible for a smiling face also stretch the vocal cords that result in a characteristic smiling voice. For artificial agents capable of multimodal expression, this linkage is modeled explicitly. In our studies, we observe the influence of visual and audio channels on the perception of the agents' emotional expression. We created videos of virtual characters and social robots either with matching or mismatching emotional expressions in the audio and visual channels. In two online studies, we measured the agents' perceived valence and arousal. Our results consistently lend support to the ‘emotional McGurk effect' hypothesis, according to which face transmits valence information, and voice transmits arousal. When dealing with dynamic virtual characters, visual information is enough to convey both valence and arousal, and thus audio expressivity need not be congruent. When dealing with robots with fixed facial expressions, however, both visual and audio information need to be present to convey the intended expression.

Practical implementations of deep reinforcement learning (deep RL) have been challenging due to an amplitude of factors, such as designing reward functions that cover every possible interaction. To address the heavy burden of robot reward engineering, we aim to leverage subjective human preferences gathered in the context of human-robot interaction, while taking advantage of a baseline reward function when available. By considering baseline objectives to be designed beforehand, we are able to narrow down the policy space, solely requesting human attention when their input matters the most. To allow for control over the optimization of different objectives, our approach contemplates a multi-objective setting. We achieve human-compliant policies by sequentially training an optimal policy from a baseline specification and collecting queries on pairs of trajectories. These policies are obtained by training a reward estimator to generate Pareto optimal policies that include human preferred behaviours. Our approach ensures sample efficiency and we conducted a user study to collect real human preferences, which we utilized to obtain a policy on a social navigation environment.

Human-in-the-loop reinforcement learning (RL) methods actively integrate human knowledge to create reward functions for various robotic tasks. Learning from preferences shows promise as alleviates the requirement of demonstrations by querying humans on state-action sequences. However, the limited granularity of sequence-based approaches complicates temporal credit assignment. The amount of human querying is contingent on query quality, as redundant queries result in excessive human involvement. This paper addresses the often-overlooked aspect of query selection, which is closely related to active learning (AL). We propose a novel query selection approach that leverages variational autoencoder (VAE) representations of state sequences. In this manner, we formulate queries that are diverse in nature while simultaneously taking into account reward model estimations. We compare our approach to the current state-of-the-art query selection methods in preference-based RL, and find ours to be either on-par or more sample efficient through extensive benchmarking on simulated environments relevant to robotics. Lastly, we conduct an online study to verify the effectiveness of our query selection approach with real human feedback and examine several metrics related to human effort.

Human-in-the-loop reinforcement learning (RL) methods actively integrate human knowledge to create reward functions for various robotic tasks. Learning from preferences shows promise as alleviates the requirement of demonstrations by querying humans on state-action sequences. However, the limited granularity of sequence-based approaches complicates temporal credit assignment. The amount of human querying is contingent on query quality, as redundant queries result in excessive human involvement. This paper addresses the often-overlooked aspect of query selection, which is closely related to active learning (AL). We propose a novel query selection approach that leverages variational autoencoder (VAE) representations of state sequences. In this manner, we formulate queries that are diverse in nature while simultaneously taking into account reward model estimations. We compare our approach to the current state-of-the-art query selection methods in preference-based RL, and find ours to be either on-par or more sample efficient through extensive benchmarking on simulated environments relevant to robotics. Lastly, we conduct an online study to verify the effectiveness of our query selection approach with real human feedback and examine several metrics related to human effort.

Multisensory integration influences emotional perception, as the McGurk effect demonstrates for the communication between humans. Human physiology implicitly links the production of visual features with other modes like the audio channel: Face muscles responsible for a smiling face also stretch the vocal cords that results in a characteristic smiling voice. For artificial agents capable of multimodal expression, this linkage is modeled explicitly. In our study, we observe the influence of visual and audio channel on the perception of the agent’s emotional state. We created two virtual characters to control for anthropomorphic appearance. We record videos of these agents either with matching or mismatching emotional expression in the audio and visual channel. In an online study we measured the agent’s perceived valence and arousal. Our results show that a matched smiling voice and smiling face increase both dimensions of the Circumplex model of emotions: ratings of valence and arousal grow. When the channels present conflicting information, any type of smiling results in higher arousal rating, but only the visual channel increases the perceived valence. When engineers are constrained in their design choices, we suggest they should give precedence to convey the artificial agent’s emotional state through the visual channel.