In spite of the rapid advancements in AI, tasks like laundry, tidying, and general household assistance remain challenging for robots due to their limited capacity to generalize manipulation skills across different variations of everyday objects.Manipulation of textiles, in particular, poses unique challenges due to their deformable nature and complex dynamics.  In this thesis, we aim to enhance the generalization of robotic manipulation skills for textiles by addressing how robots can adapt their strategies based on the physical properties of deformable objects. We begin by identifying key factors of variation in textiles relevant to manipulation, drawing insights from overlooked taxonomies in the textile industry. The core challenge of adaptation is addressed by leveraging the synergies between interactive perception and cloth dynamics models. These are utilized to tackle two fundamental estimation problems to achieve adaptation: property identification, as these properties define the system’s dynamic and how the object responds to external forces, and state estimation, which provides the feedback necessary for closing the action-perception loop.  To identify object properties, we investigate how combining exploratory actions, such as pulling and twisting, with sensory feedback can enhance a robot’s understanding of textile characteristics. Central to this investigation is the development of an adaptation module designed to encode textile properties from recent observations, enabling data-driven dynamics models to adjust their predictions accordingly to the perceived properties. To address state estimation challenges arising from cloth self-occlusions, we explore semantic descriptors and 3D tracking methods that integrate geometric observations, such as point clouds, with visual cues from RGB data.Finally, we integrate these modeling and perceptual components into a model-based manipulation framework and evaluate the generalization of the proposed method across a diverse set of real-world textiles. The results, demonstrating enhanced generalization, underscore the potential of adapting the manipulation in response to variations in textiles' properties and highlight the critical role of the action-perception loop in achieving adaptability.

Trots de snabba framstegen inom AI förblir uppgifter som att tvätta, städa och allmän hushållshjälp utmanande för robotar på grund av deras begränsade förmåga att generalisera manipulationsfärdigheter över olika variationer av vardagsföremål. Manipulation av textilier utgör i synnerhet unika utmaningar på grund av deras deformerbara natur och komplexa dynamik.I denna avhandling syftar vi till att förbättra generaliseringen av robotiska manipulationsfärdigheter för textilier genom att undersöka hur robotar kan anpassa sina strategier baserat på de fysiska egenskaperna hos deformerbara objekt. Vi börjar med att identifiera nyckelfaktorer för variation i textilier som är relevanta för manipulation och drar insikter från förbisedda taxonomier inom textilindustrin.Den centrala utmaningen med anpassning adresseras genom att utnyttja synergierna mellan interaktiv perception och modeller för textildynamik. Dessa används för att lösa två grundläggande estimeringsproblem för att uppnå anpassning: egenskapsidentifiering, eftersom dessa egenskaper definierar systemets dynamik och hur objektet reagerar på yttre krafter, samt tillståndsestimering, som ger den återkoppling som krävs för att stänga åtgärds-perceptionsslingan. För att identifiera objektets egenskaper undersöker vi hur kombinationen av utforskande handlingar, såsom att dra och vrida, med sensorisk återkoppling kan förbättra robotens förståelse för textilens egenskaper. Centralt i denna undersökning är utvecklingen av en anpassningsmodul utformad för att koda textilens egenskaper från nyligen gjorda observationer, vilket gör det möjligt för datadrivna dynamikmodeller att justera sina förutsägelser utifrån de uppfattade egenskaperna.För att hantera utmaningar med tillståndsestimering som uppstår vid textilens självocklusioner utforskar vi semantiska deskriptorer och 3D-spårningsmetoder som integrerar geometriska observationer, såsom punktmoln, med visuella ledtrådar från RGB-data.Slutligen integrerar vi dessa modellerings- och perceptionskomponenter i ett modellbaserat manipulationsramverk och utvärderar generaliseringen av den föreslagna metoden på ett brett urval av textilier i verkliga miljöer. Resultaten, som visar förbättrad generalisering, understryker potentialen i att anpassa manipulation till variationer i textilernas egenskaper och framhäver den avgörande rollen för åtgärds-perceptionsslingan i att uppnå anpassningsförmåga.

Recent progress in robotic manipulation has been fueled by large-scale datasets collected across diverse environments. Training robotic manipulation policies on these datasets is traditionally performed in a centralized manner, raising concerns regarding scalability, adaptability, and data privacy. While federated learning enables decentralized, privacy-preserving training, its application to robotic manipulation remains largely unexplored. We introduce FLAME (Federated Learning Across Manipulation Environments), the first benchmark designed for federated learning in robotic manipulation. FLAME consists of: (i) a set of large-scale datasets of over 160,000 expert demonstrations of multiple manipulation tasks, collected across a wide range of simulated environments; (ii) a training and evaluation framework for robotic policy learning in a federated setting. We evaluate standard federated learning algorithms in FLAME, showing their potential for distributed policy learning and highlighting key challenges. Our benchmark establishes a foundation for scalable, adaptive, and privacy-aware robotic learning. The code is publicly available at https://github.com/KTH-RPL/ELSA-Robotics-Challenge.

The realm of textiles spans clothing, households, healthcare, sports, and industrial applications. The deformable nature of these objects poses unique challenges that prior work on rigid objects cannot fully address. The increasing interest within the community in textile perception and manipulation has led to new methods that aim to address challenges in modeling, perception, and control, resulting in significant progress. However, this progress is often tailored to one specific textile or a subcategory of these textiles. To understand what restricts these methods and hinders current approaches from generalizing to a broader range of real-world textiles, this review provides an overview of the field, focusing specifically on how and to what extent textile variations are addressed in modeling, perception, benchmarking, and manipulation of textiles. We conclude by identifying key open problems and outlining grand challenges that will drive future advancements in the field.

We present AdaFold, a model-based feedback-loop framework for optimizing folding trajectories. AdaFold extracts a particle-based representation of cloth from RGB-D images and feeds back the representation to a model predictive control to re-plan folding trajectory at every time-step. A key component of AdaFold that enables feedback-loop manipulation is the use of semantic descriptors extracted from geometric features. These descriptors enhance the particle representation of the cloth to distinguish between ambiguous point clouds of differently folded cloths. Our experiments demonstrate AdaFold's ability to adapt folding trajectories of cloths with varying physical properties and generalize from simulated training to real-world execution.

We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth's state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time by ∼85 %.

The field of robotics faces inherent challenges in manipulating deformable objects, particularly in understanding and standardising fabric properties like elasticity, stiffness, and friction. While the significance of these properties is evident in the realm of cloth manipulation, accurately categorising and comprehending them in real-world applications remains elusive. This study sets out to address two primary objectives: (1) to provide a framework suitable for robotics applications to characterise cloth objects, and (2) to study how these properties influence robotic manipulation tasks. Our preliminary results validate the framework's ability to characterise cloth properties and compare cloth sets, and reveal the influence that different properties have on the outcome of five manipulation primitives. We believe that, in general, results on the manipulation of clothes should be reported along with a better description of the garments used in the evaluation. This paper proposes a set of these measures.

We study the problem of learning graph dynamics of deformable objects that generalizes to unknown physical properties. Our key insight is to leverage a latent representation of elastic physical properties of cloth-like deformable objects that can be extracted, for example, from a pulling interaction. In this paper we propose EDO-Net (Elastic Deformable Object - Net), a model of graph dynamics trained on a large variety of samples with different elastic properties that does not rely on ground-truth labels of the properties. EDO-Net jointly learns an adaptation module, and a forward-dynamics module. The former is responsible for extracting a latent representation of the physical properties of the object, while the latter leverages the latent representation to predict future states of cloth-like objects represented as graphs. We evaluate EDO-Net both in simulation and real world, assessing its capabilities of: 1) generalizing to unknown physical properties, 2) transferring the learned representation to new downstream tasks.

Physical interaction with textiles, such as assistivedressing or household tasks, requires advanced dexterous skills.The complexity of textile behavior during stretching and pullingis influenced by the material properties of the yarn and bythe textile’s construction technique, which are often unknownin real-world settings. Moreover, identification of physicalproperties of textiles through sensing commonly available onrobotic platforms remains an open problem. To address this,we introduce Elastic Context (EC), a method to encode theelasticity of textiles using stress-strain curves adapted fromtextile engineering for robotic applications. We employ EC tolearn generalized elastic behaviors of textiles and examine theeffect of EC dimension on accurate force modeling of real-worldnon-linear elastic behaviors.

— Identification of textile properties is an important milestone toward advanced robotic manipulation tasks that consider interaction with clothing items such as assisted dressing, laundry folding, automated sewing, textile recycling and reusing. Despite the abundance of work considering this class of deformable objects, many open problems remain. These relate to the choice and modelling of the sensory feedback as well as the control and planning of the interaction and manipulation strategies. Most importantly, there is no structured approach for studying and assessing different approaches that may bridge the gap between the robotics community and textile production industry. To this end, we outline a textile taxonomy considering fiber types and production methods, commonly used in textile industry. We devise datasets according to the taxonomy, and study how robotic actions, such as pulling and twisting of the textile samples, can be used for the classification. We also provide important insights from the perspective of visualization and interpretability of the gathered data.