Scene coordinate regression (SCR) has established itself as a promising learning-based approach to visual relocalization. After mere minutes of scene-specific training, SCR models estimate camera poses of query images with high accuracy. Still, SCR methods fall short of the generalization capabilities of more classical feature-matching approaches. When imaging conditions of query images, such as lighting or viewpoint, are too different from the training views, SCR models fail. Failing to generalize is an inherent limitation of previous SCR frameworks, since their training objective is to encode the training views in the weights of the coordinate regressor itself. The regressor essentially overfits to the training views, by design. We propose to separate the coordinate regressor and the map representation into a generic transformer and a scene-specific map code. This separation allows us to pre-train the transformer on tens of thousands of scenes. More importantly, it allows us to train the transformer to generalize from mapping images to unseen query images during pre-training. We demonstrate on multiple challenging relocalization datasets that our method, ACE-G, leads to significantly increased robustness while keeping the computational footprint attractive

Spatial understanding comprises various abilities from pose estimation of objects and cameras within a scene to shape completion given partial observations. These abilities are what enable humans to intuitively navigate and interact with the world. Despite significant progress in large-scale learning, computers still lack the same intuitive spatial understanding of humans. In robotics, this lack of abilities implies limited applicability of classical robotics pipelines in real-world environments and in augmented reality it limits achievable fidelity as well as the interaction of virtual content with real-world objects.

This thesis investigates ways to improve spatial understanding of computers using different learning- and optimization-based techniques. Learning-based methods are employed to learn useful priors about the objects and the 3D world, whereas optimization-based techniques are used to find models of objects and scenes aligning well with a set of observations. Within this framework, we investigate and propose methods for three subproblems of spatial understanding.

First, we propose a modular framework for categorical object pose and shape estimation, which combines a pre-trained generative shape model with a discriminative initialization network which regresses an initial pose and latent shape description from a partial point cloud of an object. By combining the generative shape model with a differentiable renderer we further perform iterative, joint pose and shape optimization from one or multiple views. Our approach outperforms existing methods especially on unconstrained orientations, while achieving competitive results for upright, tabletop objects.

Second, we investigate the use of neural fields for dense, volumetric mapping. Specifically, we propose to represent the scene by a set of spatially constrained, movable neural fields anchored to a pose graph. We formulate the optimization problem of the multi-field scene representation as independent optimization of each field demonstrating that this approach allows real-time loop closure integration, avoids transition artifact at field boundaries, and outperforms current neural-field-based SLAM systems on larger scenes in which significant drift can accumulate.

Third, we investigate large-scale pre-training for visual relocalization using scene coordinate regression. We split up the scene-specific regressor into a scene-agnostic regressor and a scene-specific latent map code, and propose a pre-training scheme for the scene-agnostic coordinate regressor to better generalize from mapping images to query images containing different viewpoints, lighting changes, and objects. We demonstrate that our approach outperforms existing methods under such dynamic mapping-query splits.

Rumslig förståelse omfattar olika förmågor, från pose estimering av objekt och kameror i en scen till formkomplettering av objekt utifrån partiella observationer. Dessa förmågor är vad som gör det möjligt för människor att intuitivt navigera och interagera med världen. Trots betydande framsteg inom storskalig inlärning saknar datorer fortfarande samma intuitiva rumsliga förståelse som människor har. Inom robotik innebär denna brist på förmågor en begränsad tillämpning av klassiska robotpipelines i verkliga miljöer, och inom förstärkt verklighet (augmented reality) begränsar den både den uppnåeliga verklighetsgraden och interaktionen mellan virtuellt innehåll och verkliga objekt.

Denna avhandling undersöker sätt att förbättra datorers rumsliga förståelse med hjälp av olika inlärnings- och optimeringsbaserade tekniker. Inlärningsbaserade metoder används för att lära in användbara förkunskaper om objekt och 3D-världen, medan optimeringsbaserade tekniker används för att hitta modeller av objekt och scener som stämmer väl överens med en uppsättning observationer. Inom detta ramverk undersöker och föreslår vi metoder för tre delproblem inom rumslig förståelse.

För det första föreslår vi ett modulärt ramverk för kategorisk pose- och formbestämning av objekt, vilket kombinerar en förtränad generativ formmodell med ett diskriminativt initialiseringsnätverk som skattar en initial pose och en latent form utifrån ett partiellt punktmoln av ett objekt. Genom att kombinera den generativa formmodellen med en differentierbar renderare utför vi vidare iterativ, gemensam optimering av pose och form från en eller flera vyer. Vår metod överträffar befintliga metoder, särskilt för objekt i fria orienteringar, samtidigt som den uppnår konkurrenskraftiga resultat för upprättstående objekt på en bordsyta.

För det andra undersöker vi användningen av neurala fält (neural fields) för tät, volymetrisk kartläggning. Specifikt föreslår vi att representera scenen med en uppsättning rumsligt begränsade, flyttbara neurala fält förankrade i en posegraf. Vi formulerar optimeringsproblemet för scenrepresentationen med flera fält som oberoende optimering av varje fält och visar att denna metod möjliggör integration av loop-stängning (loop closure) i realtid, undviker övergångsartefakter vid fältgränser och överträffar nuvarande neuralfältbaserade SLAM-system i större scener där betydande drift kan ackumuleras.

För det tredje undersöker vi storskalig förträning för visuell relokalisering med hjälp av regression av scenkoordinater. Vi delar upp den scenspecifika regressorn i en scenagnostisk regressor och en scenspecifik latent kartkod. Vi föreslår ett förträningsschema för den scenagnostiska koordinatregressorn för att bättre generalisera från kartläggningsbilder till sökbilder som innehåller olika synvinklar, ljusförändringar och objektplaceringar. Vi visar att vår metod överträffar befintliga metoder under sådana dynamiska uppdelningar mellan kartläggnings- och sökdata.

Neural field-based SLAM methods typically employ a single monolithic field as their scene representation. This prevents efficient incorporation of loop closure constraints and limits scalability. To address these shortcomings we propose a novel RGB-D neural mapping framework in which the scene is represented by a collection of lightweight neural fields which are dynamically anchored to the pose graph of a sparse visual SLAM system. Our approach shows the ability to integrate large-scale loop closures while requiring only minimal reintegration. Furthermore we verify the scalability of our approach by demonstrating successful building-scale mapping taking multiple loop closures into account during the optimization and show that our method outperforms existing state-of-the-art approaches on large scenes in terms of quality and runtime. Our code is available open-source at https://github.com/KTH-RPL/neural_graph_mapping. 

Autonomous agents rely on sensor data to construct representations of their environments, essential for predicting future events and planning their actions. However, sensor measurements suffer from limited range, occlusions, and sensor noise. These challenges become more evident in highly dynamic environments. This work proposes a probabilistic framework to jointly infer which parts of an environment are statically and which parts are dynamically occupied. We formulate the problem as a Bayesian network and introduce minimal assumptions that significantly reduce the complexity of the problem. Based on those, we derive Transitional Grid Maps (TGMs), an efficient analytical solution. Using real data, we demonstrate how this approach produces better maps than the state-of-the-art by keeping track of both static and dynamic elements and, as a side effect, can help improve existing SLAM algorithms.

Robots rely on visual relocalization to estimate their pose from camera images when they lose track. One of the challenges in visual relocalization is repetitive structures in the operation environment of the robot. This calls for probabilistic methods that support multiple hypotheses for robot's pose. We propose such a probabilistic method to predict the posterior distribution of camera poses given an observed image. Our proposed training strategy results in a generative model of camera poses given an image, which can be used to draw samples from the pose posterior distribution. Our method is streamlined and well-founded in theory and outperforms existing methods on localization in presence of ambiguities.

Visual localization allows autonomous robots to relocalize when losing track of their pose by matching their current observation with past ones. However, ambiguous scenes pose a challenge for such systems, as repetitive structures can be viewed from many distinct, equally likely camera poses, which means it is not sufficient to produce a single best pose hypothesis. In this work, we propose a probabilistic framework that for a given image predicts the arbitrarily shaped posterior distribution of its camera pose. We do this via a novel formulation of camera pose regression using variational inference, which allows sampling from the predicted distribution. Our method outperforms existing methods on localization in ambiguous scenes. We open-source our approach and share our recorded data sequence at github.com/efreidun/vapor.

Recently, various methods for 6D pose and shape estimation of objects have been proposed. Typically, these methods evaluate their pose estimation in terms of average precision and reconstruction quality in terms of chamfer distance. In this work, we take a critical look at this predominant evaluation protocol, including metrics and datasets. We propose a new set of metrics, contribute new annotations for the Redwood dataset, and evaluate state-of-the-art methods in a fair comparison. We find that existing methods do not generalize well to unconstrained orientations and are actually heavily biased towards objects being upright. We provide an easy-to-use evaluation toolbox with well-defined metrics, method, and dataset interfaces, which allows evaluation and comparison with various state-of-the-art approaches (https://github.com/roym899/pose and shape evaluation).

Recently, various methods for 6D pose and shape estimation of objects at a per-category level have been proposed. This work provides an overview of the field in terms of methods, datasets, and evaluation protocols. First, an overview of existing works and their commonalities and differences is provided. Second, we take a critical look at the predominant evaluation protocol, including metrics and datasets. Based on the findings, we propose a new set of metrics, contribute new annotations for the Redwood dataset, and evaluate state-of-the-art methods in a fair comparison. The results indicate that existing methods do not generalize well to unconstrained orientations and are actually heavily biased towards objects being upright. We provide an easy-to-use evaluation toolbox with well-defined metrics, methods, and dataset interfaces, which allows evaluation and comparison with various state-of-the-art approaches (https://github.com/roym899/pose_and_shape_evaluation).

Rich geometric understanding of the world is an important component of many robotic applications such as planning and manipulation. In this paper, we present a modular pipeline for pose and shape estimation of objects from RGB-D images given their category. The core of our method is a generative shape model, which we integrate with a novel initialization network and a differentiable renderer to enable 6D pose and shape estimation from a single or multiple views. We investigate the use of discretized signed distance fields as an efficient shape representation for fast analysis-by-synthesis optimization. Our modular framework enables multi-view optimization and extensibility. We demonstrate the benefits of our approach over state-of-the-art methods in several experiments on both synthetic and real data. We open-source our approach at https://github.com/roym899/sdfest.

Planning smooth and energy-efficient paths for wheeled mobile robots is a central task for applications ranging from autonomous driving to service and intralogistic robotics. Over the past decades, several sampling-based motion-planning algorithms, extend functions and post-smoothing algorithms have been introduced for such motion-planning systems. Choosing the best combination of components for an application is a tedious exercise, even for expert users. We therefore present Bench-MR, the first open-source motion-planning benchmarking framework designed for sampling-based motion planning for nonholonomic, wheeled mobile robots. Unlike related software suites, Bench-MR is an easy-to-use and comprehensive benchmarking framework that provides a large variety of sampling-based motion-planning algorithms, extend functions, collision checkers, post-smoothing algorithms and optimization criteria. It aids practitioners and researchers in designing, testing, and evaluating motion-planning systems, and comparing them against the state of the art on complex navigation scenarios through many performance metrics. Through several experiments, we demonstrate how Bench-MR can be used to gain extensive insights from the benchmarking results it generates.