Keywords: Bimanual Manipulation, Deep Learning in Grasping and Manipulation, Dual Arm Manipulation
Abstract: Folding garments reliably and efficiently is a long standing challenge in robotic manipulation due to the complex dynamics and high dimensional configuration space of garments. An intuitive approach is to initially manipulate the garment to a canonical smooth configuration before folding. In this work, we develop SpeedFolding, a reliable and efficient bimanual system, which given user-defined instructions as folding lines, manipulates an initially crumpled garment to (1) a smoothed and (2) a folded configuration. Our primary contribution is a novel neural network architecture that is able to predict pairs of gripper poses to parameterize a diverse set of bimanual action primitives. After learning from 4300 human-annotated and self-supervised actions, the robot is able to fold garments from a random initial configuration in under 120s on average with a success rate of 93%. Real-world experiments show that the system is able to generalize to unseen garments of different color, shape, and stiffness. While prior work achieved 3-6 Folds Per Hour (FPH), SpeedFolding achieves 30-40 FPH.

Keywords: Reactive and Sensor-Based Planning, Motion and Path Planning
Abstract: We present our work on a visibility graph-based planning framework. The planner constructs a polygonal representation of the environment by extracting edge points around obstacles to form enclosed polygons. With that, the method dynamically updates a global visibility graph using a two-layered data structure, expanding the visibility edges along with the navigation and removing edges that become occluded by dynamic obstacles. The planner is capable of dealing with navigation tasks in both known and unknown environments. In the latter case, the method is attemptable in discovering a way to the goal by picking up the environment layout on the fly and fast re-planning to account for the newly observed environment. We evaluate the method in both simulated and real-world settings. The method shows the capability to navigate through unknown environments and reduces the travel time by up to 12-47% from search-based methods: A*, D* Lite, and more than 24-35% than sampling-based methods: RRT*, BIT*, and SPARS.

Keywords: Field Robots, Localization, Failure Detection and Recovery
Abstract: LiDAR-based localization and mapping is one of the core components in many modern robotic systems due to the direct integration of range and geometry, allowing for precise motion estimation and generation of high quality maps in real-time. Yet, as a consequence of insufficient environmental constraints present in the scene, this dependence on geometry can result in localization failure, happening in self-symmetric surroundings such as tunnels. This work addresses precisely this issue by proposing a neural network-based estimation approach for detecting (non-)localizability during robot operation. Special attention is given to the localizability of scan-to-scan registration, as it is a crucial component in many LiDAR odometry estimation pipelines. In contrast to previous, mostly traditional detection approaches, the proposed method enables early detection of failure by estimating the localizability on raw sensor measurements without evaluating the underlying registration optimization. Moreover, previous approaches remain limited in their ability to generalize across environments and sensor types, as heuristic-tuning of degeneracy detection thresholds is required. The proposed approach avoids this problem by learning from a collection of different environments, allowing the network to function over various scenarios. Furthermore, the network is trained exclusively on simulated data, avoiding arduous data collection in challenging and degenerate, often hard-to-access, environments. The presented method is tested during field experiments conducted across challenging environments and on two different sensor types without any modifications. The observed detection performance is on par with state-of-the-art methods after environment-specific threshold tuning.

Keywords: Legged Robots, Reinforcement Learning, Art and Entertainment Robotics
Abstract: Training a high-dimensional simulated agent with an under-specified reward function often leads the agent to learn physically infeasible strategies that are ineffective when deployed in the real world. To mitigate these unnatural behaviors, reinforcement learning practitioners often utilize complex reward functions that encourage physically plausible behaviors. However, a tedious labor-intensive tuning process is often required to create hand-designed rewards which might not easily generalize across platforms and tasks. We propose substituting complex reward functions with ``style rewards" learned from a dataset of motion capture demonstrations. A learned style reward can be combined with an arbitrary task reward to train policies that perform tasks using naturalistic strategies. These natural strategies can also facilitate transfer to the real world. We build upon Adversarial Motion Priors -- an approach from the computer graphics domain that encodes a style reward from a dataset of reference motions -- to demonstrate that an adversarial approach to training policies can produce behaviors that transfer to a real quadrupedal robot without requiring complex reward functions. We also demonstrate that an effective style reward can be learned from a few seconds of motion capture data gathered from a German Shepherd and leads to energy-efficient locomotion strategies with natural gait transitions.

Keywords: Simulation and Animation, Human-Centered Robotics, Robot Companions
Abstract: In this paper, we present RCareWorld, a human-centric simulation world for physical and social robot caregiving with support for realistic human modeling, home environments with multiple levels of accessibility, and robots used for assistance. This simulation is designed using inputs from stakeholders such as expert occupational therapists, care-recipients, and caregivers. It provides a variety of benchmark ADL tasks in realistic settings. It interfaces with various physics engines to model rigid, articulated and deformable objects. It provides the capability to plan, control, and learn both human and robot control policies by interfacing it with state-of-the-art external planning and learning libraries. We performed experiments on a subset of these ADL tasks using reinforcement learning methods. We performed a representative real-world physical robotic caregiving experiment by transferring policies learned in RCareWorld directly to a real robot. Additionally, we performed a real-world social caregiving experiment using behaviors modeled in RCareWorld. Robotic caregiving, though potentially impactful towards enhancing the quality-of-life of care-recipients and caregivers, is a field with many barriers of entry due to it's interdisciplinary facets. RCareWorld takes the first step towards building a realistic simulation environment for robotic caregiving research to democratize this field that would enable robotics researchers around the world to contribute to this exciting field. Demo videos and supplementary materials can be found here: https://emprise.cs.cornell.edu/rcareworld/

Keywords: Medical Robots and Systems
Abstract: In endoluminal surgery, the miniature instruments shall be of high accuracy and flexibility for the minimal invasive diagnosis and surgical intervention. To this end, continuum robots with flexible joints have been proposed as the mechanism of endoscopic instruments. The compliance and deformability of the continuum joints enable access into the curved lumen. However, the manufacturing tolerances are not normally considered in the design procedure, and led to inaccuracy in the robotic control.

To improve the control accuracy and flexibility of endoluminal surgical robot, we propose a novel design of a metal printed continuum joint in this paper, which incorporates a variable pitch design into the spring-like structure. The design can reduce the position errors accumulated on the distal tip of the joint, especially at large bending angles. The specification of variable pitch is investigated and determined with a friction model. In addition, to eliminate the distortion of the joint induced during the metal printing process, an extensive experiment was conducted to access the effect of the variables in the design (pitch, thickness, width and number of coils), with the aim of determining optimal parameters for reducing discrepancy caused by manufacturing variations. The final results indicated that the bending error of a single joint can be reduced from 18.10% to 4.63%, and a multi-segment prototype was developed to verify its effectiveness for potential surgical applications.

Keywords: Data Sets for Robot Learning, Machine Learning for Robot Control, Imitation Learning
Abstract: General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions. In this paper, we present CALVIN (Composing Actions from Language and Vision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. We evaluate the agents in zero-shot to novel language instructions and to novel environments and objects. We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.

Keywords: Bioinspired Robot Learning, Visual Learning, Biologically-Inspired Robots
Abstract: Humans have an incredible sense of self-preservation that is both instilled, and also learned through experience. One system which contributes to this is the pain and reflex system which both minimizes damage through involuntary reflex actions and also serves as a means of `negative reinforcement' to allow learning of poor actions or decision. Equipping robots with a reflex system and parallel learning architecture could help to prolong their useful life and allow for continued learning of safe actions. Focusing on a specific mock-up scenario of cubes on a `stove' like setup, we investigate the hardware and learning approaches for a robotic manipulator to learn the presence of `hot' objects and its contextual relationship to the environment. By creating a reflex arc using analog electronics that bypasses the `brain' of the system we show an increase in the speed of release by at least two-fold. In parallel we have a learning procedure which combines visual information of the scene with this `pain signal' to learn and predict when an object may be hot, utilizing an object detection neural network. Finally, we are able to extract the learned contextual information of the environment by introducing a method inspired by `thought experiments' to generate heatmaps that indicate the probability of the environment being hot

Keywords: Continual Learning, Data Sets for Robotic Vision, Incremental Learning
Abstract: Convolutional neural networks show remarkable results in classification but struggle with learning new things on the fly. We present a novel rehearsal-free approach, where a deep neural network is continually learning new unseen object categories without saving any data of prior sequences. Our approach is called RECALL, as the network recalls categories by calculating logits for old categories before training new ones. These are then used during training to avoid changing the old categories. For each new sequence, a new head is added to accommodate the new categories. To mitigate forgetting, we present a regularization strategy where we replace the classification with a regression. Moreover, for the known categories, we propose a Mahalanobis loss that includes the variances to account for the changing densities between known and unknown categories. Finally, we present a novel dataset for continual learning (HOWS-CL-25), especially suited for object recognition on a mobile robot, including 150,795 synthetic images of 25 household object categories. Our approach RECALL outperforms the current state of the art on CORe50 and iCIFAR-100 and reaches the best performance on HOWS-CL-25.

Keywords: Data Sets for Robot Learning, Deep Learning in Grasping and Manipulation
Abstract: When humans grasp objects in the real world, we often move our arms to hold the object in a different pose where we can use it. In contrast, typical lab settings only study the stability of the grasp immediately after lifting, without any subsequent re-positioning of the arm. However, the grasp stability could vary widely based on the object’s holding pose, as the gravitational torque and gripper contact forces could change completely. To facilitate the study of how holding poses affect grasp stability, we present PoseIt, a novel multi-modal dataset that contains visual and tactile data collected from a full cycle of grasping an object, re-positioning the arm to one of the sampled poses, and shaking the object. Using data from PoseIt, we can formulate and tackle the task of predicting whether a grasped object is stable in a particular held pose. We train an LSTM classifier that achieves 85% accuracy on the proposed task. Our experimental results show that multi-modal models trained on PoseIt achieve higher accuracy than using solely vision or tactile data and that our classifiers can also generalize to unseen objects and poses. The PoseIt dataset is publicly released here: https://github.com/CMURoboTouch/PoseIt.

Keywords: Bioinspired Robot Learning, Embedded Systems for Robotic and Automation, Autonomous Vehicle Navigation
Abstract: Autonomous Driving (AD) related features represent important elements for the next generation of mobile robots and autonomous vehicles focused on increasingly intelligent, autonomous, and interconnected systems. The applications involving the use of these features must provide, by definition, real-time decisions, and this property is key to avoid catastrophic accidents. Moreover, all the decision processes must require low power consumption, to increase the lifetime and autonomy of battery-driven systems. These challenges can be addressed through efficient implementations of Spiking Neural Networks (SNNs) on Neuromorphic Chips and the use of event-based cameras instead of traditional frame-based cameras.

In this paper, we present a new SNN-based approach, called LaneSNN, for detecting the lanes marked on the streets using the event-based camera input. We develop four novel SNN models characterized by low complexity and fast response, and train them using an offline supervised learning rule. Afterward, we implement and map the learned SNNs models onto the Intel Loihi Neuromorphic Research Chip. For the loss function, we develop a novel method based on the linear composition of Weighted binary Cross Entropy (WCE) and Mean Squared Error (MSE) measures. Our experimental results show a maximum Intersection over Union (IoU) measure of about 0.62 and very low power consumption of about 1 W. The best IoU is achieved with an SNN implementation that occupies only 36 neurocores on the Loihi processor while providing a low latency of less than 8 ms to recognize an image, thereby enabling real-time performance. The IoU measures provided by our networks are comparable with the state-of-the-art, but at a much low power consumption of 1 W.

Keywords: Deep Learning for Visual Perception, Computer Vision for Transportation, AI-Based Methods
Abstract: Identifying traversable space is one of the most important problems in autonomous robot navigation and is primarily tackled using learning-based methods. To alleviate the prohibitively high annotation-cost associated with labeling large and diverse datasets, research has recently shifted from tradi-tional supervised methods to focus on unsupervised and semi-supervised approaches. This work focuses on monocular road segmentation and proposes a practical, generic, and minimally-supervised approach based on task-specific feature extraction and pseudo-labeling. Building on recent advances in monocular depth estimation models, we process approximate dense depth maps to estimate pixel-wise road-plane distance maps. These maps are then used in both unsupervised and semi-supervised road segmentation scenarios. In the unsupervised case, we propose a pseudo-labeling pipeline that reaches state-of-the-art Intersection-over-Union (IoU), while reducing complexity and computations compared to existing approaches. We also investigate a semi-supervised extension to our method and find that even minimal labeling efforts can greatly improve results. Our semi-supervised experiments using as little as 1% & 10% of ground truth data, yield models scoring 0.9063 & 0.9332 on the IoU metric respectively. These results correspond to a comparative performance of 95.9% & 98.7% of a fully-supervised model’s IoU score, which motivates a pragmatic approach to labeling.

Keywords: Deep Learning for Visual Perception, Localization, Visual Learning
Abstract: In robotics, visual place recognition (VPR) is a continuous process that receives as input a video stream to produce a hypothesis of the robot's current position within a map of known places. This work proposes a taxonomy of the architectures used to learn sequential descriptors for VPR, highlighting different mechanisms to fuse the information from the individual images. This categorization is supported by a complete benchmark of experimental results that provides evidence of the strengths and weaknesses of these different architectural choices. The analysis is not limited to existing sequential descriptors, but we extend it further to investigate the viability of Transformers instead of CNN backbones. We further propose a new ad-hoc sequence-level aggregator called SeqVLAD, which outperforms prior state of the art on different datasets. The code is available at https://github.com/vandal-vpr/vg-transformers

Keywords: Deep Learning Methods, Localization, SLAM
Abstract: Reliable and accurate registration of point clouds is a challenging problem in robotics as well as in the domain of autonomous driving. In this paper, we address the task of aligning point clouds with low overlap, containing moving objects, and without prior information about the initial guess. We enhance classical ICP-based registration with neural feature-based matching to reliably find point correspondences. Our novel 3D convolutional and attention-based network is trained in an end-to-end fashion to learn features, which are well suited for matching and to rate the quality of the point correspondences. By utilizing a compression encoder, we can directly operate on a compressed map representation, making our approach well suited for operation under memory constraints. We evaluate our approach on point clouds obtained at completely different points in time, showing that our approach is able to reliably register point clouds even under those challenging conditions. The implementation of our approach and the preprocessed data can be accessed at https://github.com/PRBonn/DCPCR.

Keywords: Deep Learning Methods, Model Learning for Control, Machine Learning for Robot Control
Abstract: Recently Koopman operator has become a promising data-driven tool to facilitate real-time control for unknown nonlinear systems. It maps nonlinear systems into equivalent linear systems in embedding space, ready for real-time linear control methods. However, designing an appropriate Koopman embedding function remains a challenging task. Furthermore, most Koopman-based algorithms only consider nonlinear systems with linear control input, resulting in lousy prediction and control performance when the system is fully nonlinear with the control input. In this work, we propose an end-to-end deep learning framework to learn the Koopman embedding function and Koopman Operator together to alleviate such difficulties. We first parameterize the embedding function and Koopman Operator with the neural network and train them end-to-end with the K-steps loss function. We then design an auxiliary control network to encode the nonlinear state-dependent control term to model the nonlinearity in control input. For linear control, this encoded term is considered the new control variable instead, ensuring the linearity of the embedding space. Then we deploy Linear Quadratic Regulator (LQR) on the linear embedding space to derive the optimal control policy and decode the actual control input from the control net. Experimental results demonstrate that our approach outperforms other existing methods, reducing the prediction error by order-of-magnitude and achieving superior control performance in several nonlinear dynamic systems like damping pendulum, CartPole, and 7 Dof robotic manipulator.

Keywords: Domestic Robotics, Service Robotics, Robotics and Automation in Construction
Abstract: Cleaning robots are one of the market dominators in the commercialized robot space. So far, numerous robots have been introduced that can perform cleaning tasks in various settings, including floor, pavement, pool, lawn, windows, etc. However, none of the existing commercial cleaning robots targets the staircase, commonly found in multi-story buildings. Even though few works in the literature introduced robotic solutions for staircase cleaning, they primarily focused on cleaning the ascending staircase often, with a loose connection to access the descending staircase. In this paper, we propose a novel autonomous reconfigurable robotic platform called sTetro-D that can autonomously detect the descending staircase, approach the step, and perform area coverage in an unknown environment. The developed autonomy framework consists of two modes which are search mode and clean mode. In search mode, we implemented an RGB-D camera-based fusion technique wherein we combined the image bounding box from DCNN (Deep Convolution Neural Network) with the depth information to find the 3D first step pose that assists the robot in approaching it precisely. After the successful stair approach, the cleaning mode enables the staircase area coverage process. We described all these aspects and concluded with an experimental analysis of the proposed robotic system in a real-world scenario. The results demonstrate that the robot has a significant performance in detecting the descending staircase, staircase approach, and area coverage.

Keywords: Service Robotics, Modeling and Simulating Humans, Probabilistic Inference
Abstract: This work presents a non-parametric spatio-temporal model for mapping human activity by mobile autonomous robots in a long-term context. Based on Variational Gaussian Process Regression, the model incorporates prior information of spatial and temporal-periodic dependencies to create a continuous representation of human occurrences. The inhomogeneous data distribution resulting from movements of the robot is included in the model via a heteroscedastic likelihood function and can be accounted for as predictive uncertainty. Using a sparse formulation, data sets over multiple weeks and several hundred square meters can be used for model building. The experimental evaluation, based on multi- week data sets, demonstrates that the proposed approach outperforms the state of the art both in terms of predictive quality and subsequent path planning.

Keywords: Service Robotics, Social HRI
Abstract: In this paper, we report on a field study in which we employed two service robots in a bakery store as a sales promotion. Previous studies have explored public applications of service robots public such as shopping malls. However, more evidence is needed that service robots can contribute to sales in real stores. Moreover, the behaviors of customers and service robots in the context of sales promotions have not been examined well. Hence, the types of robot behavior that can be considered effective and the customers’ responses to these robots remain unclear. To address these issues, we installed two tele-operated service robots in a bakery store for nearly 2 weeks, one at the entrance as a greeter and the other one inside the store to recommend products. The results show a dramatic increase in sales during the days when the robots were applied. Furthermore, we annotated the video recordings of both the robots’ and customers' behavior. We found that although the robot placed at the entrance successfully attracted the interest of the passersby, no apparent increase in the number of customers visiting the store was observed. However, we confirmed that the recommendations of the robot operating inside the store did have a positive impact. We discuss our findings in detail and provide both theoretical and practical recommendations for future research and applications.

Keywords: Service Robotics, Safety in HRI, Robot Safety
Abstract: Autonomous vehicles are designed to elevate the efficiency of assigned tasks and ensure the safety of the environment in which they operate. This paper presents a research study focused on shared autonomy using a multi-layer fuzzy logic framework to build a relationship between an autonomous self-reconfigurable robot and a human user by switching control to the teleoperator to assist the robot when it faces challenging scenarios while keeping a good performance and maintaining a safe environment. A novel multi-layer fuzzy logic decision process with shared autonomy for a safety framework is proposed. It evaluates safety based on the robot's multi-sensor inputs, the teleoperator's attention level, and the configuration state of the self-reconfigurable robot and switches the operation mode, robot speed gain, and configuration state for performance and safety without compromises. The experimental outcome successfully demonstrates the self-reconfigurable robot’s capability to navigate safely using shared autonomy in real-world pavement scenarios using the proposed algorithm during autonomous navigation.

Keywords: Service Robotics, Reactive and Sensor-Based Planning, Autonomous Vehicle Navigation
Abstract: Autonomous navigation in highly populated areas remains a challenging task for robots because of the difficulty in guaranteeing safe interactions with pedestrians in unstructured situations. In this work, we present a crowd navigation control framework that delivers continuous obstacle avoidance and post-contact control evaluated on an autonomous personal mobility vehicle. We propose evaluation metrics for accounting efficiency, controller response and crowd interactions in natural crowds. We report the results of over 110 trials in different crowd types: sparse, flows, and mixed traffic, with low- (< 0.15 ppsm), mid- (< 0.65 ppsm), and high- (< 1 ppsm) pedestrian densities. We present comparative results between two low-level obstacle avoidance methods and a baseline of shared control. Results show a 10% drop in relative time to goal on the highest density tests, and no other efficiency metric decrease. Moreover, autonomous navigation showed to be comparable to shared-control navigation with a lower relative jerk and significantly higher fluency in commands indicating high compatibility with the crowd. We conclude that the reactive controller fulfils a necessary task of fast and continuous adaptation to crowd navigation, and it should be coupled with high-level planners for environmental and situational awareness.

Keywords: Service Robotics, Product Design, Development and Prototyping, Motion and Path Planning
Abstract: Area coverage is demanded from the robots utilized in application domains such as floor cleaning. Even though many advanced coverage algorithms have been developed, the area coverage performance is limited due to the inaccessibility of narrow spaces caused by physical constraints. Reconfigurable robots have been introduced to overcome this limitation where reconfigurability could help in assessing narrow spaces. Nevertheless, the state-of-the-art reconfigurable robots are not capable of changing the morphology size and shape as a single entity. Therefore, this paper proposes a novel design of a reconfigurable robot with a size-adaptive coverage strategy. The reconfiguration mechanism is designed in such a way that the robot can independently expand or shrink its size along the principal planar axes, where the behavior allows the change of size and shape. The coverage strategy is based on boustrophedon motion and the A* algorithm modified for accessing narrow areas using the size adaptability. The design of the robot is detailed in the paper, including electro-mechanical aspects, design considerations, and the coverage path planning method. Experiments have been conducted using a prototype of the proposed design to analyze and evaluate the characteristics and the performance of the robot. The results show that the proposed robot design can improve the productivity of a floor cleaning robot in terms of area coverage and coverage time.

Keywords: Engineering for Robotic Systems, Performance Evaluation and Benchmarking
Abstract: Service robots are mobile autonomous robots, often operating in uncertain and difficult environments. While being increasingly popular, engineering service robots is challenging. Especially, evolving them from prototype to deployable product requires effective validation and verification, assuring the robot's correct and safe operation in the target environment. While testing is the most common validation and verification technique used in practice, surprisingly little is known about the actual testing practices and technologies used in the service robotics domain. We present an experience report on field testing of an industrial-strength service robot, as it transitions from lab experiments to an operational environment. We report challenges and solutions, and reflect on their effectiveness. Our long-term goal is to establish empirically-validated testing techniques for service robots. This experience report constitutes a necessary, but self-contained first step, exploring field testing practices in detail. Our data sources are detailed test artifacts and developer interviews. We model the field testing process and describe test-case design practices. We discuss experiences from performing these field tests over a 10-month test campaign.

Keywords: Service Robotics, Task Planning, Planning under Uncertainty
Abstract: Flexible task planning continues to pose a difficult challenge for robots, where a robot is unable to creatively adapt their task plans to new or unseen problems, which is mainly due to the limited knowledge it may have about its actions and world. Because of this, robots typically cannot exploit knowledge or concepts in a way that mimics human creativity. Motivated by our ability as humans to adapt, we explore how task plans from a knowledge graph, known as the Functional Object-Oriented Network (FOON), can be generated for novel problems requiring concepts that are not readily available in the robot's knowledge base. Knowledge from 140 cooking recipes are structured in a FOON knowledge graph, which is used for acquiring task plan sequences known as task trees. Task trees can be modified to replicate recipes in a FOON knowledge graph format, which can be useful for expanding FOON with new recipes containing unknown object and state combinations, by relying upon semantic similarity. We demonstrate the power of task tree generation to create task trees with never-before-seen ingredient and state combinations as seen in recipes from the Recipe1M+ dataset, with which we evaluate the quality of the trees based on how accurately they depict newly added ingredients. Our experimental results show that our framework is able to provide task sequences with 76% accuracy.

Keywords: Service Robotics, Industrial Robots, Computer Vision for Manufacturing
Abstract: With the rapid development of powder-based additive manufacturing, depowdering, a process of removing unfused powder that covers 3D-printed parts, has become a major bottleneck to further improve its productiveness. Traditional manual depowdering is extremely time-consuming and costly, and some prior automated systems either require pre-depowdering or lack adaptability to different 3D-printed parts. To solve these problems, we introduce a robotic system that automatically removes unfused powder from the surface of 3D-printed parts. The key component is a visual perception system, which consists of a pose-tracking module that tracks the 6D pose of powder-occluded parts in real-time, and a progress estimation module that estimates the depowdering completion percentage. The tracking module can be run efficiently on a laptop CPU at up to 60 FPS. Experiments show that our system can remove unfused powder from the surface of various 3D-printed parts without causing any damage. To the best of our knowledge, this is one of the first vision-based depowdering systems that adapt to parts with various shapes without the need for pre-depowdering.

Keywords: Manipulation Planning, Dexterous Manipulation, Optimization and Optimal Control
Abstract: Given an object, an environment, and a goal pose, how should a robot make contact to move it? Solving this problem requires reasoning about rigid-body dynamics, object and environment geometries, and hybrid contact mechanics. This paper proposes a hierarchical framework that solves this problem in 2D worlds, with polygonal objects and point fingers. To achieve this, we decouple the problem in three stages: 1) a high-level textit{graph search} over regions of free-space, 2) a medium-level randomized textit{motion planner} for the object motion, and 3) a low-level textit{contact-trajectory optimization} for the robot and environment contacts. In contrast to the state of the art, this approach does not rely on handcrafted primitives and can still be solved efficiently. This algorithm does not require seeding and can be applied to complex object shapes and environments. We validate this framework with extensive simulated experiments showcasing long-horizon and contact-rich interactions. We demonstrate how our algorithm can reliably solve complex planar manipulation problems in the order of seconds.

Keywords: Manipulation Planning, Motion and Path Planning
Abstract: To economically deploy robotic manipulators the programming and execution of robot motions must be swift. To this end, we propose a novel, constraint-based method to intuitively specify sequential manipulation tasks and to compute time-optimal robot motions for such a task specification. Our approach follows the ideas of constraint-based task specification by aiming for a minimal and object-centric task description that is largely independent of the underlying robot kinematics. We transform this task description into a non-linear optimization problem. By solving this problem we obtain a (locally) time-optimal robot motion, not just for a single motion, but for an entire manipulation sequence. We demonstrate the capabilities of our approach in a series of experiments involving five distinct robot models, including a highly redundant mobile manipulator.

Keywords: Manipulation Planning, Motion and Path Planning, Dexterous Manipulation
Abstract: Contact-rich manipulation is challenging due to dynamically-changing physical constraints by the contact mode changes undergone during manipulation. This paper proposes a versatile local planning and control framework for contact-rich manipulation that determines the continuous control action under variable contact modes online. We model the physical characteristics of contact-rich manipulation by quasistatic dynamics and complementarity constraints. We then propose a linear complementarity quadratic program (LCQP) to efficiently determine the control action that implicitly includes the decisions on the contact modes under these constraints. In the LCQP, we relax the complementarity constraints to alleviate ill-conditioned problems that are typically caused by measure noises or model miss-matches. We conduct dynamical simulations on a 3D physical simulator and demonstrate that the proposed method can achieve various contact-rich manipulation tasks by determining the control action including the contact modes in real-time.

Keywords: Manipulation Planning, Task Planning
Abstract: With the field of rigid-body robotics having matured in the last fifty years, routing, planning, and manipulation of deformable objects have recently emerged as a more untouched research area in many fields ranging from surgical robotics to industrial assembly and construction. Routing approaches for deformable objects which rely on learned implicit spatial representations (e.g., Learning-from-Demonstration methods) make them vulnerable to changes in the environment and the specific setup. On the other hand, algorithms that entirely separate the spatial representation of the deformable object from the routing and manipulation, often using a representation approach independent of planning, result in slow planning in high dimensional space.

This paper proposes a novel approach to routing deformable one-dimensional objects (e.g., wires, cables, ropes, sutures, threads). This approach utilizes a compact representation for the object, allowing efficient and fast online routing. The spatial representation is based on the geometrical decomposition of the space into convex subspaces, resulting in a discrete coding of the deformable object configuration as a sequence. With such a configuration, the routing problem can be solved using a fast dynamic programming sequence matching method that calculates the next routing move. The proposed method couples the routing and efficient configuration for improved planning time. Our simulation and real experiments show the method correctly computing the next manipulation action in sub-millisecond time and accomplishing various routing and manipulation tasks.

Keywords: Learning from Demonstration, Manipulation Planning, Imitation Learning
Abstract: This paper presents an approach that generates the overall trajectory of mobile manipulators for a complex mission consisting of several sub-tasks. Parametric dynamic movement primitives (PDMPs) can quickly generalize the online motion of robot manipulation by learning multiple demonstrations in offline. However, regarding complex missions consisting of multiple sub-tasks, a large number of demonstrations are required for full generalization, which is impractical. In this paper, we propose a framework that reduces the number of demonstrations for a complex mission. In the proposed method, complex demonstrations are segmented into multiple unit motions representing sub-tasks, and one PDMP is formed per each segment, resulting in multiple PDMPs. The phase decision process determines which sub-task and associated PDMPs to be executed online, allowing multiple PDMPs to be autonomously configured within an integrated framework. In order to generalize the execution time and regional goal in each phase, the Gaussian process regression (GPR) is applied. Simulation results from two different scenarios confirm that the proposed framework not only effectively reduces the number of demonstrations but also improves generalization performance. The actual experiments also demonstrate that the mobile manipulators effectively perform complex missions through the proposed framework.

Keywords: Manipulation Planning, Optimization and Optimal Control, Nonholonomic Motion Planning
Abstract: This paper is about fast slosh-free fluid transportation. Existing approaches are either computationally heavy or only suitable for specific robots and container shapes. We model the end effector as a point mass suspended by a spherical pendulum and study the requirements for slosh-free motion and the validity of the point mass model. In this approach, slosh-free trajectories are generated by controlling the pendulum’s pivot and simulating the motion of the point mass. We cast the trajectory optimization problem as a quadratic program—this strategy can be used to obtain valid control inputs. Through simulations and experiments on a 7 DoF Franka Emika Panda robot we validate the effectiveness of the proposed approach.

Keywords: Manipulation Planning, Planning under Uncertainty, Assembly
Abstract: Factory automation robot systems often depend on specially-made jigs that precisely position each part, which increases the system's cost and limits flexibility. We propose a method to determine the 3D pose of an object with high precision and confidence, using only parallel robotic grippers and no parts-specific jigs. Our method automatically generates a sequence of actions that ensures that the real-world position of the physical object matches the system's assumed pose to sub-mm precision. Furthermore, we propose the use of ``extrinsic" actions, which use gravity, the environment and the gripper geometry to significantly reduce or even eliminate the uncertainty about an object's pose. We show in simulated and real-robot experiments that our method outperforms our previous work, at success rates over 95%. The source code will be made public at github.com/omron-sinicx.

Keywords: Force and Tactile Sensing, Dexterous Manipulation
Abstract: In robots, nonprehensile manipulation operations such as pushing are a useful way of moving large, heavy, or unwieldy objects, moving multiple objects at once, or reducing uncertainty in the location or pose of objects. In this study, we propose a reactive and adaptive method for robotic pushing that uses rich feedback from a high-resolution optical tactile sensor to control push movements instead of relying on analytical or data-driven models of push interactions. Specifically, we use goal-driven tactile exploration to actively search for stable pushing configurations that cause the object to maintain its pose relative to the pusher while incrementally moving the pusher and object toward the target. We evaluate our method by pushing objects across planar and curved surfaces. For planar surfaces, we show that the method is accurate and robust to variations in initial contact position/angle, object shape, and start position; for curved surfaces, the performance is degraded slightly. An immediate consequence of our work is that it shows that explicit models of push interactions might be sufficient but are not necessary for this type of task. It also raises the interesting question of which aspects of the system should be modeled to achieve the best performance and generalization across a wide range of scenarios. Finally, it highlights the importance of testing on nonplanar surfaces and in other more complex environments when developing new methods for robotic pushing.

Keywords: Deep Learning in Grasping and Manipulation, Performance Evaluation and Benchmarking, Imitation Learning
Abstract: Autonomous fabric manipulation is a longstanding challenge in robotics, but evaluating progress is difficult due to the cost and diversity of robot hardware. Using Reach, a cloud robotics platform that enables low-latency remote execution of control policies on physical robots, we present the first systematic benchmarking of fabric manipulation algorithms on physical hardware. We develop 4 novel learning-based algorithms that model expert actions, keypoints, reward functions, and dynamic motions, and we compare these against 4 learning-free and inverse dynamics algorithms on the task of folding a crumpled T-shirt with a single robot arm. The entire lifecycle of data collection, model training, and policy evaluation was performed remotely without physical access to the robot workcell. Results suggest a new algorithm combining imitation learning with analytic methods achieves human-level performance on the flattening task and 93% of human-level performance on the folding task. See https://sites.google.com/berkeley.edu/cloudfolding for all data, code, models, and supplemental material.

Keywords: Computer Vision for Automation, Computer Vision for Transportation, Intelligent Transportation Systems
Abstract: Neural Networks are the state-of-the-art technology for environmental perception in applications such as autonomous driving. However, they require a large amount of training data in order to perform well, making the selection and annotation of sensor data a time-consuming and expensive task. Active learning is a promising approach to reduce the required amount of training data by selecting samples for annotation that are expected to improve the neural network the most. In this work, we propose a sequence-based active learning approach that selects sequences of consecutive frames instead of individual images. This allows to evaluate tracking algorithms and to reduce the annotation effort by interpolating labels between frames. Our approach is compared to a random sampling strategy as baseline. Over 15 iterations, both approaches select 1000 additional images for training in each iteration. The performance of the neural network trained on the data selected by our sequence-based active learning approach is compared to the performance of the network trained on the data select by the baseline approach. The results show that sequence-based active learning can reduce the required amount of training data by up to 25% while reaching a similar performance. Furthermore, sequence-based active learning can improve the neural network’s overall performance by 2% compared to a random sampling strategy. In this work, the proposed method was evaluated with a new dataset consisting of 15 scenes in railway environments. The dataset has 45,888 frames in total, 14,513 frames contain persons on railway stations or close to tracks.

Keywords: Computer Vision for Transportation, RGB-D Perception, Autonomous Vehicle Navigation
Abstract: This paper proposes a novel 3D single-object tracker to more stably, accurately, and faster track objects, even if they are temporarily missed. Our idea is to utilize spatial-temporal data association to achieve object tracking robustly, and it consists of two main parts. We firstly employ a temporal motion model cross frames to estimate the object's temporal information and update the region of interest(ROI). The advanced detector only focuses on ROI rather than the whole scene to generate the spatial position. Second, we introduce a new pairwise evaluation system to exploit spatial-temporal data association in point clouds. The proposed evaluation system considers detection confidence, orientation offset, and objects distance to more stably achieve object matching. Then, we update the predicted state based on the pairwise spatial-temporal data. Finally, we utilize the previous trajectory to enhance the accuracy of static tracking in the refinement scheme. Experiments on the KITTI and nuScenes tracking datasets demonstrate that our method outperforms other state-of-the-art methods by a large margin (a 10% improvement and 280 FPS on a single NVIDIA 1080Ti GPU). Compared with multi-object tracking, our tracker also has superiority.

Keywords: Deep Learning for Visual Perception, Computer Vision for Automation, RGB-D Perception
Abstract: This paper proposes a self-supervised monocular image-to-depth prediction framework that is trained with an end-to-end photometric loss that handles not only 6-DOF camera motion but also 6-DOF moving object instances. Self-supervision is performed by warping the images across a video sequence using depth and scene motion including object instances. One novelty of the proposed method is the use of the multi-head attention of the transformer network that matches moving objects across time and models their interaction and dynamics. This enables accurate and robust pose estimation for each object instance. Most image-to-depth predication frameworks make the assumption of rigid scenes, which largely degrades their performance with respect to dynamic objects. Only a few SOTA papers have accounted for dynamic objects. The proposed method is shown to outperform these methods on standard benchmarks and the impact of the dynamic motion on these benchmarks is exposed. Furthermore, the proposed image-to-depth prediction framework is also shown to be competitive with SOTA video-to-depth prediction frameworks.

Keywords: Computer Vision for Transportation, Recognition, Gesture, Posture and Facial Expressions
Abstract: Traditional video-based human activity recognition has experienced remarkable progress linked to the rise of deep learning, but this effect was slower as it comes to the downstream task of driver behavior understanding. Understanding the situation inside the vehicle cabin is essential for Advanced Driving Assistant System (ADAS) as it enables identifying distraction, predicting driver’s intent and leads to more convenient human-vehicle interaction. At the same time, driver observation systems face substantial obstacles as they need to capture different granularities of driver states while the complexity of such secondary activities grows with the rising automation and increased driver freedom. Furthermore, a model is rarely deployed under conditions identical to the ones in the training set, as sensor placements and types vary from vehicle to vehicle, constituting a substantial obstacle forreal-life deployment of data-driven models. In this work, we present a novel vision based framework for recognizing secondary driver behaviours based on visual transformers and an additional augmented feature distribution calibration module. This module operates in the latent feature-space enriching and diversifying the training set at feature-level in order to improve (1) generalization to novel data appearances, (e.g. ,sensor changes) and (2) recognition of driver behaviours un-der represented in the training set. Our framework consistently leads to better recognition rates, surpassing previous state of-the-art results of the public Drive&Act benchmark on all granularity levels. Our code will be made publicly available at https://github.com/KPeng9510/TransDARC.

Keywords: Computer Vision for Transportation, Autonomous Vehicle Navigation, Intelligent Transportation Systems
Abstract: Experienced human drivers always make safe driving decisions with selectively observing the front, rear and side-view mirrors. Several end-to-end methods have been proposed to learn driving models with multi-view visual information. However, these benchmarking methods lack semantic understanding of multi-view image contents, where human drivers usually reasoning these information for decision making with different visual region of interests. In this paper, we propose an attention-based deep learning method to learn a driving model with input of surround-view visual information and the route planner, in which a multi-view attention module is designed for obtaining region of interests from human drivers. We evaluate our model on the Drive360 dataset with comparison of benchmarking deep driving models. Results demonstrate that our model achieves a competitive accuracy in both steering angle and speed prediction than benchmarking methods. Code is available at https://github.com/jet-uestc/MVA-Net.

Keywords: Computer Vision for Transportation, Motion and Path Planning, Deep Learning for Visual Perception
Abstract: In this paper, we present a framework to assess the quality of a pedestrian detector in an autonomous driving scenario. To do this, we exploit performance metrics from the domain of computer vision on one side and so-called threat metrics from the motion planning domain on the other side. Based on a reachability analysis that accounts for the uncertainty in future motions of other traffic participants, we can determine the worst-case threat from the planning domain and relate it to the corresponding detection from the visual input. Our evaluation results for a RetinaNet on the Argoverse 1.1 [1] dataset show that already a rather simple threat metric such as time-to-collision (TTC) allows to select potentially dangerous interactions between the ego vehicle and a pedestrian when purely vision-based detections fail, even if they are passed to a subsequent object tracker. In addition, our results show that two different DNNs (Deep Neural Networks) with comparable performance differ significantly in the number of critical scenarios that we can identify with our method.

Keywords: Computer Vision for Transportation, Deep Learning for Visual Perception, Representation Learning
Abstract: Robust detection of moving vehicles is a critical task for any autonomously operating outdoor robot or self-driving vehicle. Most modern approaches for solving this task rely on training image-based detectors using large-scale vehicle detection datasets such as nuScenes or the Waymo Open Dataset. Providing manual annotations is an expensive and laborious exercise that does not scale well in practice. To tackle this problem, we propose a self-supervised approach that leverages audio-visual cues to detect moving vehicles in videos. Our approach employs contrastive learning for localizing vehicles in images from corresponding pairs of images and recorded audio. In extensive experiments carried out with a real-world dataset, we demonstrate that our approach provides accurate detections of moving vehicles and does not require manual annotations. We furthermore show that our model can be used as a teacher to supervise an audio-only detection model. This student model is invariant to illumination changes and thus effectively bridges the domain gap inherent to models leveraging exclusively vision as the predominant modality.

Keywords: Computer Vision for Transportation, Deep Learning for Visual Perception, Semantic Scene Understanding
Abstract: Semantic segmentation provides scene understanding capability by performing pixel-wise classification of objects within an image. However, the sensitivity of such algorithms towards domain changes requires fine-tuning using an annotated dataset for each novel domain, which is expensive to construct and inefficient. We highlight that irrespective of the training dataset, structural properties of scenes remain the same hence domain sensitivity arises from training methodology. Thus, in this paper, we propose a domain alignment approach wherein multiple synthetic source domains are used to train an underlying segmentation network such that it performs consistently in unknown real target domains. Towards this end, we propose a pixel-wise supervised contrastive learning framework that enforces constraints in latent space resulting in features belonging to the same class being clustered closely and away from different classes. This approach allows for better capturing of global and local semantics while providing domain invariant properties. Our approach can be easily incorporated into prior semantic segmentation approaches without the significant computational overhead. We empirically demonstrate the efficacy of the proposed approach on GTAV to Cityscapes, GTAV+Synthia to Cityscapes, and GTAV+Synthia+Synscapes to Cityscapes scenarios and report state-of-the-art (SoTA) performance without requiring access to images from the target domain.

Keywords: Deep Learning for Visual Perception, Omnidirectional Vision, Computer Vision for Transportation
Abstract: Self-supervised monocular depth estimation has been widely investigated to estimate depth images and relative poses from RGB images. This framework is promising because the depth and pose networks can be trained from just time-sequence images without the need for the ground truth depth and poses.

In this work, we estimate the depth around a robot (360 degree view) using time-sequence spherical camera images, from a camera whose parameters are unknown. We propose a learnable axisymmetric camera model which accepts distorted spherical camera images with two fisheye camera images as well as pinhole camera images. In addition, we trained our models with a photo-realistic simulator to generate ground truth depth images to provide supervision. Moreover, we introduced loss functions to provide floor constraints to reduce artifacts that can result from reflective floor surfaces. We demonstrate the efficacy of our method using the spherical camera images from the GO Stanford dataset and pinhole camera images from the KITTI dataset to compare our method’s performance with that of baseline method in learning the camera parameters.

Keywords: Aerial Systems: Applications, SLAM, Human Detection and Tracking
Abstract: Unmanned aerial vehicles (UAVs) have been used for many applications in recent years, from urban search and rescue, to agricultural surveying, to autonomous underground mine exploration. However, deploying UAVs in tight, indoor spaces, especially close to humans, remains a challenge. One solution, when limited payload is required, is to use micro-UAVs, which pose less risk to humans and typically cost less to replace after a crash. However, micro-UAVs can only carry a limited sensor suite, e.g. a monocular camera instead of a stereo pair or LiDAR, complicating tasks like dense mapping and markerless multi-person 3D human pose estimation, which are needed to operate in tight environments around people. Monocular approaches to such tasks exist, and dense monocular mapping approaches have been successfully deployed for UAV applications. However, despite many recent works on both marker-based and markerless multi-UAV single-person motion capture, markerless single-camera multi-person 3D human pose estimation remains a much earlier-stage technology, and we are not aware of existing attempts to deploy it in an aerial context. In this paper, we present what is thus, to our knowledge, the first system to perform simultaneous mapping and multi-person 3D human pose estimation from a monocular camera mounted on a single UAV. In particular, we show how to loosely couple state-of-the-art monocular depth estimation and monocular 3D human pose estimation approaches to reconstruct a hybrid map of a populated indoor scene in real time. We validate our component-level design choices via extensive experiments on the large-scale ScanNet and GTA-IM datasets. To evaluate our system-level performance, we also construct a new Oxford Hybrid Mapping dataset of populated indoor scenes.

Keywords: Aerial Systems: Applications, Environment Monitoring and Management, Probabilistic Inference
Abstract: Chemical gas dispersion poses considerable threat to humans, animals and the environment. The research areas of gas source localization and gas distribution mapping aim to localize the source of gas leaks and map the gas plume respectively, in order to help the coordination of swift rescue missions. Although very similar, these two areas are often treated separately in literature. In some cases, inferences on the gas distribution are made a posteriori from the source location, or vice-versa. In this paper, we introduce GaSLAM, a methodology that couples the estimation of the gas map and the source location using two state of the art algorithms with a novel navigation strategy based on informative quantities. The synergistic approach allows our algorithm to achieve a good estimation of both objectives and push the navigation strategies towards informative areas of the experimental volume. We validate the algorithm in simulation and with physical experiments in varying environmental conditions. We show that the algorithm improves on the source location estimate compared to a similar approach found in literature, and is able to deliver good quality maps of the gas distribution.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications, Dynamics
Abstract: Accessing and interacting with difficult to reach surfaces at various orientations is of interest within a variety of industrial contexts. Thus far, the predominant robotic solution to such a problem has been to leverage the maneuverability of a fully actuated, omnidirectional aerial manipulator. Such an approach, however, requires a specialised system with a high relative degree of complexity, thus reducing platform endurance and real-world applicability. The work here presents a new aerial system composed of a parallel manipulator and conventional, underactuated multirotor flying base to demonstrate interaction with vertical and non-vertical surfaces. Our solution enables compliance to external disturbance on both subsystems, the manipulator and flying base, independently with a goal of improved overall system performance when interacting with surfaces. To achieve this behaviour, an admittance control strategy is implemented on various layers of the flying base's dynamics together with torque limits imposed on the manipulator actuators. Experimental evaluations show that the proposed system is compliant to external perturbations while allowing for differing interaction behaviours as compliance parameters of each subsystem are altered. Such capabilities enable an adjustable form of dexterity in completing sensor installation, inspection and aerial physical interaction tasks. A video of our system interacting with various surfaces can be found here: https://youtu.be/38neGb8-lXg.

Keywords: Aerial Systems: Applications, Soft Robot Applications, Grippers and Other End-Effectors
Abstract: Rapid aerial grasping through robots can lead to many applications that utilize fast and dynamic picking and placing of objects. Rigid grippers traditionally used in aerial manipulators require high precision and specific object geometries for successful grasping. We propose RAPTOR, a quadcopter platform combined with a custom Fin Ray® gripper to enable more flexible grasping of objects with different geometries, leveraging the properties of soft materials to increase the contact surface between the gripper and the objects. To reduce the communication latency, we present a new lightweight middleware solution based on Fast DDS (Data Distribution Service) as an alternative to ROS (Robot Operating System). We show that RAPTOR achieves an average of 83% grasping efficacy in a real-world setting for four different object geometries while moving at an average velocity of 1 m/s during grasping. In a high-velocity setting, RAPTOR supports up to four times the payload compared to previous works. Our results highlight the potential of aerial drones in automated warehouses and other manipulation applications where speed, swiftness, and robustness are essential while operating in hard-to-reach places.

Keywords: Aerial Systems: Applications, Intelligent Transportation Systems, Control Architectures and Programming
Abstract: This work presents an approach for dealing with suspended-cable load transportation using unmanned aerial vehicles (UAVs), specifically when the cargo overcomes the lifting capacity. Herein, this approach is referred to as the Side-Pull Maneuver (SPM). This maneuver is an alternative and viable strategy for cases where there is no impediment or restriction to dragging the load along a surface, such as with pastures or marine environments. The proposal is based on a joint observation of the thrust and altitude of the UAV. To make this possible, the high-level rigid-body dynamics model is described and represented as an underactuated system. Its altitude-rate control input is then analyzed during flight. A flight state supervisor decides whether the cargo should be carried by lifting or by side-pulling, or whether it should be labeled as non-transportable. Comparative real experiments validate the proposal according to which maneuver (lifting or dragging) is performed for transport.

Keywords: Aerial Systems: Applications, Product Design, Development and Prototyping, Mechanism Design
Abstract: One of the major disadvantages of drones is their limited flight time. This paper introduces a new concept and mechanism for an onboard system that physically replaces batteries during flight, analogous to “aerial refueling”. This capability allows drones to remain in mid-air indefinitely while pursuing their mission without forcing them to change flight paths for logistical needs. The concept is composed of an additional UAV array that delivers new batteries from various ground points. We first describe the Flying Hot-Swap Battery (FHSB) system’s conceptual design. This unique design uses a FIFO logical process and the force of gravity to replace the energy source. The main innovation involves combining the ability to receive a battery from an external source, connect mechanically, hot-swap between the batteries, and dispose of the discharged battery. We report on the design of a dedicated battery cartridge for reception and connection in any spatial variation or orientation. Each component has a duplicate that works independently to increase system redundancy. Finally, we present the multiple experiments conducted to test the FHSB. The prototype successfully hovered and connected the battery in various reception orientations and hot-swapped the battery, thus maintaining the drone’s continuous power supply. This proof of concept of a complete battery replacement process during flight took an average replacement time of 15.2 seconds, which is only a 0.81% energy loss, thus enabling the drone to continue flying indefinitely without needing to modify its flight path (see attached video).

Keywords: Aerial Systems: Mechanics and Control, Automation Technologies for Smart Cities, Environment Monitoring and Management
Abstract: In city-wide weather prediction, wind gust information can be obtained using unmanned aerial vehicles (UAVs). Although wind sensors are available, an algorithm-based active estimation can be helpful not only as a weightless substitute but also as feedback for robust control. This paper aims to estimate the wind gusts affecting the quadrotors (a type of UAV) as the input disturbances by using a frequency-based nonlinear disturbance observer (NDOB). To obtain highly accurate estimations, frequency is considered as the main design parameter, thereby focusing the estimation on the frequency range of the wind gusts. The NDOB is developed using the Takagi-Sugeno (T-S) fuzzy framework. In this approach, the twelfth-order nonlinear model is approximated into a sixth-order T-S fuzzy model to reduce computational cost. A two-step verification method is presented, which includes MATLAB/Simulink simulations and the experiments performed using a 2.5 kg quadrotor.

Keywords: Aerial Systems: Applications
Abstract: In this paper, we address the vision-based detection and tracking problems of multiple aerial vehicles using a single camera and Inertial Measurement Unit (IMU) as well as the corresponding perception consensus problem (i.e., uniqueness and identical IDs across all observing agents). We design several vision-based decentralized Bayesian multi-tracking filtering strategies to resolve the association between the incoming unsorted measurements obtained by a visual detector algorithm and the tracked agents. We compare their accuracy in different operating conditions as well as their scalability according to the number of agents in the team. This analysis provides useful insights about the most appropriate design choice for the given task. We further show that the proposed perception and inference pipeline which includes a Deep Neural Network (DNN) as visual target detector is lightweight and capable of concurrently running control and planning with Size, Weight, and Power (SWaP) constrained robots on-board. Experimental results show the effective tracking of multiple drones in various challenging scenarios such as heavy occlusions.

Keywords: Optimization and Optimal Control, Swarm Robotics, Methods and Tools for Robot System Design
Abstract: The collective operation of robots (such as unmanned aerial vehicles or UAVs) operating as a team or swarm is affected by their individual capabilities, which in turn is dependent on their physical design, aka morphology. However, with the exception of a few (albeit ad hoc) evolutionary robotics methods, there has been very little work on understanding the interplay of morphology and collective behavior, especially given the lack of computational frameworks to concurrently search for the robot morphology and the hyper-parameters of their behavior model that jointly optimize the collective (team) performance. To address this gap, this paper proposes a new co-design framework. Here the exploding computational cost of an otherwise nested morphology/behavior co-design is effectively alleviated through the novel concept of ``talent" metrics; while also allowing significantly better solutions compared to the typically sub-optimal sequential morphologytobehavior design approach. This framework comprises four major steps: talent metrics selection, talent Pareto exploration (a multi-objective morphology optimization process), behavior optimization, and morphology finalization. This co-design concept is demonstrated by applying it to design UAVs that operate as a team to localize signal sources (e.g., in victim search and hazard localization), where the collective behavior is driven by a recently reported batch Bayesian search algorithm called Bayes-Swarm. Our case studies show that the outcome of co-design provides significantly higher success rates in signal source localization compared to a baseline design, across a variety of signal environments and teams with 6 to 15 UAVs. Moreover, this co-design process provides two orders of magnitude reduction in computing time compared to a projected nested design approach.

Keywords: Medical Robots and Systems, Computer Vision for Medical Robotics
Abstract: This paper deals with the control of a laser spot in the context of minimally invasive surgery of the middle ear, e.g., cholesteatoma removal. More precisely, our work is concerned with the exhaustive burring of residual infected cells after primary mechanical resection of the pathological tissues since the latter cannot guarantee the treatment of all the infected tissues, the remaining infected cells cause regeneration of the diseases in 20%-25% of cases, which require a second surgery 12-18 months later. To tackle such a complex surgery, we have developed a robotic platform that consists of the combination of a macro-scale system (7 degrees of freedoms (DoFs) robotic arm) and a micro-scale flexible system (2 DoFs) which operates inside the middle ear cavity. To be able to treat the residual cholesteatoma regions, we proposed a method to generate optimal laser automatically scanning trajectories inside the areas and between them. The trajectories are tacked using an image-based control scheme. The proposed method and materials were validated experimentally using the lab-made robotic platform. The obtained results in terms of accuracy and behaviour meet the laser surgery requirements perfectly.

Keywords: Medical Robots and Systems, Virtual Reality and Interfaces, Haptics and Haptic Interfaces
Abstract: Spina Bifida (SB) is a birth defect developed during the early stage of pregnancy in which there is an incomplete closing of the spine around the spinal cord. The growing interest in fetoscopic Spina Bifida repair, which is performed in fetuses who are still in the pregnant uterus, prompts the need for appropriate training. The learning curve for such procedures is steep and requires excellent procedural skills.

Computer-based virtual reality (VR) simulation systems offer a safe, cost-effective, and configurable training environment free from ethical and patient safety issues. However, to the best of our knowledge, there are currently no commercial or experimental VR training simulation systems available for fetoscopic SB-repair procedures. In this paper, we propose a novel VR simulator for core manual skills training for SB-repair.

An initial simulation realism validation study was carried out by obtaining subjective feedback (face and content validity) from 14 clinicians. The overall simulation realism was on average marked 4.07 on a 5-point Likert scale (1 - ‘very unrealistic’, 5 - ‘very realistic’). Its usefulness as a training tool for SB-repair as well as in learning fundamental laparoscopic skills was marked 4.63 and 4.80, respectively.

These results indicate that VR simulation of fetoscopic procedures may contribute to surgical training without putting fetuses and their mothers at risk. It could also facilitate wider adaptation of fetoscopic procedures in place of much more invasive open fetal surgeries.

Keywords: Medical Robots and Systems
Abstract: Percutaneous coronary intervention (PCI) involves the delivery of a flexible submillimeter guidewire and existing x-ray based approaches impose significant ironing radiation. The use of magnetic resonance imaging (MRI) for intraoperative guidance has the advantages of not only being safe but also having high positioning accuracy and excellent tissue contrast. This paper develops a pneumatically driven MR-conditional delivery mechanism for the ease of manipulation of the guidewire in vivo. It incorporates newly developed rotary pneumatic step motors and a pneumatic slip ring for actuation and decoupling of translational and rotational motions. An effective clamping mechanism for the locking and releasing of the guidewire is also incorporated. The proposed pneumatic slip ring mechanism decouples six gas lines, where four are used to supply a pneumatic step motor for translational motion, and two for the clamping mechanism. High friction sil sleeve is used to hold the guidewire firmly and the rotary pneumatic motor has excellent sealing and stability, providing an output torque of 15.75 Nm/MPa. Experiments show that the average error of translational motion is 0.37 mm. Real-time MRI-guided endovascular intervention is performed in a vascular phantom with pulsatile flows to validate its potential clinical use. The imaging artifact test under MRI shows no noticeable distortion and the loss of Signal-to-Noise Ratio (SNR) is less than 2%.

Keywords: Medical Robots and Systems, Dual Arm Manipulation, Mechanism Design
Abstract: In this study, the infant Cardiac Robotic Surgical System (iCROSS) is developed to assist a surgeon in performing the patent ductus arteriosus (PDA) closure and other infant cardiac surgeries. The iCROSS is a dual-arm robot allowing two surgical instruments to collaborate in a narrow space while keeping a sufficiently large workspace. Compared with the existing surgical robotic systems, the iCROSS meets the specific requirements of infant cardiac surgeries. Its feasibility has been validated through several teleoperated tasks performed in the experiment. In particular, the iCROSS is able to perform surgical ligation successfully within one minute.

Keywords: Medical Robots and Systems, Physical Human-Robot Interaction, Performance Evaluation and Benchmarking
Abstract: Urolithiasis is a common disease with increasing prevalence across all ages. A common treatment option for smaller kidney stones is flexible ureterorenoscopy (fURS), where a flexible ureteroscope (FU) is used for stone removal and to inspect the renal collecting system. The handling of the flexible ureteroscope and end effectors (EEs), however, is challenging and requires two surgeons. In this paper, we introduce a modular robotic system for endoscope manipulation, which enables solo surgery (SSU) and is adaptable to various hand-held FUs. Both the developed hardware components and the proposed workflow and its representation in software are described. We then present and discuss the results of an initial user study. Finally, we describe subsequent developmental steps towards more extensive testing by clinical staff.

Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles
Abstract: This letter reports the design, construction, and experimental validation of a novel hand-held robot for in-office laser surgery of the vocal folds. In-office endoscopic laser surgery is an emerging trend in Laryngology: It promises to deliver the same patient outcomes of traditional surgical treatment (i.e., in the operating room), at a fraction of the cost. Unfortunately, office procedures can be challenging to perform; the optical fibers used for laser delivery can only emit light forward in a line-of-sight fashion, which severely limits anatomical access. The robot we present in this letter aims to overcome these challenges. The end effector of the robot is a steerable laser fiber, created through the combination of a thin optical fiber (0.225 mm) with a tendon-actuated Nickel-Titanium notched sheath that provides bending. This device can be seamlessly used with most commercially available endoscopes, as it is sufficiently small (1.1 mm) to pass through a working channel. To control the fiber, we propose a compact actuation unit that can be mounted on top of the endoscope handle, so that, during a procedure, the operating physician can operate both the endoscope and the steerable fiber with a single hand. We report simulation and phantom experiments demonstrating that the proposed device substantially enhances surgical access compared to current clinical fibers.

Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles, Parallel Robots
Abstract: In retinal surgery clinicians access the internal volume of the eyeball through small scale trocar ports, typically 0.65 mm in diameter, to treat vitreoretinal disorders like idiopathic epiretinal membrane and age-related macular holes. The treatment of these conditions involves the removal of thin layers of diseased tissue, namely the epiretinal membrane and the internal limiting membrane. These membranes have an average thickness of only 60 μm and 2 μm respectively making extremely challenging even for expert clinicians to peel without damaging the surrounding tissue. In this work we present a novel Ophthalmic microsurgery Robot (OmSR) designed to operate a standard surgical forceps used in these procedures with micrometric precision, overcoming the limitations of current robotic systems associated with the offsetting of the remote centre of motion of the end effector when accessing the sclera. The design of the proposed system is presented, and its performance evaluated. The results show that the end effector can be controlled with an accuracy of less than 30 μm and the surgical forceps opening and closing positional error is less than 4.3 μm. Trajectory-following experiments and membrane peeling experiments are also presented, showing promising results in both scenarios.

Keywords: Mechanism Design, Actuation and Joint Mechanisms
Abstract: This paper presents a reduction mechanism for robot actuators that can switch between two types of reduction ratio. By fixing the carrier or ring gear of the proposed actuator which is based on the 3K compound planetary drive, the actuator can shift its reduction ratio. For compact design with reduced weight of the actuator, unique pawl brake mechanism interacting with cams and micro servos for switching mechanism is designed. The resulting prototype module has a reduction ratio of 6.91 and 44.93 for ‘low-reduction’ and ‘high-reduction’ ratios, respectively. Reduction ratios can be easily adjusted by modifying the pitch diameters of gears. Experimental results demonstrate that the proposed actuator could extend its operation region via two reduction modes that are interchangeable with gear shifting.

Keywords: Mechanism Design, Biologically-Inspired Robots, Motion Control
Abstract: In this paper, we propose a new brachiation robot, a single-rod brachiation robot. Brachiation is a method of locomotion that makes clever use of gravity and has been tried to apply to robots. Conventional brachiation robots are multiple-pendulum-like robots that mimic a gibbon. Although the multiple-pendulum-like robot can easily change the length of one brachiation step by joints, it has complex structures and generates aperiodic motions such as chaos. In contrast, the single-rod brachiation robot has the advantages of simple structure and the ability to suppress complex multiple-pendulum trajectories. The single-rod brachiation robot has a disadvantage because it is difficult to adjust the distance to the next bar. However, we can solve it by aerial brachiation, which includes an aerial phase before grasping the next bar. Using the actual robot, we showed that the swinging amplitude could be increased by appropriately moving its center of gravity like a trapeze motion. In addition, using this, we achieved continuous brachiation across three bars and brachiation including the aerial phase with a flight distance of 140 mm.

Keywords: Mechanism Design, Humanoid Robot Systems, Kinematics
Abstract: Building humanoid robots with properties similar to those of humans in terms of strength and agility is a great and unsolved challenge. This work introduces a compact and lightweight wrist joint mechanism that is singularity-free and has large range of motion. The mechanism has two degrees of freedom (DoF) and has been designed to be integrated into a human scale humanoid robot arm. It is based on a parallelmechanism with rolling contact joint behaviour and remote actuation that facilitates a compact design with low mass and inertia. The mechanism’s kinematics together with a solution of the inverse kinematics problem for the specific design, and the manipulability analysis are presented. The first prototype of the proposed mechanism shows the possible integration of actuation, sensing and electronics in small and narrow space. Experimental evaluations shows that the design feature unique performance regarding weight, speed, payload and accuracy.

Keywords: Mechanism Design, Medical Robots and Systems
Abstract: Abstract—In biomedical engineering, robotic implants provide new methods to restore and improve bodily function, and regenerate tissue. A significant challenge with the design of these devices is to safely actuate them for weeks or months, while they are residing in a patient’s body. Magnetic, and other force-at distance actuation methods, allow mechanisms to be controlled remotely and without contact or line of sight to the device. In this paper, we present a novel magnetic field driven wireless motor. The motor drives a robotic implant for the treatment of long gap esophageal atresia and short bowel syndrome. The motor is equipped with two oppositely oriented permanent magnets which experience forces in opposite directions when a magnetic field is applied tangential to the magnets’ directions. The implant can produce a force of 2 N. It is demonstrated with an ex vivo porcine esophagus.

Keywords: Mechanism Design, Soft Robot Materials and Design, Medical Robots and Systems
Abstract: Tendon-driven continuum robots show promise for use in surgical applications as they can assume complex configurations to navigate along tortuous paths. However, to achieve these complex robot shapes, multiple segments are required as each robot segment can bend only with a single constant curvature. To actuate these additional robot segments, multiple tendons must typically be added on-board the robot, complicating their integration, robot control, and actuation. This work presents a method of achieving two curvatures in a single tendon-driven continuum robot segment through use of a novel magnetic locking mechanism. Thus, the need for additional robot segments and actuating tendons is eliminated. The resulting two curvatures in a single segment are demonstrated in two and three dimensions. Furthermore, the maximum magnetic field required to actuate the locking mechanism for different robot bending angles is experimentally measured to be 6.1 mT. Additionally, the locking mechanism resists unintentional unlocking unless the robot assumes a 0° bending angle and a magnetic field of 18.1 mT is applied, conditions which are not typically reached during routine use of the system. Finally, addressable actuation of two locking mechanisms is achieved, demonstrating the capability of producing multiple curvatures in a single robot segment.

Keywords: Mechanism Design, Soft Robot Materials and Design, Search and Rescue Robots
Abstract: We propose a novel toroidal origami monotrack capable of smooth-skin driving and bending for closed-skin-drive robots. Monotracks are a promising solution for achieving high mobility in unstructured environments. Toroidal-drive mechanisms enable whole skin drive; however, conventional methods experience unexpected wrinkling and buckles that lead to a large resistance. In this study, we propose an origami bellows structure with multiple rollers that can maintain the skin tension and deal with the cause of large friction between the skin and the body. The origami structure design method is presented, and the bending angle range and required drive force were derived through a theoretical analysis. The validity of the effectiveness of the concept was verified through prototype testing.

Keywords: Mechanism Design, Actuation and Joint Mechanisms, Climbing Robots
Abstract: In this study, a modified rocker-bogie mechanism that improve mobility using only two actuators and a damper is proposed. Previous mechanisms use large number of motors or complex controls to overcome obstacles such as rough terrain or stairs. To compensate this, the damper-driven rocker-bogie (DDRB) is devised. The effectiveness of the damper was verified using quasi-static analysis, and the optimal parameters of prototype was determined using multibody dynamics. Finally, the performance of climbing stairs was successfully demonstrated through experiments with the prototype. The mobility performance is expected to be improved through additional active systems, and will be applied not only to stairs but also to various rough terrains in the future.

Keywords: Mechanism Design, Redundant Robots, Manipulation Planning
Abstract: This paper describes the development of a planar multi-closed-loop hyper-redundant manipulator. The proposed manipulator consists of two extendable tape springs and several inner tube rods. As the tape springs can be wound, the manipulator achieves a large workspace with a small footprint. The inner tube rods are arranged between the two tape springs, providing the manipulator with a triangular-lattice structure in which the inner tube rods and tape springs are subjected to purely axial loads. By controlling the two fixed drive components and eight mobile drive components in the manipulator, various redundant configurations can be produced. The kinematic model of this manipulator is established, and a configuration planning approach based on optimal stiffness is proposed. Simulations are conducted for three different cases, and a prototype is fabricated and tested to validate the proposed design and method.

Keywords: Machine Learning for Robot Control, Deep Learning Methods, Imitation Learning
Abstract: We demonstrate a hybrid approach to autonomous flipper control, focusing on a fusion of hard-coded and learned knowledge. The result is a sample-efficient and modifiable control structure that can be used in conjunction with a mapping/navigation stack. The backbone of the control policy is formulated as a state machine whose states define various flipper action templates and local control behaviors. It is also used as an interface that facilitates the gathering of demonstrations to train the transitions of the state machine. We propose a soft-differentiable state machine neural network that mitigates the shortcomings of its naively implemented counterpart and improves over a multi-layer perceptron baseline in the task of state-transition classification. We show that by training on several minutes of user-gathered demonstrations in simulation, our approach is capable of a zero-shot domain transfer to a wide range of obstacles on a similar real robotic platform. Our results show a considerable increase in performance over a previous competing approach in several essential criteria. A subset of this work was successfully used in the Defense Advanced Research Projects Agency (DARPA) Subterranean Challenge to alleviate the operator of manual flipper control. We autonomously traversed stairs and other obstacles, improving map coverage.

Keywords: Object Detection, Segmentation and Categorization, AI-Enabled Robotics, Space Robotics and Automation
Abstract: Exploring unknown environments, such as caves or planetary surfaces, requires a quick understanding of the surroundings. Beforehand, only aerial footage from satellites or images from previous missions might be available. The proposed ensemble based anomaly detection framework utilizes previously gained knowledge and incorporates it with insights gained during the mission. The modular system consists of different networks which are combined to determine anomalies in the current surroundings. By utilizing data from other missions, simulations or aerial photos, a precise anomaly detection can be achieved at the start of a mission. The system can further be improved by training new networks during the mission, which can be incorporated into the ensemble at runtime. This allows for synchronous execution of mission and training of models on a base station. The proposed system is tested and evaluated on an ANYmal C walking robot in different scenarios, however the approach is applicable for different kinds of mobile robots. The results show a clear improvement of ensembles compared to individual networks, while keeping a small memory footprint and low inference time on the mobile system.

Keywords: Object Detection, Segmentation and Categorization, Computer Vision for Transportation, Sensor Fusion
Abstract: The feature pyramid, which is a vital component of the convolutional neural networks, plays a significant role in several perception tasks, including object detection for autonomous driving. However, how to better fuse multi-level and multi-sensor feature pyramids is still a significant challenge, especially for object detection. This paper presents a FocusTR (Focusing on the valuable features by multiple Transformers), which is a simple yet effective architecture, to fuse feature pyramid for the single-stream 2D detector and two-stream 3D detector. Specifically, FocusTR encompasses several novel self-attention mechanisms, including the spatial-wise boxAlign attention (SB) for low-level spatial locations, context-wise affinity attention (CA) for high-level context information, and level-wise attention for the multi-level feature. To alleviate self-attention's computational complexity and slow training convergence, FocusTR introduces a low and high-level fusion (LHF) to reduce the computational parameters, and the Pre-LN to accelerate the training convergence. Comparative experiments on public benchmarks and datasets show that FocusTR achieves a higher detection accuracy than the baseline methods, especially for the small object detection. Our method shows a 2.1 higher detection accuracy on APs index of the small object on MS-COCO 2017 with ResNeXt-101 backbone, a 2.18 higher 3D detection accuracy (moderate difficulty category) for small object-pedestrian on KITTI, and a 6.85 higher RC index (Town05 Long) on CARLA urban driving simulator.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning Methods
Abstract: In this paper, we present methods for capturing 4D body shapes of swimming fish with affordable small training datasets and textureless 2D videos. Automated capture of spatiotemporal animal movements and postures is revolutionizing the study of collective animal behavior. 4D (including 3D space + time) shape data from animals like schooling fish contains a rich array of social and non-social information that can be used to shed light on the fundamental mechanisms underlying collective behavior. However, unlike the large datasets used for 4D shape reconstructions of the human body, there are no large amounts of labeled training datasets for reconstructing fish bodies in 4D, due to the difficulty of underwater data collection. We created a template mesh model using 3D scan data from a real fish, then extracted silhouettes (segmentation masks) and key-points of the fish body using Mask R-CNN and DeepLabCut, respectively. Next, using the Adam optimizer, we optimized the 3D template mesh model for each frame by minimizing the difference between the projected 3D model and the detected silhouettes as well as the key-points. Finally, using an LSTM-based smoother, we generated accurate 4D shapes of schooling fish based on the 3D shapes over each frame. Our results show that the method is effective for 4D shape reconstructions of swimming fish, with greater fidelity than other state-of-the-art algorithms.

Keywords: Object Detection, Segmentation and Categorization, Gesture, Posture and Facial Expressions, AI-Based Methods
Abstract: Highly accurate multi-person pose estimation at a high framerate is a fundamental problem in autonomous driving. Solving the problem could aid in preventing pedestrian--car accidents. The present study tackles this problem by proposing a new model composed of a feature pyramid and an original head to a general backbone. The original head is built using lightweight CNNs and directly estimates multi-person pose coordinates. This configuration avoids the complex post-processing and two-stage estimation adopted by other models and allows for a lightweight model. Our model can be trained end-to-end and performed in real-time on a resource-limited platform (low-cost edge device) during inference. Experimental results using the COCO and CrowdPose datasets showed that our model can achieve a higher framerate (approx. 20 frames/sec with NVIDIA Jetson AGX Xavier) than other state-of-the-art models while maintaining sufficient accuracy for practical use.

Keywords: Object Detection, Segmentation and Categorization, Intelligent Transportation Systems
Abstract: To navigate safely, it is essential for a robot to detect all kinds of moving objects that could possibly interfere with the own trajectory. For common object classes, like cars, regular pedestrians, and trucks, there are large scale datasets as well as corresponding machine learning techniques, which provide remarkable results in commonly available detection benchmarks. A big challenge that remains, are less frequent classes, which are not part of a dataset in a sufficient number and variation. Dynamic occupancy grids are a promising approach for detection of moving objects in point clouds since they impose only a few assumptions about the objects' appearance and shape. Typically, they use particle filters to detect motion of occupancy in the grid. Existing approaches, however, often generate false positives at long obstacles because particles move along them. Therefore we propose a highly efficient approach, which performs the classification in a more structured and conservative way by making extensive use of available free space information. As a result, much less false positives are generated while the number of false negatives remains low. Our approach can be used to complement CNN-based object detections in order to detect both, frequent and uncommon object classes reliably. By using polar data structures that match the polar measurement principle, we are able to process even large point clouds of modern LiDARs with 128 lasers efficiently.

Keywords: Object Detection, Segmentation and Categorization, Representation Learning, Deep Learning for Visual Perception
Abstract: In this paper we propose a novel point cloud generator that is able to reconstruct and generate 3D point clouds composed of semantic parts. Given a latent represen- tation of the target 3D model, the generation starts from a single point and gets expanded recursively to produce the high- resolution point cloud via a sequence of point expansion stages. During the recursive procedure of generation, we not only obtain the coarse-to-fine point clouds for the target 3D model from every expansion stage, but also unsupervisedly discover the semantic segmentation of the target model according to the hierarchical/parent-child relation between the points across expansion stages. Moreover, the expansion modules and other elements used in our recursive generator are mostly sharing weights thus making the overall framework light and efficient. Extensive experiments are conducted to show that our point cloud generator has comparable or even superior performance on both generation and reconstruction tasks in comparison to various baselines, and provides the consistent co-segmentation among instances of the same object class.

Keywords: Object Detection, Segmentation and Categorization, RGB-D Perception, Deep Learning for Visual Perception
Abstract: In an attempt to imitate the success of transformers in the field of natural language processing into computer vision tasks, vision transformers (ViTs) have recently gained attention. Performance breakthroughs have been achieved in coarse-grained tasks like classification. However, dense prediction tasks, such as detection, segmentation, and depth estimation, require additional modifications and have been tackled only in an ad-hoc manner, by replacing the convolutional neural network encoder backbone of an existing architecture with a ViT. This study proposes a fully convolutional transformer that can perform both coarse and dense prediction tasks. The proposed architecture is, to the best of our knowledge, the first architecture composed of attention layers, even in the decoder part of the network. This is because our newly proposed local--global attention (LGA) can flexibly perform both downsampling and upsampling of spatial features, which are key operations required for dense prediction. Against existing ViTs on classification tasks, our architecture shows a reasonable trade-off between performance and efficiency. In the depth estimation task, our architecture achieves performance comparable to that of state-of-the-art transformer-based methods.

Keywords: Autonomous Vehicle Navigation, Object Detection, Segmentation and Categorization, Intelligent Transportation Systems
Abstract: We propose DeepFusion, a modular multi-modal architecture to fuse lidars, cameras and radars in different combinations for 3D object detection. Specialized feature extractors take advantage of each modality and can be exchanged easily, making the approach simple and flexible. Extracted features are transformed into bird's-eye-view as a common representation for fusion. Spatial and semantic alignment is performed prior to fusing modalities in the feature space. Finally, a detection head exploits rich multi-modal features for improved 3D detection performance. Experimental results for lidar-camera, lidar-camera-radar and camera-radar fusion show the flexibility and effectiveness of our fusion approach. In the process, we study the largely unexplored task of faraway car detection up to 225 meters, showing the benefits of our lidar-camera fusion. Furthermore, we investigate the required density of lidar points for 3D object detection and illustrate implications at the example of robustness against adverse weather conditions. Moreover, ablation studies on our camera-radar fusion highlight the importance of accurate depth estimation.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning Methods, Intelligent Transportation Systems
Abstract: In recent years 3D object detection from LiDAR point clouds has made great progress thanks to the development of deep learning technologies. Although voxel or point based methods are popular in 3D object detection, they usually involve time-consuming operations such as 3D convolutions on voxels or ball query among points, making the resulting network inappropriate for time critical applications. On the other hand, 2D view-based methods feature high computing efficiency while usually obtaining inferior performance than the voxel or point based methods. In this work, we present a real-time view-based single stage 3D object detector, namely CVFNet to fulfill this task. To strengthen the cross-view feature learning under the condition of demanding efficiency, our framework extracts the features of different views and fuses them in an efficient progressive way. We first propose a novel Point-Range feature fusion module that deeply integrates point and range view features in multiple stages. Then, a special Slice Pillar is designed to well maintain the 3D geometry when transforming the obtained deep point-view features into bird's eye view. To better balance the ratio of samples, a sparse pillar detection head is presented to focus the detection on the nonempty grids. We conduct experiments on the popular KITTI and NuScenes benchmark, and state-of-the-art performances are achieved in terms of both accuracy and speed.

Keywords: Haptics and Haptic Interfaces, Dynamics, Physical Human-Robot Interaction
Abstract: Haptic interfacing typically requires high communication frequencies in order to render realistic interactions between a user and a virtual environment. In this work, we introduce reduced interface modelling (RIM) as a method to bridge the discrepancy in frequency requirements between haptic devices and virtual environment simulators. The method offers a model-based approach to approximate the environment behaviour between integration time steps, without relying on time history extrapolations of the environment state with no physical basis. Using a vehicle dynamics simulation interfacing with a haptic steering wheel, we show that the proposed method results in a drastic reduction in the computation time required for numerical integration and updates to the virtual environment compared to the common zero-order-hold method. T-tests on numerical ratings by participants in a multi-user study also confirm that the RIM yields better uncoupled stability and haptic rendering smoothness compared to a common time-history-based multi-rate sampling method.

Keywords: Haptics and Haptic Interfaces, Force and Tactile Sensing, Wearable Robotics
Abstract: This paper presents a proof-of-concept soft robotic glove that provides haptic feedback to the surgeon’s hand during interventional endoscopy procedures, specifically colonoscopy. The glove is connected to a force sensing soft robotic sleeve that is mounted onto a colonoscope. The glove consists of pneumatic actuators that inflate in proportion to the incident forces on the soft robotic sleeve. Thus, the glove is capable of alerting the surgeon of potentially dangerous forces exerted on the colon wall by the colonoscope during the navigation. The proposed glove is adaptable to a variety of hand sizes. It features modular actuators that facilitate convenient and rapid assembly and attachment before the procedure and removal afterward. The glove is calibrated to respond to incident forces on the soft robotic sleeve ranging from 0-3 N. The glove’s actuators are able to reach an internal pressure of 53 kPa and exert forces up to 20 N, thereby relaying and amplifying the force exerted by the colonoscope on the colon to the surgeon’s hand.

Keywords: Haptics and Haptic Interfaces, Soft Sensors and Actuators
Abstract: While haptics research has traditionally focused on the fingertips and hands, other locations on the body provide large areas of skin that could be utilized to relay large-area haptic sensations. Researchers have thus developed wearable devices that use distributed vibrotactile actuators and distributed pneumatic force displays, but these methods have limitations. In prior work, we presented a novel actuation technique involving stacking pneumatic pouches and evaluated the actuator output. In this work, we developed a wearable haptic device using this actuation technique and evaluated how the actuator output is perceived. We conducted a user study with 20 participants to evaluate users’ perception thresholds, ability to localize, and ability to detect differences in contact area and compare their perception using the stacked pneumatic pouch actuation to traditional single-layer pouch actuation. We also used our device with stacked pneumatic actuation in a demonstration of a haptic hug that replicates the dynamics, pressure profile, and mapping to the human back, showcasing how this actuation technique can be used to create novel haptic stimuli.

Keywords: Haptics and Haptic Interfaces, Telerobotics and Teleoperation
Abstract: In interacting with stiff environments through teleoperated systems, time delays cause a mismatch between haptic feedback and the expected feedback by the operator. This mismatch causes artefacts in the feedback, which decrease transparency, but so does filtering these artefacts. Through modelling of operator stiffness and the expected feedback force with EMG, the artifacts can be selectively filtered without loss of transparency. We developed several feedback modulation techniques to bring the feedback force closer to the expected force: 1) the average between the modelled operator force and the feedback force, 2) a low pass filter and 3) a scaling modulation. To control for overdamping, a transparency check is included. We show that the averaging approach yields significantly better contacts than unmodulated feedback. None of the modulation algorithms differ significantly from the unmodulated feedback in transparency.

Keywords: Haptics and Haptic Interfaces, Telerobotics and Teleoperation, In-Hand Manipulation
Abstract: In-hand pivoting is one of the important manipulation skills that leverage robot grippers’ extrinsic dexterity to perform repositioning tasks to compensate for environmental uncertainties and imprecise motion execution. Although many researchers have been trying to solve pivoting problems using mathematical modeling or learning-based approaches, the problems remain as open challenges. On the other hand, humans perform in-hand manipulation with remarkable precision and speed. Hence, the solution could be provided by making full use of this intrinsic human skill through dexterous teleoperation. For dexterous teleoperation to be successful, interfaces that enhance and complement haptic feedback are of great necessity. In this paper, we propose a cutaneous feedback interface that complements the somatosensory information humans rely on when performing dexterous skills. The interface is designed based on five-bar link mechanisms and provides two contact points in the index finger and thumb for cutaneous feedback. By integrating the interface with a commercially available haptic device, the system can display information such as grasping force, shear force, friction, and grasped object’s pose. Passive pivoting tasks inside a numerical simulator Isaac Sim is conducted to evaluate the effect of the proposed cutaneous feedback interface.

Keywords: Haptics and Haptic Interfaces, Telerobotics and Teleoperation, Human Performance Augmentation
Abstract: An avatar robot can be operated by multiple users, augmenting the operation of a single user. In this study, we assembled a collaborative operation system for a 7-DoF robotic arm with a gripper controlled by two users. The users share the role of controlling the robot with one user controlling the arm and the other controlling the gripper. The actions of the two users must be seamlessly coordinated to ensure smooth and precise operations. Therefore, this study aimed to investigate vibrotactile feedback to promote recognition of the actions of a partner. A pick-and-place task was considered as a basic operation. First, an intensity adjustment was developed for vibrotactile feedback to ensure linear sensitivity and its usefulness was verified. An estimation test for the reaching motion showed that the target position was estimated accurately based on velocity feedback, indicating that the participants imaged the arm motion. Subsequently, collaboration tests with and without vibrotactile velocity feedback revealed a potential effect of vibrotactile feedback in terms of reducing the time required for completing the target task. Our experimental results indicate that vibrotactile feedback for partner's motion can be applied to achieve smooth collaborative operations using an avatar robot.

Keywords: Haptics and Haptic Interfaces, Virtual Reality and Interfaces, Wearable Robotics
Abstract: Despite non-co-location, haptic stimulation at the wrist can potentially provide feedback regarding interactions at the fingertips without encumbering the user's hand. Here we investigate how two types of skin deformation at the wrist (normal and shear) relate to the perception of the mechanical properties of virtual objects. We hypothesized that a congruent mapping (i.e. when the most relevant interaction forces during a virtual interaction spatially match the haptic feedback at the wrist) would result in better perception than other mappings. We performed an experiment where haptic devices at the wrist rendered either normal or shear feedback during manipulation of virtual objects with varying stiffness, mass, or friction properties. Perception of mechanical properties was more accurate with congruent skin stimulation than noncongruent. In addition, discrimination performance and subjective reports were positively influenced by congruence. This study demonstrates that users can perceive mechanical properties via haptic feedback provided at the wrist with a consistent mapping between haptic feedback and interaction forces at the fingertips, regardless of congruence.

Keywords: Haptics and Haptic Interfaces, Wearable Robotics, Virtual Reality and Interfaces
Abstract: Relocation of haptic feedback from the fingertips to the wrist has been considered as a way to enable haptic interaction with mixed reality virtual environments while leaving the fingers free for other tasks. We present a pair of wrist-worn tactile haptic devices and a virtual environment to study how various mappings between fingers and tactors affect task performance. The haptic feedback rendered to the wrist reflects the interaction forces occurring between a virtual object and virtual avatars controlled by the index finger and thumb. We performed a user study comparing four different finger-to tactor haptic feedback mappings and one no-feedback condition as a control. We evaluated users’ ability to perform a simple pick-and-place task via the metrics of task completion time, path length of the fingers and virtual cube, and magnitudes of normal and shear forces at the fingertips. We found that multiple mappings were effective, and there was a greater impact when visual cues were limited. We discuss the limitations of our approach and describe next steps toward multi-degree-of-freedom haptic rendering for wrist-worn devices to improve task performance in virtual environments.

Keywords: Haptics and Haptic Interfaces, Prosthetics and Exoskeletons, Telerobotics and Teleoperation
Abstract: Tactile feedback is necessary for closing the sensorimotor loop in prosthetic and tele-operable control, which would allow for more precise manipulation and increased acceptance of use of such devices. Pressure stimuli are commonly presented to users in haptic devices through a sensory substitution to vibration. The precise nature of this substitution affects pressure sensitivity, as well as the comfort and intuitiveness of the device for the user. This study determines the effects of different vibrational encodings for pressure on user-preference and performance in a 4-alternative absolute identification task. 4 different encoding patterns for pressure were examined: short pulse and long pulse amplitude modulation along with sine and square wave frequency modulation. Of the methods examined, frequency modulation methods led to the best discrimination of stimuli (p < 0.001). There was a notable difference in user preference between the two frequency modulated systems, with sinusoidal stimulation being highest ranked across all the preference metrics and square-wave being ranked lowest in two of the three. This difference trended towards, but did not achieve statistical significance (TLX rankings, p = 0.098). This suggests that prostheses or teleoperated devices utilising vibrotactile feedback may benefit from implementing a discrete frequency-based sinusoidal pattern to indicate changes in grip force

Keywords: Human Factors and Human-in-the-Loop, Physical Human-Robot Interaction, Human-Centered Robotics
Abstract: Existing work in physical robot caregiving is limited in its ability to provide long-term assistance. This is due to (i) lack of well-defined problems, (ii) diversity of tasks, and (iii) limited access to stakeholders from the caregiving community. We propose Structuring Physically Assistive Robotics for Caregiving with Stakeholders-in-the-loop (SPARCS) to address these challenges. SPARCS is a framework for physical robot caregiving comprising (i) Building Blocks that define physical robot caregiving scenarios, (ii) Structured Workflows - hierarchical workflows that enable us to answer the Whats and Hows of physical robot caregiving, and (iii) SPARCS-box, a web-based platform to facilitate dialogue between all stakeholders. We collect clinical data for six care recipients with varying disabilities and demonstrate the use of SPARCS in designing well-defined caregiving scenarios and identifying their care requirements. All the data and workflows from this study are available on SPARCS-box. We demonstrate the utility of SPARCS in building a robot-assisted feeding system for one of the care recipients. We also perform experiments to show the adaptability of this system to different caregiving scenarios. Finally, we identify open challenges in physical robot caregiving by consulting care recipients and caregivers. Supplementary material can be found at emprise.cs.cornell.edu/sparcs/.

Keywords: Human Factors and Human-in-the-Loop, Manipulation Planning, Motion and Path Planning
Abstract: We present a predictive system for non-prehensile, physics-based motion planning in clutter with a human-in-the-loop. Recent shared-autonomous systems present motion planning performance improvements when high-level reasoning is provided by a human. Humans are usually good at quickly identifying high-level actions in high-dimensional spaces, and robots are good at converting high-level actions into valid robot trajectories. In this paper, we present a novel framework that permits a single human operator to effectively guide a fleet of robots in a virtual warehouse. The robots are tackling the problem of Reaching Through Clutter (RTC), where they are reaching onto cluttered shelves to grasp a goal object while pushing other obstacles out of the way. We exploit information from the motion planning algorithm to predict which robot requires human help the most and assign that robot to the human. With twenty virtual robots and a single human-operator, the results suggest that this approach improves the system’s overall performance compared to a baseline with no predictions. The results also show that there is a cap on how many robots can effectively be guided simultaneously by a single human operator.

Keywords: Robot Companions, Gesture, Posture and Facial Expressions, Design and Human Factors
Abstract: People and robots may need to cross each other in narrow spaces when they are sharing environments. It is expected that autonomous robots will behave in these contexts safely but also show social behaviors. Thereby, developing an acceptable behavior for autonomous robots in the area mentioned above is a foreseeable problem for the Human-Robot Interaction (HRI) field. Our current work focuses on integrating legible non-verbal behaviors into the robot's social navigation to make nearby humans aware of its intended trajectory. Results from a within-subjects study involving 33 participants show that deictic gestures as navigational cues for humanoid robots result in fewer navigation conflicts than the use of a simulated gaze. Additionally, an increase in the perceived anthropomorphism is found when the robot uses the deictic gesture as a cue. These findings show the importance of social behaviors for people avoidance and suggest a paradigm of such behaviors in future humanoid robotic applications.

Keywords: Human Factors and Human-in-the-Loop, Human-Centered Robotics, Human-Robot Collaboration
Abstract: In hybrid industrial environments, workers' comfort and positive perception of safety are essential requirements for successful acceptance and usage of collaborative robots. This paper proposes a novel human-robot interaction framework in which the robot behaviour is adapted online according to the operator's cognitive workload and stress. The method exploits the generation of B-spline trajectories in the joint space and formulation of a multi-objective optimisation problem to online adjust the total execution time and smoothness of the robot trajectories. The former ensures human efficiency and productivity of the workplace, while the latter contributes to safeguarding the user's comfort and cognitive ergonomics. The performance of the proposed framework was evaluated in a typical industrial task. Results demonstrated its capability to enhance the productivity of the human-robot dyad while mitigating the cognitive workload induced in the worker.

Keywords: Physical Human-Robot Interaction, Compliance and Impedance Control, Medical Robots and Systems
Abstract: Ultrasound (US) imaging is a common but physically demanding task in the medical field, and sonographers may need to put in considerable physical effort for producing high-quality US images. During physical human-robot interaction on US imaging, robot compliance is a critical feature that can ensure human user safety while automatic force regulation ability can help to improve task performance. However, higher robot compliance may mean lower force regulation accuracy, and vice versa. Especially, the contact/non-contact status transition can largely affect the control system stability. In this paper, a novel electromyography (EMG)-based hybrid impedance-force control system is developed for US imaging task. The proposed control system incorporates the robot compliance and force regulation ability via a hybrid controller while the EMG channel enables the user to online modulate the trade-off between the two features as necessary. Two experiments are conducted to examine the hybrid controller and show the necessity of involving an EMG-based modulator. A proof-of-concept study on US imaging is performed with implementing the proposed EMG-based control system, and the effectiveness is demonstrated. The proposed control system is promising to ensure robot's stability and patient's safety, thus obtain high-quality US images, while monitoring and reducing sonographer's fatigue. Furthermore, it can be easily adapted to other physically demanding tasks in the field of medicine.

Keywords: Human Factors and Human-in-the-Loop, Physical Human-Robot Interaction, Failure Detection and Recovery
Abstract: In human-robot collaboration, robot errors are inevitable---damaging user trust, willingness to work together, and task performance. Prior work has shown that people naturally respond to robot errors socially and that in social interactions it is possible to use human responses to detect errors. However, there is little exploration in the domain of non-social, physical human-robot collaboration such as assembly and tool retrieval. In this work, we investigate how people's organic, social responses to robot errors may be used to enable timely automatic detection of errors in physical human-robot interactions. We conducted a data collection study to obtain facial responses to train a real-time detection algorithm and a case study to explore the generalizability of our method with different task settings and errors. Our results show that natural social responses are effective signals for timely detection and localization of robot errors even in non-social contexts and that our method is robust across a variety of task contexts, robot errors, and user responses. This work contributes to robust error detection without detailed task specifications.

Keywords: Human Factors and Human-in-the-Loop, Virtual Reality and Interfaces, Telerobotics and Teleoperation
Abstract: Self-body awareness refers to the recognition of one's body as one's own and consists of two senses: "sense of body ownership" and "sense of agency." In telexistence/telepresence robot operation, time delays in the robot's motion degrade self-body awareness of the robot body. We investigated how self-body recognition can be affected in a telexistence robot operation in a VR space when the robot is presented with a real robot arm that simulates a real robot with a delay and a virtual robot arm, or several virtual robot arms, with a delay less than that of the real robot. These experimental conditions include a 'Predictive Display,' which is well known as a time delay countermeasure. The results suggest that virtual robot arms presented consecutively with less delay than a real robot arm do not induce a sense of body ownership to the real robot arm, but they enhance the sense of agency to the real robot arm, and that sense of agency is stronger when the task requires precision.

Keywords: Human Factors and Human-in-the-Loop, Multi-Modal Perception for HRI, Reinforcement Learning
Abstract: Interactive Reinforcement Learning (IRL) has shown promising results in decreasing the learning times of Reinforcement Learning algorithms by incorporating human feedback and advice. In particular, the integration of multimodal feedback channels such as speech and gestures into IRL systems can enable more versatile and natural interaction of everyday users. In this paper, we propose a novel approach to integrate human advice from multiple modalities into IRL algorithms. For each advice modality we assume an individual base classifier that outputs a categorical probability distribution and fuse these distributions using the Bayesian fusion method Independent Opinion Pool. While existing approaches rely on heuristic fusion, our Bayesian approach is theoretically founded and fully exploits the uncertainty represented in the distributions. Experimental evaluations in a simulated grid world scenario and on a real-world human-robot interaction task with a 7-DoF robot arm show that our method clearly outperforms the closest related approach for multimodal IRL. In particular, our novel approach is more robust against misclassifications of the modalities' individual base classifiers.

Keywords: Human Factors and Human-in-the-Loop, Computer Vision for Transportation, Deep Learning for Visual Perception
Abstract: The visual attention of navigators is imperative to understanding the logic of navigation and the surveillance of the navigators' status and operations. Currently, existing studies are implemented under the help of wearable eye-tracker glasses, while the high expenditure demanded by the equipment and service and the limitations on usability have impeded the relevant research to be performed extensively. In this letter, the authors propose a framework which is the first attempt in the maritime domain to provide a camera-based deep-learning (CaBDeeL) visual attention recognition solution to outperform the intrusive eye tracker in terms of the shortcomings. A wide-angle camera is configured in front of the navigator in the advanced ship-bridge simulator so that visual attention reflected by the facial and head movements is captured at the front view. A pair of eye-tracker glasses are used to classify the captured visual attention images to establish the primary database. While the camera-captured images are classified, a convolutional neural network (CNN) is built as an automatic classifier. The CNN is applied to two scenarios, and it scores an overall 95 % precision.

Keywords: Deep Learning for Visual Perception, Visual Learning, Deep Learning Methods
Abstract: Recovering depth information from a single image is a long-standing challenge, and self-supervised depth estimation methods have gradually attracted attention due to not relying on high-cost ground truth. Constructing an accurate photometric loss based on photometric consistency is crucial for these self-supervised methods to obtain high-quality depth maps. However, the photometric loss in most studies treats all pixels indiscriminately, resulting in poor performance. In this paper, we propose two modules based on the spatial and temporal cues to refine the photometric loss. Delving into the geometric model of photometric consistency, we introduce a depth-aware pixel correspondence module (DPC) inside the monocular depth estimation pipeline. It reduces the uncertainty of photometric errors by applying the homography matrix to the projection of corresponding pixels in far regions instead of the fundamental matrix. Furthermore, we design an omnidirectional auto-masking module (OA) to boost the robustness of our model, which utilizes temporal sequences to generate disturbance poses and hypothetical views to distinguish dynamic objects with different directions that violate the photometric consistency. Experiments on the KITTI and the Make3d datasets reveal that our framework achieves state-of-the-art performance.

Keywords: Deep Learning for Visual Perception, Visual Learning
Abstract: Depth estimation and semantic edge detection are two key tasks in computer vision, which have made great progress. To date, how to associatively predict the depth and the semantic edge is rarely explored. In this work, we first propose a flexible two-branch framework that can make the two tasks take advantage of each other, achieving a win-win situation. Specifically, for the semantic edge detection branch, an Enhanced Edge Weighting strategy (EEW) is designed, which learns weight information from the by-product of depth branch, depth edge, to enhance edge perception in features. Meanwhile, we make depth estimation benefit from semantic edge detection through introducing Depth Edge Semantic Classification module (DESC). Furthermore, a double reconstruction (D-reconstruction) approach is presented, together with semantic edge-guided disparity smoothing loss to mitigate the ambiguities of the self-supervised manner for depth estimation. Experiments on the Cityscapes dataset demonstrate that our framework outperforms the state-of-the-art method in depth estimation along with a significant improvement in semantic edge detection.

Keywords: Deep Learning for Visual Perception, Visual Learning, Perception for Grasping and Manipulation
Abstract: Current RGB-based 6D object pose estimation methods have achieved noticeable performance on datasets and real world applications. However, predicting 6D pose from single 2D image features is susceptible to disturbance from changing of environment and textureless or resemblant object surfaces. Hence, RGB-based methods generally achieve less competitive results than RGBD-based methods, which deploy both image features and 3D structure features. To narrow down this performance gap, this paper proposes a framework for 6D object pose estimation that learns implicit 3D information from 2 RGB images. Combining the learned 3D information and 2D image features, we establish more stable correspondence between the scene and the object models. To seek for the methods best utilizing 3D information from RGB inputs, we conduct an investigation on three different approaches, including Early- Fusion, Mid-Fusion, and Late-Fusion. We ascertain the Mid- Fusion approach is the best approach to restore the most precise 3D keypoints useful for object pose estimation. The experiments show that our method outperforms state-of-the-art RGB-based methods, and achieves comparable results with RGBD-based methods.

Keywords: Deep Learning for Visual Perception, Visual Learning, Computer Vision for Transportation
Abstract: Recently, self-supervised learning of depth and ego-motion from thermal images shows strong robustness and reliability under challenging lighting and weather conditions. However, the inherent thermal image properties such as weak contrast, blurry edges, and noise hinder to generate effective self-supervision from thermal images. Therefore, most previous researches just rely on additional self-supervisory sources such as RGB video, generative models, and Lidar information. In this paper, we conduct an in-depth analysis of thermal image characteristics that degenerates self-supervision from thermal images. Based on the analysis, we propose an effective thermal image mapping method that significantly increases image information, such as overall structure, contrast, and details, while preserving temporal consistency. By resolving the fundamental problem of the thermal image, our depth and pose network trained only with thermal images achieves state-of-the-art results without utilizing any extra self-supervisory source. As our best knowledge, this work is the first self-supervised learning approach to train monocular depth and relative pose networks with only thermal images.

Keywords: Visual Learning, Object Detection, Segmentation and Categorization, Data Sets for Robotic Vision
Abstract: The problem of image change detection via everyday indoor robot navigation is explored from a novel perspective of the self-attention technique. Detecting semantically non-distinctive and visually small changes remains a key challenge in the robotics community. Intuitively, these small non-distinctive changes may be better handled by the recent paradigm of the attention mechanism, which is the basic idea of this work. However, existing self-attention models require significant retraining cost per domain, so it is not directly applicable to robotics applications. We propose a new self-attention technique with an ability of unsupervised on-the-fly domain adaptation, which introduces an attention mask into the intermediate layer of an image change detection model, without modifying the input and output layers of the model. Experiments, in which an indoor robot aims to detect visually small changes in everyday navigation, demonstrate that our attention technique significantly boosts the state-of-the-art image change detection model. We will make this new dataset publicly available for future research in this area.

Keywords: Visual Learning, Machine Learning for Robot Control, Collision Avoidance
Abstract: In this paper, we introduce a new concept of incorporating factorized flow maps as mid-level representations, for bridging the perception and the control modules in modular learning based robotic frameworks. To investigate the advantages of factorized flow maps and examine their interplay with the other types of mid-level representations, we further develop a configurable framework, along with four different environments that contain both static and dynamic objects, for analyzing the impacts of factorized optical flow maps on the performance of deep reinforcement learning agents. Based on this framework, we report our experimental results on various scenarios, and offer a set of analyses to justify our hypothesis. Finally, we validate flow factorization in real world scenarios.

Keywords: Visual Learning, Representation Learning, Task Planning
Abstract: Visual action planning particularly excels in applications where the state of the system cannot be computed explicitly, such as manipulation of deformable objects, as it enables planning directly from raw images. Even though the field has been significantly accelerated by deep learning techniques, a crucial requirement for their success is the availability of a large amount of data. In this work, we propose the Augment-Connect-Explore (ACE) paradigm to enable visual action planning in cases of data scarcity. We build upon the Latent Space Roadmap (LSR) framework which performs planning with a graph built in a low dimensional latent space. In particular, ACE is used to i) Augment the available training dataset by autonomously creating new pairs of datapoints, ii) create new unobserved Connections among representations of states in the latent graph, and iii) Explore new regions of the latent space in a targeted manner. We validate the proposed approach on both simulated box stacking and real-world folding task showing the applicability for rigid and deformable object manipulation tasks, respectively.

Keywords: Visual Learning, Deep Learning in Grasping and Manipulation
Abstract: Currently, task-oriented grasp detection approaches are mostly based on pixel-level affordance detection and semantic segmentation. These pixel-level approaches heavily rely on the accuracy of 2D affordance mask, and the generated grasp candidates are restricted to a small workspace. To mitigate these limitations, we proposed a novel 6-DoF task-oriented grasp detection framework, which takes the observed object point cloud as input and predicts diverse 6-DoF grasp poses for different tasks. Specifically, our implicit estimation network and visual affordance network in this framework could directly predict coarse grasp candidates, and corresponding 3D affordance heatmap for each potential task, respectively. Furthermore, the grasping scores from coarse grasps are combined with heatmap values to generate more accurate and finer candidates. Our proposed framework shows significant improvements compared to baselines for existing and novel objects on our self-constructed simulation dataset. Although our framework is trained based on the simulated objects and environment, the final generated grasp candidates can be accurately and stably executed in the real robot experiments when the object is randomly placed on a support surface.

Keywords: Visual Learning, RGB-D Perception, Deep Learning for Visual Perception
Abstract: Articulated objects pose a unique challenge for robotic perception and manipulation. Their increased number of degrees-of-freedom makes tasks such as localization computationally difficult, while also making the process of realworld dataset collection unscalable. With the aim of addressing these scalability issues, we propose Neural Articulated Radiance Fields (NARF22), a pipeline which uses a fully-differentiable, configuration-parameterized Neural Radiance Field (NeRF) as a means of providing high quality renderings of articulated objects. NARF22 requires no explicit knowledge of the object structure at inference time. We propose a two-stage parts-based training mechanism which allows the object rendering models to generalize well across the configuration space even if the underlying training data has as few as one configuration represented. We demonstrate the efficacy of NARF22 by training configurable renderers on a real-world articulated tool dataset collected via a Fetch mobile manipulation robot. We show the applicability of the model to gradient-based inference methods through a configuration estimation and 6 degree-of-freedom pose refinement task.

Keywords: Localization, Multi-Robot Systems, Range Sensing
Abstract: Inter-agent relative localization is critical for any multi-robot system operating in the absence of external positioning infrastructure or prior environmental knowledge. We propose a novel inter-agent relative 2D pose estimation system where each participating agent is equipped with several ultra-wideband (UWB) ranging tags. Prior work typically supplements noisy UWB range measurements with additional continuously transmitted data, such as odometry, making these approaches scale poorly with increased swarm size or decreased communication throughput. This approach addresses these concerns by using only locally collected UWB measurements with no additionally transmitted data. By modeling observed ranging biases and systematic antenna obstructions in our proposed optimization solution, our experimental results demonstrate an improved mean position error (while remaining competitive in other metrics) over a similar state-of-the-art approach that additionally relies on continuously transmitted odometry.

Keywords: Localization, Range Sensing, Cooperating Robots
Abstract: Ranging information from ultra-wideband (UWB) ranging radios can be used to improve estimated navigation accuracy of a ground robot with other on-board sensors. However, all ranging-aided navigation methods demand the locations of ranging nodes to be known, which is not suitable for time-pressed situations, dynamic cluttered environments, or collaborative navigation applications. This paper describes a new ranging-aided navigation approach that does not require the locations of ranging radios. Our approach formulates relative pose constraints using ranging readings. The formulation is based on geometric relationships between each stationary ranging node and two ranging antennas on the moving robot across time. Our experiments show that estimated navigation accuracy of the ground robot is substantially enhanced with ranging information using our approach under a variety of scenarios, when ranging nodes are placed at unknown locations. We analyze and compare our performance with a traditional ranging-aided method, which requires mapping the positions of ranging nodes. We also demonstrate the applicability of our approach for collaborative navigation in large-scale unknown environments, by using ranging information from one mobile robot to improve navigation estimation of the other robot. This application does not require the installation of ranging nodes at fixed locations.

Keywords: Mapping, Localization, SLAM
Abstract: This paper proposes an efficient and probabilistic adaptive voxel mapping method for LiDAR odometry. The map is a collection of voxels; each contains one plane (or edge) feature that enables the probabilistic representation of the environment and accurate registration of a new LiDAR scan. We further analyze the need for coarse-to-fine voxel mapping and then use a novel voxel map organized by a Hash table and octrees to build and update the map efficiently. We apply the proposed voxel map to an iterated extended Kalman filter and construct a maximum a posteriori probability problem for pose estimation. Experiments on the open KITTI dataset show the high accuracy and efficiency of our method compared to other state-of-the-art methods. Outdoor experiments on unstructured environments with non-repetitive scanning LiDARs further verify the adaptability of our mapping method to different environments and LiDAR scanning patterns (see our attached video). Our codes and dataset are open-sourced on Github。

Keywords: Mapping, Omnidirectional Vision, SLAM
Abstract: As an abstract representation of the environment structure, a topological map has advantageous properties for path-planning and navigation. Here we proposed an online topological mapping method, 360ST-Mapping, using omnidirectional vision. The 360-degree field-of-view allows the agent to obtain consistent observation and incrementally extract topological environment information. Moreover, we leverage semantic information to guide topological place recognition, further improving performance. The topological map possessing semantic information has the potential to support semantics-related advanced tasks. After integrating the topological mapping module into the omnidirectional visual SLAM system, we conducted extensive experiments in several large-scale indoor scenes and validated the method’s effectiveness.

Keywords: Mapping
Abstract: Gaussian process (GP) implicit surface models provide environment and object representations which elegantly address noise and uncertainty while remaining sufficiently flexible to capture complex geometry. However, GP models quickly become intractable as the size of the observation set grows - a trait which is difficult to reconcile with the rate at which modern range sensors produce data. Furthermore, naive applications of GPs to implicit surface models allocate model resources uniformly, thus using precious resources to capture simple geometry. In contrast to prior work addressing these challenges though model sparsification, spatial partitioning, or ad-hoc filtering we propose introducing model bias online through the GP's mean function. We achieve more accurate distance fields using smaller models by creating a distance field prior from features which are easy to extract and have analytic distance fields. In particular, we demonstrate this approach using linear features. We show the proposed distance field halves model size in a 2D mapping task using data from a SICK S300 sensor. When applied to a single 3D scene from the TUM RGB-D SLAM dataset we achieve a fivefold reduction in model size. Our proposed prior results in more accurate GP implicit surfaces, while allowing existing models to function in larger environments or with larger spatial partitions due to reduced model size.

Keywords: Mapping, SLAM
Abstract: Implicit representations are widely used for object reconstruction due to their efficiency and flexibility. In 2021, a novel structure named neural implicit map has been invented for incremental reconstruction. A neural implicit map alleviates the problem of inefficient memory cost of previous online 3D dense reconstruction while producing better quality. However, the neural implicit map suffers the limitation that it does not support remapping as the frames of scans are encoded into a deep prior after generating the neural implicit map. This means, that neither this generation process is invertible, nor a deep prior is transformable. The non-remappable property makes it not possible to apply loop-closure techniques. We present a neural implicit map based transformation algorithm to fill this gap. As our neural implicit map is transformable, our model supports remapping for this special map of latent features. Experiments show that our remapping module is capable to well-transform neural implicit maps to new poses. Embedded into a SLAM framework, our mapping model is able to tackle the remapping of loop closures and demonstrates high-quality surface reconstruction. Our implementation is available at githubfootnote{url{https://github.com/Jarrome/IMT_Mapping}} for the research community.

Keywords: Mapping, SLAM, Range Sensing
Abstract: The ability to process environment maps across multiple sessions is critical for robots operating over extended periods of time. Specifically, it is desirable for autonomous agents to detect changes amongst maps of different sessions so as to gain a conflict-free understanding of the current environment. In this paper, we look into the problem of change detection based on a novel map representation, dubbed Plane Signed Distance Fields (PlaneSDF), where dense maps are represented as a collection of planes and their associated geometric components in SDF volumes. Given point clouds of the source and target scenes, we propose a three-step PlaneSDF-based change detection approach: (1) PlaneSDF volumes are instantiated within each scene and registered across scenes using plane poses; 2D height maps and object maps are extracted per volume via height projection and connected component analysis. (2) Height maps are compared and intersected with the object map to produce a 2D change location mask for changed object candidates in the source scene. (3) 3D geometric validation is performed using SDF-derived features per object candidate for change mask refinement. We evaluate our approach on both synthetic and real-world datasets and demonstrate its effectiveness via the task of changed object detection. Supplementary video is available at: https://youtu.be/oh-MQPWTwZI

Keywords: Mapping, Localization, SLAM
Abstract: Long-time operations of autonomous vehicles and mobile robots in logistics and service applications are still a challenge. To avoid a continuous re-mapping, the map can be updated to obtain a consistent representation of the current environment. In this paper, we propose a novel LIDAR-based occupancy grid map updating algorithm for dynamic environments taking into account possible localisation and measurement errors. The proposed approach allows robust long-term operations as it can detect changes in the working area even in presence of moving elements. Results highlighting maps quality and localisation performance, both in simulation and experiments, are reported. We will extend localisation performance analysis to real-world experiments employing an external tracking system.

Keywords: Localization, Aerial Systems: Applications, Visual Tracking
Abstract: In this paper, we propose a ground-based monocular UAV localisation system that detects and localises an LED marker attached to the underside of a UAV. Our system removes the need for extensive infrastructure and calibration unlike existing technologies such as UWB, radio frequency and multi-camera systems often used for localisation in GPS-denied environment. To improve deployablity for real-world applications without the need to collect extensive real dataset, we train a CNN on synthetic binary images as opposed to using real images in existing monocular UAV localisation methods, and factor in the camera’s zoom to allow tracking of UAVs flying at further distances. We propose NoisyCutout algorithm for augmenting synthetic binary images to simulate binary images processed from real images and show that it improves localisation accuracy as compared to using existing salt-and-pepper and Cutout augmentation methods. We also leverage uncertainty propagation to modify the CNN’s loss function and show that this also improves localisation accuracy. Real-world experiments are conducted to evaluate our methods and we achieve an overall 3D RMSE of approximately 0.41m.

Keywords: Human-Centered Robotics, Reinforcement Learning, Human-Robot Collaboration
Abstract: Intelligent robots designed to interact with humans in the real world need to adapt to the preferences of different individuals. Preference-based reinforcement learning (RL) has shown great potential for teaching robots to learn personalized behaviors from interacting with humans without a meticulous, hand-crafted reward function, replaced by learning reward based on a human's preferences between two robot trajectories. However, poor feedback efficiency and poor exploration in the state and reward spaces make current preference-based RL algorithms perform poorly in complex interactive tasks. To improve the performance of preference-based RL, we incorporate prior knowledge of the task into preference-based RL. Specifically, we decouple the task from preference in human-robot interaction. We utilize a sketchy task reward derived from task priori to instruct robots to conduct more effective task exploration. Then a learned reward from preference-based RL is used to optimize the robot's policy to align with human preferences. In addition, these two parts are combined organically via reward shaping. The experimental results show that our method is a practical and effective solution for personalized human-robot interaction. Code is available at https://github.com/Wenminggong/PbRL_for_PHRI.

Keywords: Human-Centered Robotics, Virtual Reality and Interfaces, Human-Robot Collaboration
Abstract: Although there is extensive research regarding legged manipulators, comparatively little focuses on their User Interfaces (UIs). Towards extending the state-of-art in this domain, in this work, we integrate a Boston Dynamics (BD) Spot with a light-weight 7 DoF Kinova robot arm and a Robotiq 2F-85 gripper into a legged manipulator. Furthermore, we jointly control the robotic platform using an affordance-aware Augmented Reality (AR) Head-Mounted Display (HMD) UI developed for the Microsoft HoloLens 2. We named the combined platform Holo-SpoK. Moreover, we explain how this manipulator colocalises with the HoloLens 2 for its control through AR. In addition, we present the details of our algorithms for autonomously detecting grasp-ability affordances and for the refinement of the positions obtained via vision-based colocalisation. We validate the suitability of our proposed methods with multiple navigation and manipulation experiments. To the best of our knowledge, this is the first demonstration of an AR HMD UI for controlling legged manipulators.

Keywords: Human-Centered Robotics, Reinforcement Learning, Human-Robot Collaboration
Abstract: Interactive Reinforcement Learning (IntRL) allows human teachers to accelerate the learning process of Reinforcement Learning (RL) robots. However, IntRL has largely been limited to tasks with discrete-action spaces in which actions are relatively slow. This limits IntRL’s application to more complicated and challenging robotic tasks, the very tasks that modern RL is particularly well-suited for. We seek to bridge this gap by presenting Continuous Action-space Interactive Reinforcement learning (CAIR): the first continuous action-space IntRL algorithm that is capable of using teacher feedback to out-perform state-of-the-art RL algorithms in those tasks. CAIR combines policies learned from the environment and the teacher into a single policy that proportionally weights the two policies based on their agreement. This allows a CAIR agent to learn a relatively stable policy despite potentially noisy or coarse teacher feedback. We validate our approach in two simulated robotics tasks with easy-to-design and -understand heuristic oracle teachers. Furthermore, we validate our approach in a human subjects study through Amazon Mechanical Turk and show CAIR out-performs the prior state-of-the-art in Interactive RL.

Keywords: Human-Centered Robotics, Physical Human-Robot Interaction
Abstract: To communicate, the sim1.5 million Americans living with deafblindess use tactile American Sign Language (t-ASL). To provide DeafBlind (DB) individuals with a means of using their primary communication language without the use of an interpreter, we developed an assistive technology that promotes their autonomy. The TATUM (Tactile ASL Translational User Mechanism) anthropomorphic arm hand system leverages previous developments of a fingerspelling hand to sign more complex ASL words and phrases. The TATUM hand-wrist system is attached onto a 4 DOF robot arm and a human motion recognition and human to robot gesture transfer framework is used for signing recognition and replication. In particular, signing trajectories based on vision-based motion capture data from a sign demonstrator were used to control the robot's actuators. The performance of the system was evaluated through tactile based sign recognition performed by a blinded user and for its accuracy with novice, sighted users.

Keywords: Human-Centered Robotics, Gesture, Posture and Facial Expressions, Aerial Systems: Applications
Abstract: Unmanned aerial vehicles (UAVs) are becoming more common, presenting the need for effective human-robot communication strategies that address the unique nature of unmanned aerial flight. Visual communication via drone flight paths, also called gestures, may prove to be an ideal method. However, the effectiveness of visual communication techniques is dependent on several factors including an observer's position relative to a UAV. Previous work has studied the maximum line-of-sight at which observers can identify a small UAV. However, this work did not consider how changes in distance may affect an observer's ability to perceive the shape of a UAV's motion. In this study, we conduct a series of online surveys to evaluate how changes in line-of-sight distance and gesture size affect observers' ability to identify and distinguish between UAV gestures. We first examine observers' ability to accurately identify gestures when adjusting a gesture's size relative to the size of a UAV. We then measure how observers' ability to identify gestures changes with respect to varying line-of-sight distances. Lastly, we consider how altering the size of a UAV gesture may improve an observer's ability to identify drone gestures from varying distances. Our results show that increasing the gesture size across varying UAV to gesture ratios did not have a significant effect on participant response accuracy. We found that between 17 m and 75 m from the observer, their ability to accurately identify a drone gesture was inversely proportional to the distance between the observer and the drone. Finally, we found that maintaining a gesture's apparent size improves participant response accuracy over changing line-of-sight distances.

Keywords: Human-Centered Robotics, Long term Interaction, Modeling and Simulating Humans
Abstract: Human trajectory prediction is a crucial yet challenging problem, which is of fundamental importance to robotics and autonomous driving vehicles. The core challenge lies in effectively modelling the socially aware spatial interaction and complex temporal dependencies among crowds. However, previous methods either model spatial and temporal information separately or only use individual features without exploring the intra-structure of the crowds. We propose a novel trajectory prediction framework termed GA-STT, a group aware spatial-temporal transformer network to address these issues. Specifically, we first get the individual representations supervised by group-based annotations. Then, we model the complex spatial-temporal interactions with spatial and temporal transformers separately and fuse spatial-temporal embedding through the cross-attention mechanism. Results on the publicly available ETH/UCY datasets show that our model outperforms the state-of-the-art method by 19.4% in ADE and 16.9% in FDE and successfully predicts complex spatial-temporal interactions.

Keywords: Human-Centered Robotics, Reinforcement Learning, Acceptability and Trust
Abstract: Explainable artificial intelligence is a research field that tries to provide more transparency for autonomous intelligent systems. Explainability has been used, particularly in reinforcement learning and robotic scenarios, to better understand the robot decision-making process. Previous work, however, has been widely focused on providing technical explanations that can be better understood by AI practitioners than non-expert end-users. In this work, we make use of human-like explanations built from the probability of success to complete the goal that an autonomous robot shows after performing an action. These explanations are intended to be understood by people who have no or very little experience with artificial intelligence methods. This paper presents a user trial to study whether these explanations that focus on the probability an action has of succeeding in its goal constitute a suitable explanation for non-expert end-users. The results obtained show that non-expert participants rate robot explanations that focus on the probability of success higher and with less variance than technical explanations generated from Q-values, and also favor counterfactual explanations over standalone explanations.

Keywords: Human-Centered Robotics, Acceptability and Trust
Abstract: Autonomous service robots have great potential to support humans in tasks they cannot perform due to, amongst others, time constraints, work overload, or staff shortages. An important step for such service robots to be trusted or accepted by society is the provision of transparency. Its purpose is not only to communicate what a robot is doing according to the human interaction partners' needs, it should also regard social and legal requirements. A black box recorder (inspired by flight recorders) increases the system's transparency by facilitating the investigation of the cause of an incident, clarifying responsibilities, or improving the user's understanding about the robot. In this work we propose the needed requirements of such a black box recorder for increased transparency of autonomous service robots, based on the related work. Further, we present a new modular and portable black box recorder design meeting these requirements. The applicability of the system is evaluated based on real-world robot data, using the realized open-source reference implementation.

Keywords: Localization
Abstract: Accurate and robust localization is an essential task for autonomous driving systems. In this paper, we propose a novel 3D LiDAR-aided visual-inertial localization method. Our method fully explores the complementarity of visual and LiDAR observations. On the one hand, the association between semantic features in images and a given semantic map provides constraints for the absolute pose. On the other hand, LiDAR odometry (LO) can provide an accurate and robust 6DOF relative pose. The Error State Kalman Filter (ESKF) framework is exploited to estimate the vehicle pose relative to the semantic map, which fuses the global constraints between the image and the semantic map, the relative pose from the LO, and the raw IMU data. The method achieves centimeter-level localization accuracy in a variety of challenging scenarios. We validate the robustness and accuracy of our method in real-world scenes over 50 km. The experimental results show that the proposed method is able to achieve an average lateral accuracy of 0.059 m and longitudinal accuracy of 0.158 m, which demonstrates the practicality of the proposed system in autonomous driving applications.

Keywords: Localization
Abstract: Robust localization in a given map is a crucial component of most autonomous robots. In this paper, we address the problem of localizing in an indoor environment that changes and where prominent structures have no correspondence in the map built at a different point in time. To overcome the discrepancy between the map and the observed environment caused by such changes, we exploit human-readable localization cues to assist localization. These cues are readily available in most facilities and can be detected using RGB camera images by utilizing text spotting. We integrate these cues into a Monte Carlo localization framework using a particle filter that operates on 2D LiDAR scans and camera data. By this, we provide a robust localization solution for environments with structural changes and dynamics by humans walking. We evaluate our localization framework on multiple challenging indoor scenarios in an office environment. The experiments suggest that our approach is robust to structural changes and can run on an onboard computer. We release an open source implementation of our approach (upon paper acceptance), which uses off-the-shelf text spotting, written in C++ with a ROS wrapper.Robust localization in a given map is a crucial component of most autonomous robots. In this paper, we address the problem of localizing in an indoor environment that changes and where prominent structures have no correspondence in the map built at a different point in time. To overcome the discrepancy between the map and the observed environment caused by such changes, we exploit human-readable localization cues to assist localization. These cues are readily available in most facilities and can be detected using RGB camera images by utilizing text spotting. We integrate these cues into a Monte Carlo localization framework using a particle filter that operates on 2D LiDAR scans and camera data. By this, we provide a robust localization solution for environments with structural changes and dynamics by humans walking. We evaluate our localization framework on multiple challenging indoor scenarios in an office environment. The experiments suggest that our approach is robust to structural changes and can run on an onboard computer. We release an open source implementation of our approach (upon paper acceptance), which uses off-the-shelf text spotting, written in C++ with a ROS wrapper.

Keywords: Localization
Abstract: Autonomous robots need to recognize the environment by identifying the scene. Scan context is one of global descriptors, and it encodes the three-dimensional scan data of the scene for the identification in a matrix form. Scan context is in a matrix form that is simple to store, but the matching of scan contexts can require computational effort because the descriptor is orientation-dependent. Because a scan context of an omnidirectional LiDAR scan becomes periodic in azimuth, this paper proposes to compute the scan context matching efficiently incorporating the cross-correlation with fast Fourier transform, and, hence, the method is named fast scan context matching. The effectiveness of the proposed method for computation time, accuracy, and robustness are reported in this paper. It is also shown that the method was also tested as a loop closure detector of a SLAM package as a practical application and that the proposed method outperformed the conventional scan context matching.

Keywords: Localization, Mapping
Abstract: Robots deployed in settings such as warehouses and parking lots must cope with frequent and substantial changes when localizing in their environments. While many previous localization and mapping algorithms have explored methods of identifying and focusing on long-term features to handle change in such environments, we propose a different approach – can a robot understand the distribution of movable objects and relate it to observations of such objects to reason about global localization? In this paper, we present probabilistic object maps (POMs), which represent the distributions of movable objects using pose-likelihood sample pairs derived from prior trajectories through the environment and use a Gaussian process classifier to generate the likelihood of an object at a query pose. We also introduce POM-Localization, which uses an observation model based on POMs to perform inference on a factor graph for globally consistent long-term localization. We present empirical results showing that POM-Localization is indeed effective at producing globally consistent localization estimates in challenging real-world environments and that POM-Localization improves trajectory estimates even when the POM is formed from partially incorrect data.

Keywords: Localization, SLAM
Abstract: In this paper, we propose an object-based camera pose estimation from a single RGB image and a pre-built map of objects, represented with ellipsoidal models. We show that contrary to point correspondences, the definition of a cost function characterizing the projection of a 3D object onto a 2D object detection is not straightforward. We develop an ellipse-ellipse cost based on level sets sampling, demonstrate its nice properties for handling partially visible objects and compare its performance with other common metrics. Finally, we show that the use of a predictive uncertainty on the detected ellipses allows a fair weighting of the contribution of the correspondences which improves the computed pose. The code is released at gitlab.inria.fr/tangram/level-set-based-camera-pose-estimation.

Keywords: Localization, Multi-Robot Systems
Abstract: Position estimation in Multi-Robot Systems (MRS) relies on relative angle or distance measurements between the robots, which generally deteriorate as distances increase. Moreover, the localization accuracy is strongly influenced both by the quality of the raw measurements but also by the overall geometry of the network. In this paper, we design a cost function that accounts for these two issues and can be used to develop motion planning algorithms that optimize the localizability in MRS, i.e., the ability of individual robots to localize themselves accurately. This cost function is based on computing new Cramer Rao Lower Bounds characterizing the achievable positioning performance with range and angle measurements that deteriorate with increasing distances. We describe a gradient-based motion-planning algorithm for MRS deployment that can be implemented in a distributed manner, as well as a non-myopic strategy to escape local minima. Finally, we test the proposed methodology experimentally for range measurements obtained using ultra-wide band transceivers and illustrate the improvements resulting from leveraging the more accurate measurement model in the robot placement algorithms.

Keywords: Localization, Autonomous Vehicle Navigation
Abstract: 6-DoF visual localization systems utilize principled approaches rooted in 3D geometry to perform accurate camera pose estimation of images to a map. Current techniques use hierarchical pipelines and learned 2D feature extractors to improve scalability and increase performance. However, despite gains in typical recall@0.25m type metrics, these systems still have limited utility for real-world applications like autonomous vehicles because of their worst areas of performance - the locations where they provide insufficient recall at a certain required error tolerance. Here we investigate the utility of using place specific configurations, where a map is segmented into a number of places, each with its own configuration for modulating the pose estimation step, in this case selecting a camera within a multi-camera system. On the Ford AV benchmark dataset, we demonstrate substantially improved worst-case localization performance compared to using off-the-shelf pipelines - minimizing the percentage of the dataset which has low recall at a certain error tolerance, as well as improved overall localization performance. Our proposed approach is particularly applicable to the crowdsharing model of autonomous vehicle deployment, where a fleet of AVs are regularly traversing a known route.

Keywords: Localization
Abstract: We present a novel online capable hybrid interval-probabilistic localization method using publicly available 2D building maps. Given an initially large uncertainty for the orientation and position derived from GNSS data, our novel interval-based approach first narrows down the orientation to a smaller interval and provides a set described by a minimal polygon for the position of the vehicle that encloses the feasible set of poses by taking the building geometry into account using 3D Light Detection and Ranging (LiDAR) sensor data. Second, we perform a probabilistic Maximum Likelihood Estimation (MLE) to determine the best solution within the determined feasible set. The MLE is converted into a least-squares problem that is solved by an optimization approach that takes the bounds of the solution set into account so that only a solution within the feasible set is selected as the most likely one. We experimentally show with real data that the novel interval-based localization provides sets of poses that contain the true pose for more than 99% of the frames and that the bounded optimization provides more reliable results compared to a classical unbounded optimization and a Monte Carlo Localization approach.

Keywords: Localization, Sensor Networks, Range Sensing
Abstract: This paper proposes a novel adaptive sample space-based Viterbi algorithm for target localization in an online manner. The method relies on discretizing the target's motion space into cells representing a finite number of hidden states. Then, the most probable trajectory of the tracked target is computed via dynamic programming in a Hidden Markov Model (HMM) framework. The proposed method uses a Bayesian estimation framework which is neither limited to Gaussian noise models nor requires a linearized target motion model or sensor measurement models. However, an HMM-based approach to localization can suffer from poor computational complexity in scenarios where the number of hidden states increases due to high-resolution modeling or target localization in a large space. To improve this poor computational complexity, this paper proposes a belief propagation in the most probable belief space with a low to high-resolution sequentially, reducing the required resources significantly. The proposed method is inspired by the k-d Tree algorithm (e.g., quadtree) commonly used in the computer vision field. Experimental tests using an ultra-wideband (UWB) sensor network demonstrate our results.

Keywords: Representation Learning, Human Factors and Human-in-the-Loop, Multi-Modal Perception for HRI
Abstract: Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible language-based interface for human-robot collaboration, which allows a user to reshape existing trajectories for an autonomous agent. We take advantage of recent advancements in the field of large language models (BERT and CLIP) to encode the user command, and then combine these features with trajectory information using multi-modal attention transformers. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands, and treat the trajectory generation process as a sequence prediction problem, analogously to how language generation architectures operate. We evaluate the system in multiple simulated trajectory scenarios, and show a significant performance increase of our model over baseline approaches. In addition, our real-world experiments with a robot arm show that users significantly prefer our natural language interface over traditional methods such as kinesthetic teaching or cost-function programming. Our study shows how the field of robotics can take advantage of large pre-trained language models towards creating more intuitive interfaces between robots and machines. Project webpage: https://arthurfenderbucker.github.io/NL_trajectory_reshaper

Keywords: Reinforcement Learning, Representation Learning, Machine Learning for Robot Control
Abstract: The present paper proposes a novel reinforcement learning method with world models, DreamingV2, a collaborative extension of DreamerV2 and Dreaming. DreamerV2 is a cutting-edge model-based reinforcement learning from pixels that uses discrete world models to represent latent states with categorical variables. Dreaming is also a form of reinforcement learning from pixels that attempts to avoid the autoencoding process in general world model training by involving a reconstruction-free contrastive learning objective. The proposed DreamingV2 is a novel approach of adopting both the discrete representation of DreamingV2 and the reconstruction-free objective of Dreaming. Compared to DreamerV2 and other recent model-based methods without reconstruction, DreamingV2 achieves the best scores on five simulated challenging 3D robot arm tasks. We believe that DreamingV2 will be a reliable solution for robot learning since its discrete representation is suitable to describe discontinuous environments, and the reconstruction-free fashion well manages complex vision observations.

Keywords: Representation Learning, Bioinspired Robot Learning, Imitation Learning
Abstract: One of the key challenges in visual imitation learning is collecting large amounts of expert demonstrations for a given task. While methods for collecting human demonstrations are becoming easier with teleoperation methods and the use of low-cost assistive tools, we often still require 100-1000 demonstrations for every task to learn a visual representation and policy. To address this, we turn to an alternate form of data that does not require task-specific demonstrations -- play. Playing is a fundamental method children use to learn a set of skills and behaviors and visual representations in early learning. Importantly, play data is diverse, task-agnostic, and relatively cheap to obtain. In this work, we propose to use playful interactions in a self-supervised manner to learn visual representations for downstream tasks. We collect 2 hours of playful data in 19 diverse environments and use self-predictive learning to extract visual representations. Given these representations, we train policies using imitation learning for two downstream tasks: Pushing and Stacking. We demonstrate that our visual representations generalize better than standard behavior cloning and can achieve similar performance with only half the number of required demonstrations. Our representations, which are trained from scratch, compare favorably against ImageNet pretrained representations. Finally, we provide an experimental analysis on the effects of different pretraining modes on downstream task learning.

Keywords: Representation Learning, AI-Enabled Robotics, Deep Learning for Visual Perception
Abstract: Learning representations that generalize across tasks and domains is challenging yet necessary for autonomous systems. Although task-driven approaches are appealing, designing models specific to each application can be difficult in the face of limited data, especially when dealing with highly variable multimodal input spaces arising from different tasks in different environments. We introduce the first general-purpose pretraining pipeline, COntrastive Multimodal Pretraining for AutonomouS Systems (COMPASS), to overcome the limitations of task-specific models and existing pretraining approaches. COMPASS constructs a multimodal graph by considering the essential information for autonomous systems and the properties of different modalities. Through this graph, multimodal signals are connected and mapped into two factorized spatio-temporal latent spaces: a 'motion pattern space' and a 'current state space'. By learning from multimodal correspondences in each latent space, COMPASS creates state representations that models necessary information such as temporal dynamics, geometry, and semantics. We pretrain COMPASS on a large-scale multimodal simulation dataset TartanAir~cite{tartanair2020iros} and evaluate it on drone navigation, vehicle racing, and visual odometry tasks. The experiments indicate that COMPASS can tackle all three scenarios and can also generalize to unseen environments and real-world data.

Keywords: AI-Enabled Robotics, Representation Learning, Human-Centered Robotics
Abstract: Learned knowledge graph representations supporting robots contain a wealth of domain knowledge that drives robot behavior. However, there does not exist an inference reconciliation framework that expresses how a knowledge graph representation affects a robot's sequential decision making. We use a pedagogical approach to explain the inferences of a learned, black-box knowledge graph representation, a knowledge graph embedding. Our interpretable model uses a decision tree classifier to locally approximate the predictions of the black-box model and provides natural language explanations interpretable by non-experts. Results from our algorithmic evaluation affirm our model design choices, and the results of our user studies with non-experts support the need for the proposed inference reconciliation framework. Critically, results from our simulated robot evaluation indicate that our explanations enable non-experts to correct erratic robot behaviors due to nonsensical beliefs within the black-box.

Keywords: Representation Learning, Deep Learning for Visual Perception, Legged Robots
Abstract: We propose a learning-based method to reconstruct the local terrain for locomotion with a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot's trajectory, the algorithm estimates the topography in the robot's vicinity. The raw measurements from these cameras are noisy and only provide partial and occluded observations that in many cases do not show the terrain the robot stands on. Therefore, we propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement. The model consists of a 4D fully convolutional network on point clouds that learns the geometric priors to complete the scene from the context and an auto-regressive feedback to leverage spatio-temporal consistency and use evidence from the past. The network can be solely trained with synthetic data, and due to extensive augmentation, it is robust in the real world, as shown in the validation on a quadrupedal robot, ANYmal, traversing challenging settings. We run the pipeline on the robot's onboard low-power computer using an efficient sparse tensor implementation and show that the proposed method outperforms classical map representations.

Keywords: Representation Learning, Deep Learning for Visual Perception
Abstract: This paper proposes a new network for reconstructing multi-object point cloud. Different from previous networks which reconstruct multi-object point cloud as a whole, our network iteratively reconstructs each individual object point cloud from a frame of multi-object point cloud. To achieve this goal, we have designed MO-Transformer, a transformer-based autoregressive network. During training, MO-Transformer takes a frame of multi-object point cloud and individual object point clouds as input. During testing, MOTransformer iteratively reconstructs individual object point clouds only based on the input multi-object point cloud. To train the proposed MO-Transformer, we design a new loss function called separate Chamfer distance (SCD). In addition, we prove that SCD is an upper bound of the traditional Chamfer distance calculated based on the entire multi-object point cloud. The reconstruction experiment verifies the efficacy of our network in multi-object point cloud reconstruction. Furthermore, the reconstruction experiment also investigates the effect of different dimensions using a series of datasets. The ablation study experiment verifies the necessity of SCD in training MO-Transformer.

Keywords: Representation Learning, Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception
Abstract: Self-supervised learning on point clouds has gained a lot of attention recently, since it addresses the label-efficiency and domain-gap problems on point cloud tasks. In this paper, we propose a novel self-supervised framework to learn informative features from partial point clouds. We leverage partial point clouds scanned by LiDAR that contain both content and pose attributes, and we show that disentangling such two factors from partial point clouds enhances feature learning. To this end, our framework consists of three main parts: 1) a completion network to capture holistic semantics of point clouds; 2) a pose regression network to understand the viewing angle where partial data is scanned from; 3) a partial reconstruction network to encourage the model to learn content and pose features. To demonstrate the robustness of the learnt feature representations, we conduct several downstream tasks including classification, part segmentation, and registration, with comparisons against state-of-the-art methods. Our method not only outperforms existing self-supervised methods, but also shows a better generalizability across synthetic and real-world datasets.

Keywords: Deep Learning in Grasping and Manipulation, Learning from Demonstration, Reinforcement Learning
Abstract: A key challenge in learning to perform manipulation tasks is selecting a suitable skill representation. While specific skill representations are often easier to learn, they are often only suitable for a narrow set of tasks. In most prior works, roboticists manually provide the robot with a suitable skill representation to use, e.g., a neural network or DMPs. By contrast, we propose to allow the robot to select the most appropriate skill representation for the underlying task. Given the large space of skill representations, we utilize a single demonstration to select a small set of potential task-relevant representations. This set is then further refined using reinforcement learning to select the most suitable skill representation. Experiments in both simulation and real world show how our proposed approach leads to improved sample efficiency and enables directly learning on the real robot.

Keywords: Automation at Micro-Nano Scales, Reinforcement Learning, Robust/Adaptive Control
Abstract: Laser-driven micro-tools are formulated by treating highly-focused laser beams as actuators, to control the tool's motion to contact then manipulate a micro-object, which allows it to manipulate opaque micro-objects, or large cells without causing photodamage. However, most existing laser-driven tools are limited to relatively simple tasks, such as moving and caging, and cannot carry out in-hand dexterous tasks. This is mainly because in-hand manipulation involves continuously coordinating multiple laser beams, micro-tools, and the object itself, which has high degrees of freedom (DoF) and poses up the challenge for planner and controller design. This paper presents a new hierarchical formulation for the grasping and manipulation of micro-objects using multiple laser-driven micro-tools. In hardware, multiple laser-driven tools are assembled to act as a robotic hand to carry out in-hand tasks (e.g., rotating); in software, a hierarchical scheme is developed to shrunken the action space and coordinate the motion of multiple tools, subject to both the parametric uncertainty in the tool and the unknown dynamic model of the object. Such a formulation provides a potential for achieving robotic in-hand manipulation at a micro scale. The performance of the proposed system is validated in simulation studies under different scenarios.

Keywords: Micro/Nano Robots, Automation at Micro-Nano Scales, Motion Control
Abstract: This article proposes a method for the correction of angular deviations caused during the fixing process of samples prepared for Atomic Force Microscopy (AFM). The correction is done using the angular control of a 6-DOF PPPS parallel platform were the sample is placed, while the AFM scan is performed by a 3-DOF serial cartesian robot with a tuning fork probe designed to perform FM-AFM. The method uses the generic x, y, and z data provided by the AFM after performing a scan on a free surface of the sample substrate. This is used to calculate the plane that closest approximates the points by solving a system of linear equations. This plane is then used to estimate the angular corrections that the 6-DOF parallel robot has to do in order to compensate the deviations. The proposed algorithm can be performed iteratively in order to refine the correction. The method also does not require any special preparation of the substrate. It only requires to have a free surface to scan. Experiments are performed using this algorithm to correct the orientation deviation of a substrate of V1 High-grade mica. The results show that the method is able to correct the angular deviation of the sample relatively to the AFM probe with an error of 0.2 ° after only two iterations of the algorithm.

Keywords: Micro/Nano Robots, Nanomanufacturing, Automation at Micro-Nano Scales
Abstract: Sub-structures such as micro-structured magnetic teeth fabricated with an artificial bacteria flagellum (ABF) are designed for achieving more motion modes, higher precision, and better controllability. To achieve these, a more precise model considering the non-circular cross-sectional features is setup without simplifying the structure as a helical filament with a circular cross-section as having been used in previous investigations, making it possible to include the effects of the substructures into the motion equation. Analyses and experiments verified the correctness. Besides of the geometric effects, our experimental observation also shows an anomalous step-out frequency appeared in an ABF. This asynchronous motion is attributed to the lag of magnetization with respect to the external rotating magnetic field due to the geometries and the soft-magnetic materials of the ribbons, which is different from the regular asynchronous motion solely caused by low Reynolds number of fluid to microscopic swimmers. While the lag of magnetization can be further attributed initiatively to the soft magnetic materials adopted, the feasibility to arrange the easy axis will enable many new possibilities, which is of particular interest in generating more modes for swarms such as cascade stepping out of ABFs with the same nominal overall sizes and for more precise positioning using stepping motion.

Keywords: Micro/Nano Robots, Deep Learning Methods, Visual Servoing
Abstract: Magnetic resonance imaging (MRI)-guided robots emerged as a promising tool for minimally invasive medical operations. Recently, MRI scanners have been proposed for actuating magnetic microrobots and localizing them in the patient’s body using two-dimensional (2D) MR images. However, three- dimensional (3D) magnetic microrobots tracking during motion is still an untackled issue in MRI-powered microrobotics. Here, we present a deep learning-based 3D magnetic microrobot tracking method using 2D MR images during microrobot motion. The proposed method comprises a convolutional neural network (CNN) and complementary particle filter for 3D microrobot tracking. The CNN localizes the microrobot position relative to the 2D MRI slice and classifies the microrobot visibility in the MR images. First, we create an ultrasound (US) imaging-mentored MRI-based microrobot imaging and actuation system to train the CNN. Then, we trained the CNN using the MRI data generated by automated experiments using US image-based visual servoing of a microrobot with a 500 μm-diameter magnetic core. We showed that the proposed CNN can localize the microrobot and classified its visibility in an in vitro environment with ±0.56 mm and 87.5% accuracy in 2D MR images, respectively. Furthermore, we demonstrated ex-vivo 3D microrobot tracking with ±1.43 mm accuracy, improving tracking accuracy by 60% compared to the previous studies. The presented tracking strategy will enable MRI-powered microrobots to be used in high-precision targeted medical applications in the future.

Keywords: Dexterous Manipulation, Micro/Nano Robots, Biological Cell Manipulation
Abstract: Existing micromanipulation methods, whether contact mi-cromanipulation or non-contact micromanipulation, can hardly meet the requirements of low damage and multiple functions in the biomedical field. This study provides a high-speed mi-cromanipulator that can simulate circular vibrations and gen-erate microflow, which could be utilized to actuate non-invasive multiple operations of biological targets at the microscale. We design a PZT-driven 6-DOF micromanipulator with a hybrid structure and flexible hinges. Two 3-PRS parallel modules are serially connected in mirror style to achieve high speed, high accuracy, and a big workspace simultaneously, which enables the highly-controllable circular vibration simulation and strong whirling flow generation. The static, dynamic, and trajectory tracking performances were evaluated. The experimental result showed the 324 × 331 × 40 μm3 workspace, which was also the range of trajectories that could be generated. Trajectory track-ing performance evaluation showed that it could realize ~200 Hz circular vibration with an error of about 2.4% through open-loop control. Finally, the microflow generation experiment indicated the great potential of the proposed micromanipulator and the method of whirling flow generation in operating the biological targets at the microscale.

Keywords: Automation at Micro-Nano Scales, Micro/Nano Robots, Grippers and Other End-Effectors
Abstract: Acoustic holography is a newly emerging and promising technique to dynamically generate arbitrary desired holographic acoustic field in 3D space for contactless robotic manipulation. The latest technology supporting complex dynamic holographic acoustic field reconstruction is through phased transducer array (PTA), where the phase profile of emitted acoustic wave from discrete transducers is controlled independently by sophisticated circuits to modulate the acoustic interference field. While the forward kinematics of a phased array based robotic manipulation system is simple and straightforward, the inverse kinematics of the required holographic acoustic field is mathematically non-linear and unsolvable, which substantially limits the application of dynamic holographic acoustic field for robot manipulation. In this work, we propose a physics-based deep learning framework for this phase retrieval inverse kinematics problem so that the target complex hologram could be reconstructed precisely with average MAE of 0.022 and in real time with prediction time of 47 milliseconds on GPU. The accuracy and real time of the proposed method for dynamic holographic acoustic field reconstruction from PTA are demonstrated experimentally.

Keywords: Soft Robot Materials and Design, Micro/Nano Robots, Assembly
Abstract: The magnetic microrobot has become a promising approach in many biomedical applications due to its small volume, flexible motion, and untethered micromachines. The micro-chain robot is one of the most popular magnetic microrobots. However, the uncontrollable magnetic moment direction and quantity of the magnetic beads consisted in the existing self-assembled micro-chain robot limit their locomotion and applications. This paper proposed an on-chip micro-chain robot fabrication method to assemble the magnetic beads with controllable magnetic moment direction and quantity. The bead quantity can be controlled by the structure limits of the microchannel, and the direction of the magnetic moment can be adjusted by the integrated external magnetic field. The assembled magnetic beads are then glued by the hydrogel under UV exposure. The micro-chain robots with different quantities and magnetic moment directions of the magnetic beads were successfully fabricated and tested in experiments. Due to the array structure of the microfluidic device, batch manufacturing of low-cost magnetic robots was achieved in our method. The movement of dual-bead microrobots with two orthogonal magnetic moment directions was analyzed and compared. One of the dual-bead microrobots was applied in the transportation of the hydrogel module using pushing and pulling modes. It indicated that the proposed controllable on-chip fabrication of the magnetic micro-chain robots has the potential to enhance the microrobot ability in biomedical applications.

Keywords: Automation at Micro-Nano Scales, Biological Cell Manipulation, Robotics and Automation in Life Sciences
Abstract: Zebrafish embryo injection is often required in biomedical research using zebrafish. In the injecting operation, trapping and rotating the zebrafish embryo to achieve a proper posture is essential for the high success rate. We proposed an on-chip platform capable of efficient and automatic trapping and rotating for the injection of zebrafish embryos. A low-cost 3D-printed microchannel is designed to trap zebrafish embryos into the cavity array. The blind-hole design at each cavity can generate microbubbles, and the bubbles exposed to the acoustic wave with a specific frequency can trap and rotate the zebrafish embryos. The progress, including trapping and rotating, can be monitored and executed automatically with computer vision assistance. Experimental results show that we realized on-chip trapping and rotating operations successfully. The success rate of trapping zebrafish embryos was up to 99%, and the time of trapping a single embryo was as low as 0.2 s. Embryo rotation could be achieved by two different modes, including continuous rotation and intermittent rotation. The accuracy and maximal velocity of rotating the embryo reached 5° and 3.5 r/s, respectively. Thus, we believe the proposed efficient automatic on-chip trapping and rotating platform could support the zebrafish embryo injection well.

Keywords: Micro/Nano Robots, Multi-Robot Systems, Motion Control
Abstract: Magnetically actuated small-scale robots have great potential for numerous applications in remote, confined, or enclosed environments. Multiple small-scale robots enable cooperation and increase the operating efficiency. However, independent control of multiple magnetic small-scale robots is a great challenge, because the robots receive identical control inputs from the same external magnetic field. In this article, we propose a novel strategy of completely decoupled independent control of magnetically actuated flexible swimming millirobots. A flexible millirobot shows a crawling motion on a flat plane within an oscillating magnetic field. Millirobots with different magnetization directions have the same velocity response curve to the oscillating magnetic field but with a difference of phase. We designed and fabricated a group of up to four heterogeneous millirobots with identical geometries and different magnetization directions. According to their velocity response curves, an optimal direction of oscillating magnetic field is calculated to induce a desired velocity vector for the millirobot group, one of which is nonzero and the others are approximately zero. The strategy is verified by experiments of independent position control of up to four millirobots and independent path following control of up to three millirobots with small errors. We further expect that with this independent control strategy, the millirobots will be able to cooperate to finish complicated tasks.

Keywords: Manipulation Planning, Machine Learning for Robot Control, Reinforcement Learning
Abstract: Pushing objects through cluttered scenes is a challenging task, especially when the objects to be pushed have initially unknown dynamics and touching other entities has to be avoided to reduce the risk of damage. In this paper, we approach this problem by applying deep reinforcement learning to generate pushing actions for a robotic manipulator acting on a planar surface where objects have to be pushed to goal locations while avoiding other items in the same workspace. With the latent space learned from a depth image of the scene and other observations of the environment, such as contact information between the end effector and the object as well as distance to the goal, our framework is able to learn contact-rich pushing actions that avoid collisions with other objects. As the experimental results with a six degrees of freedom robotic arm show, our system is able to successfully push objects from start to end positions while avoiding nearby objects. Furthermore, we evaluate our learned policy in comparison to a state-of-the-art pushing controller for mobile robots and show that our agent performs better in terms of success rate, collisions with other objects, and continuous object contact in various scenarios.

Keywords: Manipulation Planning, Deep Learning in Grasping and Manipulation, Grasping
Abstract: In this paper, we present a Domain Randomization and a Domain Adaptation approach to transfer experience for entanglement detection and separation from simulation into a real-world bin-picking application. We investigate the influence of different randomization options in image processing and use a CycleGAN as a further Domain Adaptation method to synthesize simulation data as realistically as possible. On the basis of this adapted data we re-train our detection and separation methods and validate the usefulness of these Sim-to-Real methods. In numerous real-world experiments we show that we achieve a significant increase of up to 71.74% in the performance of the overall system by using the Sim-to-Real approaches as opposed to the direct transfer.

Keywords: Manipulation Planning, Reinforcement Learning, Perception for Grasping and Manipulation
Abstract: Object rearranging is one of the most common deformable manipulation tasks, where the robot needs to rearrange a deformable object into a goal configuration. Previous studies focus on designing an expert system for each specific task by model-based or data-driven approaches and the application scenarios are therefore limited. Some research has been attempting to design a general framework to obtain more advanced manipulation capabilities for deformable rearranging tasks, with lots of progress achieved in simulation. However, transferring from simulation to reality is difficult due to the limitation of the end-to-end CNN architecture. To address these challenges, we design a local GNN (graph neural network) based learning method, which utilizes two representation graphs to encode keypoints detected from images. Self-attention is applied for graph updating and cross-attention is applied for generating manipulation actions. Extensive experiments have been conducted to demonstrate that our framework is effective in multiple 1-D (rope, rope ring) and 2-D (cloth) rearranging tasks in simulation and can be easily transferred to a real robot by fine-tuning a keypoint detector.

Keywords: Manipulation Planning, Dynamics, Kinematics
Abstract: Dynamic movements are ubiquitous in human motor behavior as they tend to be more efficient and can solve a broader range of skill domains than their quasi-static counterparts. For decades, robotic juggling tasks have been among the most frequently studied dynamic manipulation problems since the required dynamic dexterity can be scaled to arbitrarily high difficulty. However, successful approaches have been limited to basic juggling skills, indicating a lack of understanding of the required constraints for dexterous toss juggling. We present a detailed analysis of the toss juggling task, identifying the key challenges of the switching contacts task and formalizing it as a trajectory optimization problem. Building on our state-of-the-art, real-world toss juggling platform, we reach the theoretical limits of toss juggling in simulation, evaluate a resulting real-time controller in environments of varying difficulty and achieve robust toss juggling of up to 17 balls on two anthropomorphic manipulators. https://sites.google.com/view/controlling-the-cascade

Keywords: Manipulation Planning
Abstract: Robot manipulation in cluttered environments often requires complex and sequential rearrangement of multiple objects in order to achieve the desired reconfiguration of the target objects. Due to the sophisticated physical interactions involved in such scenarios, rearrangement-based manipulation is still limited to a small range of tasks and is especially vulnerable to physical uncertainties and perception noise. This paper presents a planning framework that leverages the efficiency of sampling-based planning approaches, and closes the manipulation loop by dynamically controlling the planning horizon. Our approach interleaves planning and execution to progressively approach the manipulation goal while correcting any errors or path deviations along the process. Meanwhile, our framework allows the definition of manipulation goals without requiring explicit goal configurations, enabling the robot to flexibly interact with all objects to facilitate the manipulation of the target ones. With extensive experiments both in simulation and on a real robot, we evaluate our framework on three manipulation tasks in cluttered environments: grasping, relocating, and sorting. In comparison with two baseline approaches, we show that our framework can significantly improve planning efficiency, robustness against physical uncertainties, and task success rate under limited time budgets.

Keywords: Manipulation Planning, Deep Learning in Grasping and Manipulation, Grasping
Abstract: We propose a novel Parallel Monte Carlo tree search with Batched Simulations (PMBS) algorithm for accelerating long-horizon, episodic robotic planning tasks. Monte Carlo tree search (MCTS) is an effective heuristic search algorithm for solving episodic decision-making problems whose underlying search spaces are expansive. Leveraging a GPU-based large-scale simulator, PMBS introduces massive parallelism into MCTS for solving planning tasks through the batched execution of a large number of concurrent simulations, which allows for more efficient and accurate evaluations of the expected cost-to-go over large action spaces. When applied to the challenging manipulation tasks of object retrieval from clutter, PMBS achieves a speedup of over 30x with an improved solution quality, in comparison to a serial MCTS implementation. We show that PMBS can be directly applied to real robot hardware with negligible sim-to-real differences. Supplementary material, including video, can be found at https://github.com/arc-l/pmbs.

Keywords: Humanoid and Bipedal Locomotion, Robust/Adaptive Control, Reinforcement Learning
Abstract: Recent advances in legged locomotion have enabled quadrupeds to walk on challenging terrain. However, bipedal robots are inherently more unstable and hence it's harder to design walking controllers for them. In this work, we leverage the recent advances in rapid adaptation for walking, and extend them to work on bipedal robots. Similar to existing works, we start with a base policy which produces actions while taking as input an estimated extrinsics vector from an adaptation module. This extrinsics vector contains information about the environment and enables the walking controller to rapidly adapt online. However, the extrinsics estimator could be imperfect, which might lead to poor performance of the base policy which expects a perfect estimator. We propose A-RMA (adapting RMA), which additionally adapts the base policy for the imperfect extrinsics estimator by finetuing it using model free RL. We demonstrate that A-RMA outperforms a number of RL based baseline controllers and model based controllers in simulation, and show zero-shot deployment of a single A-RMA policy to enable Cassie (a bipedal robot) to walk in a variety of different scenarios beyond what it has seen during training. Videos at url{https://ashish-kmr.github.io/a-rma/}

Keywords: Humanoid and Bipedal Locomotion, Whole-Body Motion Planning and Control, Underactuated Robots
Abstract: This paper presents a novel framework for point-foot biped running in three-dimensional space. The proposed approach generates center of mass (CoM) reference trajectories based on a differentially flat spring-loaded inverted pendulum (SLIP) model. A foothold planner is used to select touch down location that renders optimal CoM trajectory for upcoming step in real time. Dynamically feasible trajectories of CoM and orientation are subsequently generated by a simplified single rigid body (SRB) model based model predictive control (MPC). A task-space controller is then applied online to compute whole-body joint torques which embeds these target dynamics into the robot. The proposed approach is evaluated on physical simulation of a 12 degree-of-freedom (DoF), 7.95 kg point-foot bipedal robot. The robot achieves stable running at at varying speeds with maximum value of 1.1 m/s. The proposed scheme is shown to be able to reject vertical disturbances of 8 Ns and lateral disturbance of 6.5 Ns applied at the robot base.

Keywords: Telerobotics and Teleoperation
Abstract: We consider the problem of controlling a bilateral leader-follower robotic surgical set-up to allow kinesthetic haptic feedback to the user when the instrument approaches a forbidden area like sensitive organs arteries or veins that should be protected from injuries during surgery. The leader is a haptic device while the follower is a general purpose manipulator holding an elongated tool with an articulated instrument that should be manipulated through an entry port. We propose a control framework that is proved passive, incorporating a target admittance model for the follower that is designed in a way to impose a remote center of motion (RCM) while being subject to repulsive forces generated by properly designed artificial potentials associated with forbidden areas. Simulation and experimental results utilizing a virtual intraoperative environment provided as a point cloud of a kidney and its surrounding vessels characterized as forbidden areas, validate and demonstrate the performance of the proposed control scheme.

Keywords: Biologically-Inspired Robots, Motion Control, Field Robots
Abstract: WORMESH is a unique robot concept inspired by flatworm locomotion and its key feature is the use of multiple traveling waves for locomotion. This paper presents the steering method for anisotropic module configuration (AMC) of WORMESH-II based on the kinematics of skid steering of mobile robots. AMC of WORMESH-II used two parallel pedal waves to generate locomotion. The kinematic model of WORMESH-II shows its longitudinal and angular velocities depend on the summation and the difference of two synchronous left and right pedal waves amplitudes Al and Ar respectively. When both pedal waves have the same amplitude, the robot moves on a straight line, whereas the trajectory becomes a curve for different wave amplitudes. The radius of curve trajectory is inversely proportional to j Al 􀀀Ar j. The proposed method was ineffective when Ai  0(i = l; r). The proposed method was confirmed by the dynamic simulation of WORMESH-II using a physics engine. Moreover, the recommended skid steering method was tested using the prototype and verified.

Keywords: Soft Robot Applications, Biologically-Inspired Robots, Tendon/Wire Mechanism
Abstract: The paper presents a novel way to realize in-hand manipulation inspired by the peristaltic motion of a large-sized caterpillar's locomotion. The sharp contrast of the proposed soft-bodied finger to the traditional hard/rigid robotic ones is peristaltic motion with both compression and bending deformations. The design is based on the biological fact that large-size caterpillars (e.g., Bombyx mori) utilize bending and compression/extension of the body to produce crawling locomotion. Exploiting the multi-modal deformations, the prototype hand consisting of two proposed fingers can rotate and transport the grasped object, we demonstrated. We also found that the two fingers' motion's time gap is needed to stabilize in-hand manipulation. The design can shed new light on designing a gripper inspired by soft-bodied creatures.

Keywords: Biologically-Inspired Robots, Biomimetics
Abstract: Undulation is the most common gait generated by legless creatures, which enables their robust and efficient locomotion in various environments. Such advantages inspired the control design of many kinds of locomotion robots. Despite their technical details, most of them realize the undulation gait via tracking predetermined trajectories called serpenoid curves, which are a group of sinusoidal waveforms with specified phase differences. This technique, however, sounds quite redundant in terms of sensing and control. Here, we investigate the research question: whether the sinusoidal waveform is necessary to be encoded in the control signal to make the whole body an “S-shape”? We use a 4-link rigid body dynamics model as a simple example, by which numerical simulations are conducted. Together with theoretical analysis, we show that undulation gait emerges naturally based on embodied position controller and filter, where binary actuation torques are required only. Our results not only discover locomotion mechanisms for significantly reducing the sensing and control requirement of generating artificial undulation gait, but also provide additional understandings for biological systems from the mechanical engineering point of view.

Keywords: Biologically-Inspired Robots, Biomimetics
Abstract: Enlightened by the creeping gait of natural snakes, snake-like robots swing joints side to side at similar tracks for generating propelling forces. However, it is not always essential to control all joints of a snake-like robot to realize the creeping gait. Therefore, in this paper, a creeping snake-like robot with partially actuated joints has been investigated, towards reducing the redundancy caused by full actuation. Essentially, this approach is composed of the following two concepts: 1) joint equipped with torsion spring mechanism bridges the passive joint to generate rhythm oscillation, and 2) harmonic joint trajectories assist the robot in generating more efficient locomotion. We hereafter demonstrated that the actuated joint dominates passive dynamics of the system, which contributes to overall motion. Meanwhile, different spring stiffness affects the motion performance. Additionally, the interaction between robot and environment through Coulomb friction has been considered to reveal the contributing factors that assist the snake-like robot to yield better locomotion performance.

Keywords: Biologically-Inspired Robots, Mechanism Design, Computer Vision for Manufacturing
Abstract: This paper presents a novel snake robot with the docking function, which can help the snake robots to connect with each other to achieve a stronger one with double length and double degrees of freedom. First, the mechanical design of the snake robot with docking function is introduced, including the body link and the head-tail passive docking mechanical structure. Second, the control system is built, and the control strategies of locomotion and docking are separately proposed. Then, the visual perception function is implemented for the target recognition during the docking process. Finally, the prototype is developed. The mobility and the docking function are fully verified and analyzed through the physical experiments.

Keywords: Biologically-Inspired Robots, Mechanism Design, Legged Robots
Abstract: In this study, trajectory-tracking control for a lizard-inspired single-actuated robot (LISA) is implemented using a novel morphology. A high degree of reproducibility is obtained between the results of kinematic analysis and the behavior of the actual robot. Several 1-degree-of-freedom (DOF)-driven robots have been proposed. However, the application of these robots is severely restricted because of their inherent morphology limitations. LISA is a multilegged robot that can overcome these disadvantages and propel and turn with 1-DOF. In this study, we formulate the kinematics, such as the turning angle and stride length, of LISA. Furthermore, a unique robot coordinate is defined to enable the symmetrical representation of critical robot state quantities. Next, a trajectory-tracking control system based on the proportional-integral-derivative (PID) control is designed, and it is verified by both experiments and numerical simulations. Finally, the results of the experiments and numerical simulations are quantitatively compared according to the root mean square, and the reproducibility of the kinematics analysis and the behavior of the actual system are discussed. This study makes the following contributions: (1) The morphology of LISA facilitates analytical results with only kinematic analysis, which is considerably simpler than dynamics analysis. (2) LISA achieved sufficiently good kinematic performance for the required motions. Experiments revealed that the designed trajectory-tracking control system allows for LISA to appropriately track several types of trajectories.

Keywords: Biologically-Inspired Robots, Motion Control
Abstract: This letter proposes a real-time optimal robust path following control scheme for planar snake robots without sideslip constraints using model predictive control (MPC). One of the features is that a linear double-integrator model rather than the complex dynamic model of snake robots is used for the MPC design to improve calculation efficiency. Moreover, in addition to constraints on joint angles, velocity, and acceleration for security, constraints on the joint offset and velocity are also considered to keep the snake robot moving forward efficiently. And the multiple inequality constraints are handled by a novel constraints translator. Furthermore, to achieve robust path following control subject to unknown and varied friction forces and disturbances, two reduced-order structure-improved extended state observers are designed to avoid complex environmental modeling. Extensive comparative experiments were conducted to verify the constraints design and the effectiveness of the proposed control scheme.

Keywords: Biologically-Inspired Robots, Redundant Robots, Probability and Statistical Methods
Abstract: Reinforcement learning commonly suffers from slow convergence speed and requires thousands of episodes, which makes it hard to be applied for physical robotic applications. Because of the complexity of a large degree of freedom, little study has been done on snake robot control using reinforcement learning. Existing methods either adopts an asynchronous A3C structure or a joint state representation. In this paper, a fully distributed coach-based reinforcement learning method is proposed for snake robot control that can considerably accelerate training while using fewer episodes. The major contributions comprises: 1) we extend our previously presented graphical coach based RL control method into a completely distributed framework; 2) an explicit stochastic density propagation rule for each robot link is mathematically derived; 3) the various interactions with uncertainty have been well modeled and estimated to achieve an efficient and robust control for snake robots. The preliminary results of both simulation and real-world experiments have demonstrated promising performance comparing with other recent methods.

Keywords: Biologically-Inspired Robots, Climbing Robots, Legged Robots
Abstract: Climbing robots have the potential to revolutionize the maintenance and inspection operations of many types of vertical structures. In nature, parrots exhibit a remarkable capacity for manipulation during climbing behaviors, for which robotics can benefit from studying. In this paper we present a novel tripedal robot that is inspired by the morphology of these impressive birds, which use their legs and beak in a tripedal fashion when climbing. We propose several foot placement, trajectory generation, and control methods for this system along with performance evaluation in simulation. A video of select simulations and live bird data is included in the supplementary material, and can also be found at https: //youtu.be/vRVGraIyQgQ.

Keywords: Formal Methods in Robotics and Automation, Hybrid Logical/Dynamical Planning and Verification, Multi-Robot Systems
Abstract: In this work, we synthesize control for high-level, reactive robot tasks that include timing constraints and choices over goals and constraints. We enrich Event-based Signal Temporal Logic by adding disjunctions, and propose a framework for synthesizing controllers that satisfy such specifications. If there are multiple ways to satisfy a specification, we choose, at run-time, a controller that instantaneously maximizes robustness. During execution, we automatically generate feedback in the form of pre-failure warnings that give users insight as to why a specification may be violated in the future. We demonstrate our work through physical and simulated multi-robot systems operating in complex environments.

Keywords: Formal Methods in Robotics and Automation, Manipulation Planning, Learning from Demonstration
Abstract: For performing robotic manipulation tasks, the core problem is determining suitable trajectories that fulfill the task requirements. Various approaches to compute such trajectories exist, being learning and optimization the main driving techniques. Our work builds on the learning-from-demonstration (LfD) paradigm, where an expert demonstrates motions, and the robot learns to imitate them. However, expert demonstrations are not sufficient to capture all sorts of task specifications, such as the timing to grasp an object. In this paper, we propose a new method that considers formal task specifications within LfD skills. Precisely, we leverage Signal Temporal Logic (STL), an expressive form of temporal properties of systems, to formulate task specifications and use black-box optimization (BBO) to adapt an LfD skill accordingly. We demonstrate our approach in simulation and on a real industrial setting using several tasks that showcase how our approach addresses the LfD limitations using STL and BBO.

Keywords: Formal Methods in Robotics and Automation
Abstract: Time-series data classification is central to the analysis and control of autonomous systems, such as robots and self-driving cars. Temporal logic-based learning algorithms have been proposed recently as classifiers of such data. However, current frameworks are either inaccurate for real-world applications, such as autonomous driving, or they generate long and complicated formulae that lack interpretability. To address these limitations, we introduce a novel learning method, called Boosted Concise Decision Trees (BCDTs), to generate binary classifiers that are represented as Signal Temporal Logic (STL) formulae. Our algorithm leverages an ensemble of Concise Decision Trees (CDTs) to improve the classification performance, where each CDT is a decision tree that is empowered by a set of techniques to generate simpler formulae and improve interpretability. The effectiveness and classification performance of our algorithm are evaluated on naval surveillance and urban-driving case studies.

Keywords: Formal Methods in Robotics and Automation
Abstract: We develop a method for synthesizing control policies for stochastic, linear, time-varying systems that must perform tasks specified in signal temporal logic. We build upon an efficient, sampling-based framework that computes the probability of the system satisfying its specification. By exploiting the properties of linear systems and robustness score in temporal logic specifications, we obtain sample-efficient gradients of the satisfaction probability with respect to controller parameters. Therefore, by applying gradient descent we obtain locally optimized controllers that maximize the chances of satisfying the specification. We demonstrate our approach through examples of a mobile robot and a mobile manipulator in simulation.

Keywords: Formal Methods in Robotics and Automation, Kinematics, Robot Safety
Abstract: In this paper, we propose a preliminary definition and analysis of the novel concept of sensor observability index. The goal is to analyse and evaluate the performance of distributed directional or axial-based sensors to observe specific axes in task space as a function of joint configuration in serial robot manipulators. For example, joint torque sensors are often used in serial robot manipulators and assumed to be perfectly capable of estimating end effector forces, but certain joint configurations may cause one or more task-space axes to be unobservable as a result of how the joint torque sensors are aligned. The proposed sensor observability provides a method to analyse the quality of the current robot configuration to observe the task space. Parallels are drawn between sensor observability and the traditional kinematic Jacobian for the particular case of joint torque sensors in serial robot manipulators. Although similar information can be retrieved from kinematic analysis of the Jacobian transpose in serial manipulators, sensor observability is shown to be more generalizable in terms of analysing non-joint-mounted sensors and other sensor types. In addition, null-space analysis of the Jacobian transpose is susceptible to false observability singularities. Simulations and experiments using the robot Baxter demonstrate the importance of maintaining proper sensor observability in physical interactions.

Keywords: Formal Methods in Robotics and Automation, Optimization and Optimal Control, Hybrid Logical/Dynamical Planning and Verification
Abstract: Mobility-on-demand systems are transforming the way we think about the transportation of people and goods. Most research effort has been placed on scalability issues for systems with a large number of agents and simple pick-up/drop-off demands. In this paper, we consider fair multi-vehicle route planning with streams of complex, temporal logic transportation demands. We consider the envy-free fair allocation of demands to limited-capacity vehicles based on agents' accumulated utility over a finite time horizon, representing for example monetary reward or utilization level. We propose a scalable approach based on the construction of assignment graphs that relate agents to routes and demands and pose the problem as an Integer Linear Program (ILP). Routes for assignments are computed using automata-based methods for each vehicle and demands sets of size at most the capacity of the vehicle while taking into account their pick-up wait time and delay tolerances. In addition, we integrate utility-based weights in the assignment graph and ILP to ensure approximative fair allocation. We demonstrate the computational and operational performance of our methods in ride-sharing case studies over a large environment in mid-Manhattan and Linear Temporal Logic demands with stochastic arrival times. We show that our method significantly decreases the utility deviation between agents and the vacancy rate.

Keywords: Formal Methods in Robotics and Automation, Motion and Path Planning, Computational Geometry
Abstract: We present an incremental topology-based motion planner that, while planning paths in the configuration space, performs metric gluing on the constructed Vietoris-Rips simplicial complex of each sub-space (voxel). By incrementally capturing topological and geometric information in batches of voxel graphs, our algorithm avoids the time overhead of analyzing the properties of the entire configuration space. We theoretically prove in this paper that the simplices of all voxel graphs joined together are homotopy-equivalent to the union of the simplices in the configuration space. Experiments were carried out in seven different environments using various robots, including the articulated linkage robot, the Kuka YouBot, and the PR2 robot. In all environments, the results show that our algorithm achieves better convergence for path cost and computation time with a memory-efficient roadmap than state-of-the-art methods.

Keywords: Hardware-Software Integration in Robotics, Engineering for Robotic Systems, Software-Hardware Integration for Robot Systems
Abstract: We propose a novel hybrid system (both hardware and software) of an Unmanned Aerial Vehicle (UAV) carrying a miniature Unmanned Ground Vehicle (miniUGV) to perform a complex search and manipulation task. This system leverages the heterogeneous robots to accomplish a task that cannot be done using a single robot system. It enables the UAV to explore a hidden space with a narrow opening through which the miniUGV can easily enter and escape. The hidden space is assumed to be navigable for the miniUGV. The miniUGV uses Infrared (IR) sensors and a monocular camera to search an object in the hidden space. The proposed system takes advantage of a wider field of view (fov) of camera as well as the stochastic nature of the object detection algorithms to guide the robot in the hidden space to find the object. Upon finding the object the miniUGV grabs it using visual servoing and then returns back to its start point from where the UAV retracts it back and transports the object to a safe place. In case there is no object found in the hidden space, UAV continues the aerial search. The tethered miniUGV gives the UAV an ability to act beyond its reach and perform a search and manipulation task which was not possible before for any of the robots individually. The system has a wide range of applications and we have demonstrated its feasibility through repetitive experiments

Keywords: Formal Methods in Robotics and Automation
Abstract: This paper is interested in formalizing human trajectories in human-robot encounters. Inspired by robot navigation tasks in human-crowded environments, we consider the case where a human and a robot walk towards each other, and where humans have to avoid colliding with the incoming robot. Further, humans may describe different behaviors, ranging from being in a hurry/minimizing completion time to maximizing safety. We propose a decision tree-based algorithm to extract STL formulae from multi-label data. Our inference algorithm learns STL specifications from data containing multiple classes, where instances can be labelled by one or many classes. We base our evaluation on a dataset of trajectories collected through an online study reproducing human-robot encounters.

Keywords: SLAM, Mapping, Computer Vision for Automation
Abstract: Object-level SLAM introduces semantic-meaningful and compact object landmarks that help both indoor robot applications and outdoor autonomous driving tasks. However, the back end of object-level SLAM suffers from singularity problems because existing methods parameterize object landmarks separately by their scale and pose. Under that parameterization method, the same abstract object can be represented by rotating the object coordinate frame by 90 degrees and swapping its length with width value, making the pose of the same object landmark not globally consistent. To avoid the singularity problem, we first introduce the symmetric positive-definite (SPD) matrix manifold as an improved object-level landmark representation and further improve the cost functions in the back end to make them compatible with it. Our method demonstrates a faster convergence rate and more robustness in simulation experiments. And experiments on real datasets reveal that using the same front-end data, our strategy improves mapping accuracy by 22% on average.

Keywords: SLAM, Mapping, Localization
Abstract: Simultaneous Localization and Mapping (SLAM) has greatly assisted in exploring perceptually-degraded underground environments, such as human-made tunnels, mine tunnels, and caves. However, the recurring sensor failures and spurious loop closures in these scenes bring significant challenges to applying SLAM. This paper proposes an architecture for underground localization and semantic mapping (ULSM) that promotes the robustness of odometry estimation and map-building. In this architecture, a two-stage robust motion compensation method is proposed to adapt to sensor-failure situations. The proposed salient region loop closure detection contributes to avoiding spurious loop closures. Meanwhile, the 2D pose as the initial value for point cloud registration is estimated without additional input. We also design a multi-robot cooperative mapping scheme based on descriptors of the salient region. Extensive experiments are conducted on datasets collected in the Tunnel Circuit of DARPA Subterranean Challenge.

Keywords: SLAM, Range Sensing
Abstract: Loop closure detection is a key technology for long-term robot navigation in complex environments. In this paper, we present a global descriptor, named Normal Distribution Descriptor (NDD), for 3D point cloud loop closure detection. The descriptor encodes both the probability density score and entropy of a point cloud as the descriptor. We also propose a fast rotation alignment process and use correlation coefficient as the similarity between descriptors. Experimental results show that our approach outperforms the state-of-the-art point cloud descriptors in both accuracy and efficency. The source code will be available and can be integrated into existing LiDAR odometry and mapping (LOAM) systems.

Keywords: SLAM, Sensor Fusion, Visual-Inertial SLAM
Abstract: In recent years, Visual-Inertial Odometry (VIO) has achieved many significant progresses. However, VIO methods suffer from localization drift over long trajectories. In this paper, we propose a First-Estimates Jacobian Visual-Inertial-Ranging Odometry (FEJ-VIRO) to reduce the localization drifts of VIO by incorporating ultra-wideband (UWB) ranging measurements into the VIO framework consistently. Considering that the initial positions of UWB anchors are usually unavailable, we propose a long-short window structure to initialize the UWB anchors' positions as well as the covariance for state augmentation. After initialization, the FEJ-VIRO estimates the UWB anchors' positions simultaneously along with the robot poses. We further analyze the observability of the visual-inertial-ranging estimators and proved that there are four unobservable directions in the ideal case, while only three unobservable directions exist in the actual case. Based on these analyses, we propose to leverage the FEJ technique to fuse ranging measurements consistently. Finally, we validate our analysis and the proposed FEJ-VIRO with both simulation and real-world experiments.

Keywords: SLAM, Visual Tracking, Sensor Fusion
Abstract: This work presents a novel dense RGB-D SLAM approach for dynamic planar environments that enables simultaneous multi-object tracking, camera localisation and background reconstruction. Previous dynamic SLAM methods either rely on semantic segmentation to directly detect dynamic objects; or assume that dynamic objects occupy a smaller proportion of the camera view than the static background and can, therefore, be removed as outliers. With the aid of camera motion prior, our approach enables dense SLAM when the camera view is largely occluded by multiple dynamic objects. The dynamic planar objects are separated by their different rigid motions and tracked independently. The remaining dynamic non-planar areas are removed as outliers and not mapped into the background. The evaluation demonstrates that our approach outperforms the state-of-the-art methods in terms of localisation, mapping, dynamic segmentation and object tracking. We also demonstrate its robustness to large drift in the camera motion prior.

Keywords: SLAM, Localization, RGB-D Perception
Abstract: Visual SLAM methods based on point features have achieved acceptable results in texture-rich static scenes, but they often suffer from a deficiency of texture and the existence of dynamic objects in real indoor scenes, which limits the application of these methods. In this paper, we have presented DRG-SLAM, which combines line features and plane features into point features to improve the robustness of the system. We tested the proposed algorithm on publicly available datasets, and the results demonstrate that the algorithm has superior accuracy and robustness in indoor dynamic scenes compared with the state-of-the-art methods.

Keywords: SLAM, Localization, Mapping
Abstract: We present a generic framework for scale-aware direct monocular odometry based on depth prediction from a deep neural network. In contrast with previous methods where depth information is only partially exploited, we formulate a novel depth prediction residual which allows us to incorporate multi-view depth information. In addition, we propose to use a truncated robust cost function which prevents considering inconsistent depth estimations. The photometric and depth-prediction measurements are integrated into a tightly-coupled optimization leading to a scale-aware monocular system which does not accumulate scale drift. Our proposal does not particularize for a concrete neural network, being able to work along with the vast majority of the existing depth prediction solutions. We demonstrate the validity and generality of our proposal evaluating it on the KITTI odometry dataset, using two publicly available neural networks and comparing it with similar approaches and the state-of-the-art for monocular and stereo SLAM. Experiments show that our proposal largely outperforms classic monocular SLAM, being 5 to 9 times more precise, beating similar approaches and having an accuracy which is closer to that of stereo systems.

Keywords: SLAM, Localization
Abstract: 2D SLAM is useful for mobile robots that are constrained to a 2D plane, for example in a warehouse, simplifying calculations in respect to the 3D case. The use of an IMU in such a context can enrich the estimation and make it more robust. In this paper we reformulate the IMU preintegration widely used in 3D problems for the 2D case, making use of Lie Theory. The Lie theory based formalization, first derived for a perfectly horizontal plane, allows us to easily extend it to problems where the plane is not orthogonal to the gravity vector. We implement the theory in a factor graph based estimation library, and carry out experiments to validate it on a mobile platform. Two experiments are carried out; on a horizontal and a sloped environment, and the sensor data is processed using our two 2D methods and a state-of-the-art 3D method.

Keywords: SLAM, Mapping, Localization
Abstract: The scale ambiguity problem is inherently unsolvable to monocular SLAM without the metric baseline between moving cameras. In this paper, we present a novel scale estimation approach based on an object-level SLAM system. To obtain the absolute scale of the reconstructed map, we formulate an optimization problem to make the scaled dimensions of objects conform to the distribution of their sizes in the physical world, without relying on any prior information about gravity direction. The dual quadric is adopted to represent objects for its ability to describe objects compactly and accurately, thus providing reliable dimensions for scale estimation. In the proposed monocular object-level SLAM system, semantic objects are initialized first from fitted 3-D oriented bounding boxes and then further optimized under constraints of 2-D detections and 3-D map points. Experiments on indoor and outdoor public datasets show that our approach outperforms existing methods in terms of accuracy and robustness.