Keywords: Robotics in Hazardous Fields, Search and Rescue Robots, Human Factors and Human-in-the-Loop
Abstract: This paper reports on insights by robotics researchers that participated in a 5-day robot-assisted nuclear disaster response field exercise conducted by Kerntechnische Hilfdienst GmbH (KHG) in Karlsruhe, Germany. The German nuclear industry established KHG to provide a robot-assisted emergency response capability for nuclear accidents. We present a systematic description of the equipment used; the robot operators' training program; the field exercise and robot tasks; and the protocols followed during the exercise. Additionally, we provide insights and suggestions for advancing disaster response robotics based on these observations. Specifically, the main degradation in performance comes from the cognitive and attentional demands on the operator. Furthermore, robotic platforms and modules should aim to be robust and reliable in addition to their ease of use. Last, as emergency response stakeholders are often skeptical about using autonomous systems, we suggest adopting a variable autonomy paradigm to integrate autonomous robotic capabilities with the human-in-the-loop gradually. This middle ground between teleoperation and autonomy can increase end-user acceptance while directly alleviating some of the operator's robot control burden and maintaining the resilience of the human-in-the-loop.

Keywords: Aerial Systems: Mechanics and Control, Robot Safety, Force Control
Abstract: As the performance of autonomous systems increases, safety concerns arise, especially when operating in non-structured environments. To deal with these concerns, this work presents a safety layer for mechanical systems that detects and responds to unstable dynamics caused by external disturbances. The safety layer is implemented independently and on top of already present nominal controllers, like pose or wrench tracking, and limits power flow when the system's response would lead to instability. This approach is based on the computation of the Largest Lyapunov Exponent (LLE) of the system’s error dynamics, which represent a measure of the dynamics’ divergence or convergence rate. By actively computing this metric, divergent and possibly dangerous system behaviors can be promptly detected. The LLE is then used in combination with Control Barrier Functions (CBFs) to impose power limit constraints on a jerk controlled system. The proposed architecture is experimentally validated on an Omnidirectional Micro Aerial Vehicle (OMAV) both in free flight and interaction tasks.

Keywords: Aerial Systems: Perception and Autonomy, Motion and Path Planning
Abstract: This paper contributes a novel strategy towards risk-aware motion planning for collision-tolerant aerial robots subject to localization uncertainty. Attuned to the fact that micro aerial vehicles are often tasked to navigate within GPS-denied, possibly unknown, confined and obstacle-filled environments the proposed method exploits collision-tolerance at the robot design level to mitigate the risks of collisions especially as their likelihood increases with growing uncertainty. Accounting for the maximum kinetic energy with which an impact is considered safe, alongside the robot dynamics, the planner builds a set of admissible uncertainty-aware and collision-inclusive paths over a horizon involving multiple motion steps. The first step of the best path is executed by the robot, while the procedure is then repeated in a receding horizon manner. Evaluated in extensive simulation studies and experimental results with a collision-tolerant flying robot, the planner successfully considers the interplay between uncertainty and the likelihood of a collision, balances the risks of possible impacts and enables to navigate safely within highly cluttered environments.

Keywords: Aerial Systems: Applications, Aerial Systems: Perception and Autonomy, Manipulation Planning
Abstract: While the variety of applications for Aerial Manipulators (AMs) has increased over the last years, they are mostly limited to push-and-slide tasks. More complex manipulations of dynamic environments are poorly addressed and still require handcrafted designs of hardware, control, and trajectory planning. In this paper we focus on the active manipulation of articulated objects with AMs. We present a novel planning and control approach that allows the AM to execute complex interaction maneuvers with as little as possible priors given by the operator. Our framework combines sampling-based predictive control to generate pose trajectories with an impedance controller for compliant behaviours, applied to a fully-actuated flying platform. The framework leverages a physics engine to simulate the dynamics of the platform and the environment in order to find optimal motions to execute manipulation tasks. Experiments on two selected examples of pulling open a door and of turning a valve show the feasibility of the proposed approach.

Keywords: Robot Safety, Aerial Systems: Mechanics and Control, Multi-Robot Systems
Abstract: The weight, space, and power limitations of small aerial vehicles often prevent the application of modern control techniques without significant model simplifications. Moreover, high-speed agile behavior, such as that exhibited in drone racing, make these simplified models too unreliable for safety-critical control. In this work, we introduce the concept of time-varying backup controllers (TBCs): user-specified maneuvers combined with backup controllers that generate reference trajectories which guarantee the safety of nonlinear systems. TBCs reduce conservatism when compared to traditional backup controllers and can be directly applied to multi-agent coordination to guarantee safety. Theoretically, we provide conditions under which TBCs strictly reduce conservatism, describe how to switch between several TBC's and show how to embed TBCs in a multi-agent setting. Experimentally, we verify that TBCs safely increase operational freedom when filtering a pilot's actions and demonstrate robustness and computational efficiency when applied to decentralized safety filtering of two quadrotors.

Keywords: Mobile Manipulation, Vision-Based Navigation, Machine Learning for Robot Control
Abstract: While both navigation and manipulation are challenging topics in isolation, many tasks require the ability to both navigate and manipulate in concert. To this end, we propose a mobile manipulation system that leverages novel navigation and shape completion methods to manipulate an object with a mobile robot. Our system utilizes uncertainty in the initial estimation of a manipulation target to calculate a predicted next-best-view. Without the need of localization, the robot then uses the predicted panoramic view at the next-best-view location to navigate to the desired location, capture a second view of the object, create a new model that predicts the shape of object more accurately than a single image alone, and uses this model for grasp planning. We show that the system is highly effective for mobile manipulation tasks through simulation experiments using real world data, as well as ablations on each component of our system.

Keywords: Robot Safety, Manipulation Planning, Integrated Planning and Control
Abstract: Recent advances allow for the automation of food preparation in high-throughput environments, yet the successful deployment of these robots requires the planning and execution of quick, robust, and ultimately collision-free behaviors. In this work, we showcase a novel framework for modifying previously generated trajectories of robotic manipulators in highly detailed and dynamic collision environments using Control Barrier Functions (CBFs). This method dynamically re-plans previously validated behaviors in the presence of changing environments---and does so in a computationally efficient manner. Moreover, the approach provides rigorous safety guarantees of the resulting trajectories, factoring in the true underlying dynamics of the manipulator. This methodology is extensively validated on a full-scale robotic manipulator in a real-world cooking environment, and has resulted in substantial improvements in computation time and robustness over re-planning.

Keywords: Compliance and Impedance Control, Computational Geometry, Medical Robots and Systems
Abstract: We propose an approach to robotic ultrasound scanning systems using a passivity-based impedance control scheme on arbitrary surfaces. First, we introduce task coordinates depending on the geometry of the surface, which enables hands-on guidance of the robot along the surface, as well as teleoperated and autonomous ultrasound image acquisition. Our coordinates allow controlling the signed distance of the robot to the surface and alignment of the tool to the surface normal using classical impedance control. This corresponds to implicitly obtaining a foliation of parallel surfaces. By setting the desired signed distance negative, i.e. into the surface, we obtain passive contact forces. We extend the approach to also incorporate coordinates that allow controlling the specific point on the surface and, likewise, on all parallel surfaces. Finally, we demonstrate the performance of the controller on the seven degrees of freedom lightweight robot DLR MIRO. In the experiments the robot can track complex trajectories while keeping the distance error below 1mm and applying an almost constant contact force.

Keywords: Medical Robots and Systems, AI-Based Methods, Mechanism Design
Abstract: Medical percussion is a common manual examination procedure used by physicians to determine the state of underlying tissues from their acoustic responses. Although it has been used for centuries, there is a limited quantitative understanding of its dynamics, leading to subjectivity and a lack of detailed standardisation. This paper presents a novel compliant two-degree-of-freedom robotic device inspired by the human percussion action, and validates its performance in two tissue characterisation experiments. In Experiment 1, spectrotemporal analysis using 1-D Continuous Wavelet Transform (CWT) proved the potential of the device to identify hard nodules, mimicking lipomas, embedded in silicone phantoms representing a patient’s abdominal region. In Experiment 2, Gaussian Mixture Modelling (GMM) and Neural Network (NN) predictive models were implemented to classify composite phantom tissues of varying density and thickness. The proposed device and methods showed up to 97.5% accuracy in the classification of phantoms, proving the potential of robotic solutions to standardise and improve the accuracy of percussion diagnostic procedures.

Keywords: Climbing Robots, Field Robots, Robotics and Automation in Construction
Abstract: This paper presents a novel design of a multi-directional bicycle robot, which is developed for the inspection of steel structures, in particular, steel-reinforced bridges. The locomotion concept is based on arranging two magnetic wheels in a bicycle-like configuration with two independent steering actuators. This configuration allows the robot to possess multi-directional mobility. An additional free joint helps the robot adapt naturally to non-flat and complex steel structures. The robot’s design provides the advantage of being mechanically simple and providing high-level mobility across diverse steel structures. In addition, a visual sensor is equipped that allows the data collection for steel defect detection with offline training and validation. The paper also provides a novel pipeline for Steel Defect Detection, which utilizes multiple datasets (one for training and one for validation) from real bridges. The quantitative results have been reported for three Deep Encoder-Decoder Networks (i.e., LinkNet, UNet, DeepLab) with their corresponding Encoder modules (i.e., ResNet-18, ResNet-34, RegNet-X2, EfficientNet-B0, and EfficientNet-B2). Due to space concerns, the qualitative results have been outlined in Appendix, with a link in Fig. 11 caption to access the result provided.

Keywords: Computer Vision for Automation, Force and Tactile Sensing, Machine Learning for Robot Control
Abstract: An industrial connector insertion task requires submillimeter positioning and grasp pose compensation for a plug. Thus, highly accurate estimation of the relative pose between a plug and socket is fundamental for achieving the task. World models are promising technologies for visuomotor control because they obtain appropriate state representation to jointly optimize feature extraction and latent dynamics model. Recent studies show that the NewtonianVAE, a type of the world model, acquires latent space equivalent to mapping from images to physical coordinates. Proportional control can be achieved in the latent space of NewtonianVAE. However, applying NewtonianVAE to high-accuracy industrial tasks in physical environments is an open problem. Moreover, the existing framework does not consider the grasp pose compensation in the obtained latent space. In this work, we proposed tactile-sensitive NewtonianVAE and applied it to a USB connector insertion with grasp pose variation in the physical environments. We adopted a GelSight-type tactile sensor and estimated the insertion position compensated by the grasp pose of the plug. Our method trains the latent space in an end-to-end manner, and no additional engineering and annotation are required. Simple proportional control is available in the obtained latent space. Moreover, we showed that the original NewtonianVAE fails in some situations, and demonstrated that domain knowledge induction improves model accuracy. This domain knowledge can be easily obtained using robot specification and grasp pose error measurement. We demonstrated that our proposed method achieved a 100% success rate and 0.3 mm positioning accuracy in the USB connector insertion task in the physical environment. It outperformed SOTA CNN-based two-stage goal pose regression with grasp pose compensation using coordinate transformation.

Keywords: Grippers and Other End-Effectors, Compliant Joints and Mechanisms, Compliance and Impedance Control
Abstract: We propose a parallel massage robot with compliant joints based on the series elastic actuator (SEA), offering a unified force-position control approach. First, the kinematic and static force models are established for obtaining the corresponding control variables. Then, a novel force-position control strategy is proposed to separately control the force-position along the normal direction of the surface and another two-direction displacement, without the requirement of a robotic dynamics model. To evaluate its performance, we implement a series of robotic massage experiments. The results demonstrate that the proposed massage manipulator can successfully achieve desired forces and motion patterns of massage tasks, arriving at a high-score user experience.

Keywords: Grippers and Other End-Effectors, Underactuated Robots, Methods and Tools for Robot System Design
Abstract: In this paper we present a potential energy map based approach that provides a framework for the design and control of a robotic grasper. Unlike other potential energy map approaches, our framework considers friction for a more realistic perspective on grasper performance. Our analysis establishes the importance of considering dynamically variable geometry in grasper design, namely palm width, link lengths, and transmission ratios, which are assumed to be able to change in real-time. Our analysis assumes a two-phalanx tendon-pulley underactuated grasper, but it can be extended to other underactuated mechanisms. We demonstrate the utility of these novel potential energy maps and the method used to generate them in order by showing how various design parameters impact the grasping and in-hand manipulation performance of a particular design across a range of object sizes and friction coefficients. Optimal grasping designs have palms that scale with object size and transmission ratios that scale with the coefficient of friction. Using a custom in-hand manipulation metric, we compared the in-hand manipulation capabilities of a grasper that only dynamically varied its palm size, link lengths, and transmission ratios to a grasper with a variable palm and controllable actuation efforts. The analysis revealed the advantage of dynamically variable geometry; by varying only its palm size, link lengths, and transmission ratios in real-time, safe, caged in-hand manipulation of a wide range of objects could be performed.

Keywords: Grippers and Other End-Effectors, Human-Robot Collaboration, Safety in HRI
Abstract: While collaborative robotic arms offer significant safety benefits, safety of the overall manipulator system cannot be guaranteed unless equally strict safety requirements are satisfied by the accompanying end-effector. Current robot grippers are not made in a way that fulfills such a requirement, resulting in collaborative robots needing to operate in a protected environment. This paper presents a novel permanent magnet actuator inside of a conventional industrial electric gripper which results in an end-effector that has an unmatched force range of 1-2N to 43N and exhibits interesting characteristics suited to the requirements of a safe gripper such as torque holding without power, variable stiffness and force sensing.

Keywords: Deep Learning in Grasping and Manipulation, Perception for Grasping and Manipulation, Force and Tactile Sensing
Abstract: Research around tactile sensing for grasp stability prediction in robotic manipulators continues to be popular, however few works are able to achieve a high classification accuracy. Due to simulation complexity, data-driven methods are often forced to rely on experimental data, yielding small, often unbalanced, data sets. In this work, the authors use a 3972 sample data set to explore the effects of the data set composition on the performance of a classifier. While maintaining a similar overall accuracy, the ability to recognize a grasp failure was significantly impacted by the composition of the data set. The authors propose an autonomous pipeline designed to generate more diverse failure grasps. On failure-rich data, a tactile-based classifier with a balanced training set achieved a classification accuracy of 84.68% while maintaining a recall of the grasp failure class of 76%. This represents a 71.79% improvement in recall over a model trained on a larger but unbalanced data set.

Keywords: Grippers and Other End-Effectors, Dexterous Manipulation, Industrial Robots
Abstract: This paper explores a novel approach to dexterous manipulation, aimed at levels of speed, precision, robustness, and simplicity suitable for practical deployment. The enabling technology is a Direct-drive Hand (DDHand) comprising two fingers, two DOFs each, that exhibit high speed and a light touch. The test application is the dexterous manipulation of three small and irregular parts, moving them to a grasp suitable for a subsequent assembly operation, regardless of initial presentation. We employed four primitive behaviors that use ground contact as a “third finger”, prior to or during the grasp process: pushing, pivoting, toppling, and squeeze- grasping. In our experiments, each part was presented from 30 to 90 times randomly positioned in each stable pose. Success rates varied from 83% to 100%. The time to manipulate and grasp was 6.32 seconds on average, varying from 2.07 to 16 seconds. In some cases, performance was robust, precise, and fast enough for practical applications, but in other cases, pose uncertainty required time-consuming vision and arm motions. The paper concludes with a discussion of further improvements required to make the primitives robust, eliminate uncertainty, and reduce this dependence on vision and arm motion.

Keywords: Grippers and Other End-Effectors, Multifingered Hands, Force and Tactile Sensing
Abstract: Humans can use different grasping poses and forces for everyday objects of different shapes and sizes. Grasping and manipulating everyday objects have been longstanding challenges in robotics. Performing multiple grasping configurations is difficult for robotic end-effectors with limited degrees of freedom (DOF). Integrating tactile sensing into robotic grippers will facilitate grasping and manipulating a wider range of objects. In this letter, we present a robotic gripper with a reconfigurable mechanism and tactile sensors (GTac) integrated into the fingers and palm. Each finger consists of two phalanges with a 2 DOF underactuated design and a metacarpophalangeal (MCP) joint. Our gripper with four adaptive fingers can perform 5 grasping configurations and obtain 228 tactile feedback signals (normal and shear forces) at 150 Hz. Our results show that the gripper can grasp various everyday objects and achieve in-hand manipulation including translation and rotation with closed-loop control. In the YCB benchmark assessment, the gripper achieved a score of 93% (round objects), 0% (flat objects), 78% (tools), 90% (articulated objects), and 65% in total This research provides a new hardware design and could be beneficial to various robotic applications in the domestic and industrial fields.

Keywords: Grippers and Other End-Effectors, Mechanism Design, Grasping
Abstract: This paper proposes a novel single-fingered reconfigurable robotic gripper for grasping objects in narrow working spaces. The finger of the developed gripper realizes two configurations, namely, the insertion and grasping modes, using only a single motor. In the insertion mode, the finger assumes a thin shape such that it can insert its tip into a narrow space. The grasping mode of the finger is activated through a folding mechanism. Mode switching can be achieved in two ways: switching the mode actively by a motor, or combining passive rotation of the fingertip through contact with the support surface and active motorized construction of the claw. The latter approach is effective when it is unclear how much finger insertion is required for a specific task. The structure provides a simple control scheme. The performance of the proposed robotic gripper design and control methodology was experimentally evaluated. The minimum width of the insertion space required to grasp an object is 4 mm (1 mm, when using a strategy).

Keywords: Grippers and Other End-Effectors, Soft Robot Applications, Force Control
Abstract: We propose the framework of Series Elastic End Effectors in 6D (SEED), which combines a spatially compliant element with visuotactile sensing to grasp and manipulate tools in the wild. Our framework generalizes the benefits of series elasticity to 6-dof, while providing an abstraction of control using visuotactile sensing. We propose an algorithm for relative pose estimation from visuotactile sensing, and a spatial hybrid force-position controller capable of achieving stable force interaction with the environment. We demonstrate the effectiveness of our framework on tools that require regulation of spatial forces. Video link: https://youtu.be/2-YuIfspDrk

Keywords: Multifingered Hands, Underactuated Robots, Soft Sensors and Actuators
Abstract: The ability to grasp a wider range of objects in size and shape directly relates to the performance of robotic grippers. Adapting to complex geometries of objects requires large degrees of freedom to allow complex configurations. However, complexity in controlling many individual joints leads to introduction of underactuated mechanisms, in which traditional finger designs composed of revolute joints allow only flexion/extension motions. In this article, we propose a length-adjustable linkage mechanism in the underactuated finger controlled by an antagonistic tendon pair. The resulting gripper can elongate the fingers for an increased task space or shorten them for a finer spatial resolution. For tactile sensing, hyperelastic soft sensors are used to stretch with finger elongation. Contact pressures measured by the soft sensors are used in force-feedback control for which either the joint angles or the link lengths are adjusted. Lastly, a multimodal control scheme that combines elongation and flexion modes is demonstrated with tasks of dexterous manipulation.

Keywords: Sensor-based Control, Dexterous Manipulation, Perception for Grasping and Manipulation
Abstract: Deformable object manipulation (DOM) is an emerging research problem in robotics. The ability to manipulate deformable objects endows robots with higher autonomy and promises new applications in the industrial, services, and healthcare sectors. However, compared to rigid object manipulation, the manipulation of deformable objects is considerably more complex, and is still an open research problem. Addressing DOM challenges demand breakthroughs in almost all aspects of robotics, namely, hardware design, sensing, (deformation) modeling, planning, and control. In this article, we review recent advances and highlight the main challenges when considering deformation in each sub-field. A particular focus of our paper lies in the discussions of these challenges and proposing future directions of research.

Keywords: Manipulation Planning, Task and Motion Planning, Task Planning
Abstract: This paper develops a flexible and robust robotic system for autonomously drawing on 3D surfaces. The system takes 2D drawing strokes and a 3D target surface (mesh or point clouds) as input. It maps the 2D strokes onto the 3D surface and generates a robot motion to draw the mapped strokes using visual recognition, grasp pose reasoning, and motion planning. The system is flexible compared to conventional robotic drawing systems as we do not fix drawing tools to the end of a robot arm. Instead, a robot recognizes and picks up pens online and holds the pens to draw 3D strokes. Meanwhile, the system has high robustness thanks to the following crafts: First, a high-quality mapping method is developed to minimize deformation in the strokes. Second, visual detection is used to re-estimate the drawing tool's pose before executing each drawing motion. Third, force control is employed to compensate for noisy visual detection and calibration and ensure a firm touch between the pen tip and the surface. Fourth, error detection and recovery are implemented to deal with slippage and other anomalies. The planning and executions are performed in a closed-loop manner until the strokes are successfully drawn. We evaluate the system and analyze the necessity of the various crafts using different real-world tasks. The results show that the proposed system is flexible and robust to generate robotic motion that picks up the pens and successfully draws 3D strokes on given surfaces.

Keywords: Deep Learning in Grasping and Manipulation, Assembly, Deep Learning for Visual Perception
Abstract: Vision-based robotic assembly is a crucial yet challenging task as the interaction with multiple objects requires high levels of precision. In this paper, we propose an integrated 6D robotic system to perceive, grasp, manipulate and assemble blocks with tight tolerances. Aiming to provide an off-the-shelf RGB-only solution, our system is built upon a monocular 6D object pose estimation network trained solely with synthetic images leveraging physically-based rendering. Subsequently, pose-guided 6D transformation along with collision-free assembly is proposed to construct any designed structure with arbitrary initial poses. Our novel 3-axis calibration operation further enhances the precision and robustness by disentangling 6D pose estimation and robotic assembly. Both quantitative and qualitative results demonstrate the effectiveness of our proposed 6D robotic assembly system.

Keywords: Deep Learning in Grasping and Manipulation, Human-Robot Collaboration, Human-Aware Motion Planning
Abstract: This work explores how contextual information and human intention affect the motion prediction of humans during a handover operation with a social robot. By classifying human intention in four different classes, we developed a model able to generate a different motion for each intention class. Furthermore, the model uses a multi-headed attention architecture to add contextual information to the pipeline, such as the position of the robot end effector (REE) or the position of obstacles in the interaction scene. We generate predictions up to two and half seconds in the future given an input sequence of one second containing the previous motion of the human. The results show an improvement of the prediction accuracy, both for the full skeleton prediction and the human hand used for the delivery. The model also allows to generate different sequences with the desired human intention.

Keywords: Deep Learning in Grasping and Manipulation, Probability and Statistical Methods, In-Hand Manipulation
Abstract: We study the problem of estimating the pose of an object which is being manipulated by a multi-fingered robotic hand by only using proprioceptive feedback. To address this challenging problem, we propose a novel variant of differentiable particle filters, which combines two key extensions. First, our learned proposal distribution incorporates recent measurements in a way that mitigates weight degeneracy. Second, the particle update works on non-euclidean manifolds like Lie-groups, enabling learning-based pose estimation in 3D on SE(3). We show that the method can represent the rich and often multi-modal distributions over poses that arise in tactile state estimation. The models are trained in simulation, but by using domain randomization, we obtain state estimators that can be employed for pose estimation on a real robotic hand (equipped with joint torque sensors). Moreover, the estimator runs fast, allowing for online usage with update rates of more than 100Hz on a single CPU core. We quantitatively evaluate our method and benchmark it against other approaches in simulation. We also show qualitative experiments on the real torque-controlled DLR-Hand II.

Keywords: Deep Learning in Grasping and Manipulation, Multifingered Hands, Grasping
Abstract: In this work, we investigate the problem of planning stable grasps for object manipulations using an 18-DOF robotic hand with four fingers. The main challenge here is the high-dimensional search space, and we address this problem using a novel two-stage learning process. In the first stage, we train an autoregressive network called the hand-pose-generator, which learns to generate a distribution of valid 6D poses of the palm for a given volumetric object representation. In the second stage, we employ a network that regresses 12D finger positions and scalar grasp qualities from given object representations and palm poses. To train our networks, we use synthetic training data generated by a novel grasp planning algorithm, which also proceeds stage-wise: first the palm pose, then the finger positions. Here, we devise a Bayesian Optimization scheme for the palm pose and a physics-based grasp pose metric to rate stable grasps. In experiments on the YCB benchmark data set, we show a grasp success rate of over 83%, as well as qualitative results on real scenarios of grasping unknown objects.

Keywords: Deep Learning in Grasping and Manipulation, Reinforcement Learning, Transfer Learning
Abstract: Multi-goal policy learning for robotic manipulation is challenging. Prior successes have used state-based representations of the objects or provided demonstration data to facilitate learning. In this paper, by hand-coding a high-level discrete representation of the domain, we show that policies to reach dozens of goals can be learned with a single network using Q-learning from pixels. The agent focuses learning on simpler, local policies which are sequenced together by planning in the abstract space. We compare our method against standard multi-goal RL baselines, as well as other methods that leverage the discrete representation, on a challenging block construction domain. We find that our method can build more than a hundred different block structures, and demonstrate forward transfer to structures with novel objects. Lastly, we deploy the policy learned in simulation on a real robot.

Keywords: Deep Learning in Grasping and Manipulation, Deep Learning for Visual Perception, Telerobotics and Teleoperation
Abstract: Studies in robot teleoperation have been centered around action specifications -- from continuous joint control to discrete end-effector pose control. However, these "robot-centric" interfaces often require skilled operators with extensive robotics expertise. To make teleoperation accessible to non-expert users, we propose the framework "Scene Editing as Teleoperation" (SEaT), where the key idea is to transform the traditional "robot-centric" interface into a "scene-centric" interface -- instead of controlling the robot, users focus on specifying the task's goal by manipulating digital twins of the real-world objects. As a result, a user can perform teleoperation without any expert knowledge of the robot hardware. To achieve this goal, we utilize a category-agnostic scene-completion algorithm that translates the real-world workspace (with unknown objects) into a manipulable virtual scene representation and an action-snapping algorithm that refines the user input before generating the robot's action plan. To train the algorithms, we procedurely generated a large-scale, diverse kit-assembly dataset that contains object-kit pairs that mimic real-world object-kitting tasks. Our experiments in simulation and on a real-world system demonstrate that our framework improves both the efficiency and success rate for 6DoF kit-assembly tasks. A user study demonstrates that SEaT framework participants achieve a higher task success rate and report a lower subjective workload compared to an alternative robot-centric interface.

Keywords: Deep Learning in Grasping and Manipulation, Grasping
Abstract: Automated grasping of arbitrary objects is an essential skill for many applications such as smart manufacturing and human robot interaction. This makes grasp detection a vital skill for automated robotic systems. Recent work in model-free grasp detection uses point cloud data as input and typically outperforms the earlier work on RGB(D)-based methods. We show that RGB(D)-based methods are being underestimated due to suboptimal label encodings used for training. Using the evaluation pipeline of the GraspNet-1Billion dataset, we investigate different encodings and propose a novel encoding that significantly improves grasp detection on depth images. Additionally, we show shortcomings of the 2D rectangle grasps supplied by the GraspNet-1Billion dataset and propose a filtering scheme by which the ground truth labels can be improved significantly. Furthermore, we apply established methods for uncertainty estimation on our trained models since knowing when we can trust the model's decisions provides an advantage for real-world application. By doing so, we are the first to directly estimate uncertainties of detected grasps. We also investigate the applicability of the estimated aleatoric and epistemic uncertainties based on their theoretical properties. Additionally, we demonstrate the correlation between estimated uncertainties and grasp quality, thus improving selection of high quality grasp detections. By all these modifications, our approach using only depth images can compete with point-cloud-based approaches for grasp detection despite the lower degree of freedom for grasp poses in 2D image space.

Keywords: Autonomous Vehicle Navigation, Motion and Path Planning, Service Robotics
Abstract: This paper presents a unified approach to the design of Model Predictive Controllers (MPC), custom-tailored for path following by Automated Guided Vehicles (AGVs). The approach can be applied in a unified manner to several relevant AGV kinematic configurations, including tricycle, differential, and double steer-drive. By leveraging Linear Parameter Varying (LPV) MPC, it provides maximum maneuverability and industrial-grade positioning accuracy. We incorporate state-of-the-art optimized velocity planning, to maximize vehicle utilization. Experimental validation is performed on three different kinematic configurations, including a real forklift with tricycle configuration, using industrially-relevant positioning maneuvers.

Keywords: Autonomous Vehicle Navigation, Motion and Path Planning, Control Architectures and Programming
Abstract: Collision avoidance in emergency situations is a crucial and challenging task in motion planning for autonomous vehicles. Especially in the field of optimization-based planning using nonlinear model predictive control, many efforts to achieve real-time performance are still ongoing. Among various approaches, the iterative linear quadratic regulator (iLQR) is known as an efficient means of nonlinear optimization. Additionally, parallel computing architectures, such as GPUs, are more widely applied in autonomous vehicles. In this paper, we propose 1) a parallel computing framework for iLQR with input constraints considering the characteristics of the problem and 2) a proper environmental formulation that is suitable for lower-precision GPU computations. The GPU-accelerated framework was tested on a real-time simulation-in-the-loop system using CarMaker and ROS at a 20 Hz sampling rate on a low-performance mobile computer and was compared against the same framework realized with a CPU.

Keywords: Autonomous Vehicle Navigation, Imitation Learning, Sensor Fusion
Abstract: In this paper, we present a novel autonomous driving framework, called a road graph and image attention network (RIANet), which computes the attention scores of objects in the image using the road graph feature. The process of the proposed method is as follows: First, the feature encoder module encodes the road graph, image, and additional features of the scene. The attention network module then incorporates the encoded features and computes the scene context feature via the attention mechanism. Finally, the low-level controller module drives the ego-vehicle based on the scene context feature. In the experiments, we use an urban scene driving simulator named CARLA to train and test the proposed method. The results show that the proposed method outperforms existing autonomous driving methods.

Keywords: Computer Vision for Transportation, Intelligent Transportation Systems, Object Detection, Segmentation and Categorization
Abstract: Camera-LiDAR fusion provides precise distance measurements and fine-grained textures, making it a promising option for 3D vehicle detection in autonomous driving scenarios. Previous camera-LiDAR based 3D vehicle detection approaches mainly focused on employing image-based pre-trained models to fetch semantic features. However, these methods may perform inferior to the LiDAR-based ones when lacking semantic segmentation labels in autonomous driving tasks. Motivated by this observation, we propose a novel semantic augmentation method, namely Sem-Aug, to guide high-confidence camera-LiDAR fusion feature generation and boost the performance of multimodal 3D vehicle detection. The key novelty of semantic augmentation lies in the 2D segmentation mask auto-labeling, which provides supervision for semantic segmentation sub-network to mitigate the poor generalization performance of camera-LiDAR fusion. Using semantic-augmentation-guided camera-LiDAR fusion features, Sem-Aug achieves remarkable performance on the representative autonomous driving KITTI dataset compared to both the LiDAR-based baseline and previous multimodal 3D vehicle detectors. Qualitative and quantitative experiments demonstrate that Sem-Aug provides significant improvements in challenging Hard detection scenarios caused by occlusion and truncation.

Keywords: Autonomous Vehicle Navigation, Planning under Uncertainty, Motion and Path Planning
Abstract: In the narrow lane scene of autonomous driving, it is critical for the ego car to recognize the intentions of social vehicles and cooperate with them. However, cooperating with social vehicles is challenging due to insufficient information. This paper proposes an Explorative Game that adopts Participant Game and Perfect Bayesian Equilibrium to exploratively perform some aggressive actions to obtain additional information, thus the autonomous vehicle can cooperate robustly and efficiently. Explorative Game assumes each vehicle maintains a unique belief about the current situation and attributes insecurity and instability to the conflict of various Perfect Bayesian Equilibriums formed by various beliefs. Aggressive actions enable the ego car to proactively guide social vehicles to cooperate as it expects and encourage them to express their intentions as quickly and clearly as possible so that the equilibriums can converge and the conflict can be eliminated. Additional information reduces the error between the actual intentions of social vehicles and the estimated intentions from the ego car, helping rationally prune potential interactions and update parameters of the reward function. We demonstrate our algorithm on recorded data as well as virtual environments with manually controlled social vehicles to prove the efficiency of cooperation and the robustness of decision-making. And it has been running for more than 20 kilometers in the real world.

Keywords: Autonomous Vehicle Navigation, Mapping, Localization
Abstract: High-definition (HD) map is crucial for intelligent vehicles to perform high-level localization and navigation. To improve the availability and usability of HD map, it is meaningful to investigate crowd-sourced mapping solutions and low-cost map-aided localization schemes which don't rely on high-end sensors. In this paper, we propose a novel vision-based mapping and localization system, which could generate compact instance-level road maps automatically and provide high-availability map-aided localization. The spatial uncertainties of the map elements are taken into consideration by analyzing the inverse perspective mapping (IPM) model, which enables more flexible map usages in both mapping and localization phases of the system. Besides, a pose graph optimization framework is developed for accurate pose estimation by fusing global positioning (GNSS), local navigation (odometry) and map matching information together. Real-world experiments in urban environment were conducted to validate different phases of the system, including on-vehicle mapping, multi-source map merging and map-aided localization.

Keywords: Autonomous Vehicle Navigation, Motion and Path Planning, Optimization and Optimal Control
Abstract: In this study, a topology-guided unified nonlinear model predictive control (NMPC) approach is proposed for autonomous navigation of a class of Hybrid Terrestrial and Aerial Quadrotors (HyTAQs) in unknown environments. The approach can fully exploit the hybrid terrestrial-aerial locomotion of the vehicle and as such ensure a high navigation efficiency. A unified terrestrial-aerial NMPC is first formulated with a type of complementarity constraints involving the hybrid dynamics, together with the collision avoidance constraints for safety. Further, a topological roadmap with both terrestrial and aerial paths is leveraged to guide the kinodynamic path searching and thus the unified NMPC. Then, a complete and distinctive navigation framework is established and validated on our self-developed HyTAQ. Compared with the existing unified terrestrial-aerial planning methods, ours takes the vehicle dynamics into account for the first attempt and achieves a more reasonable decision of modes switching. Experimental results are presented to demonstrate the effectiveness and superiority of the proposed approach.

Keywords: Software Tools for Benchmarking and Reproducibility, Social HRI, Modeling and Simulating Humans
Abstract: We present SEAN 2.0, an open-source system designed to advance social navigation via the training and benchmarking of navigation policies in varied social contexts. A key limitation of current social navigation research is that policies are often trained and evaluated considering only a few social contexts, which are fragmented across prior work. Inspired by work in psychology, we describe navigation context based on social situations, which encompass the robot task and environmental factors, and propose logic-based classifiers for five common examples. SEAN 2.0 allows a robot to experience these social situations via different methods for specifying and generating pedestrian motion, including a novel Behavior Graph method. Our experiments show that when data collected using the Behavior Graph method is used to learn a robot navigation policy, that policy outperforms others trained using alternative methods for pedestrian control. Also, social situations were found to be useful for understanding performance across social contexts. Other components of SEAN 2.0 include vision and depth sensors, several physical environments, different means of specifying robot tasks, and a range of evaluation metrics for social robot navigation. User feedback for SEAN 2.0 indicated that the system was "easier to navigate and more user friendly" than our prior work, SEAN 1.0.

Keywords: SLAM, Range Sensing
Abstract: This paper presents a method for the robust selection of measurements in a simultaneous localization and mapping (SLAM) framework. Existing methods check consistency or compatibility on a pairwise basis, however many measurement types are not sufficiently constrained in a pairwise scenario to determine if either measurement is inconsistent with the other. This paper presents group-k consistency maximization (GkCM) that estimates the largest set of measurements that is internally group-k consistent. Solving for the largest set of group-k consistent measurements can be formulated as an instance of the maximum clique problem on generalized graphs and can be solved by adapting current methods. This paper evaluates the performance of GkCM using simulated data and compares it to pairwise consistency maximization (PCM) presented in previous work.

Keywords: SLAM, Mapping, Semantic Scene Understanding
Abstract: Processing large indoor scenes is a challenging task, as scan registration and camera trajectory estimation methods accumulate errors across time. As a result, the quality of reconstructed scans is insufficient for some applications, such as visual-based localization and navigation, where the correct position of walls is crucial.

For many indoor scenes, there exists an image of a technical floorplan that contains information about the geometry and main structural elements of the scene, such as walls, partitions, and doors. We argue that such a floorplan is a useful source of spatial information, which can guide a 3D model optimization.

The standard RGB-D 3D reconstruction pipeline consists of a tracking module applied to an RGB-D sequence and a bundle adjustment (BA) module that takes the posed RGB-D sequence and corrects the camera poses to improve consistency. We propose a novel optimization algorithm expanding conventional BA that leverages the prior knowledge about the scene structure in the form of a floorplan. Our experiments on the Redwood dataset and our self-captured data demonstrate that utilizing floorplan improves accuracy of 3D reconstructions.

Keywords: SLAM, Visual Tracking, Deep Learning Methods
Abstract: Visual SLAM systems targeting static scenes have been developed with satisfactory accuracy and robustness. Dynamic 3D object tracking has then become a significant capability in visual SLAM with the requirement of understanding dynamic surroundings in various scenarios including autonomous driving, augmented and virtual reality. However, performing dynamic SLAM solely with monocular images remains a challenging problem due to the difficulty of associating dynamic features and estimating their positions. In this paper, we present MOTSLAM, a dynamic visual SLAM system with the monocular configuration that tracks both poses and bounding boxes of dynamic objects. MOTSLAM first performs multiple object tracking (MOT) with associated both 2D and 3D bounding box detection to create initial 3D objects. Then, neural-network-based monocular depth estimation is applied to fetch the depth of dynamic features. Finally, camera poses, object poses, and both static, as well as dynamic map points, are jointly optimized using a novel bundle adjustment. Our experiments on the KITTI dataset demonstrate that our system has reached best performance on both camera ego-motion and object tracking on monocular dynamic SLAM.

Keywords: SLAM, Sensor Fusion, Range Sensing
Abstract: Visual and lidar Simultaneous Localization and Mapping (SLAM) algorithms benefit from the Inertial Measurement Unit (IMU) modality. The high-rate inertial data complement the other lower-rate modalities. Moreover, in the absence of constant acceleration, the gravity vector makes two attitude angles out of three observable in the global coordinate frame. In visual odometry, this is already being used to reduce the 6-Degrees Of Freedom (DOF) pose estimation problem to 4-DOF. In lidar SLAM, the gravity measurements are often used as a penalty in the back-end global map optimization to prevent map deformations. In this work, we propose an Iterative Closest Point (ICP)-based front-end which exploits the observable DOF and provides pose estimates aligned with the gravity vector. We believe that this front-end has the potential to support the loop closure identification, thus speeding up convergences of global map optimizations. The presented approach has been extensively tested against accurate ground-truth localization in large-scale outdoor environments as well as in the Subterranean Challenge organized by Defense Advanced Research Projects Agency (DARPA). We show that it can reduce the localization drift by 30% when compared to the standard 6-DOF ICP. Moreover, the code is readily available to the community as a part of the libpointmatcher library.

Keywords: SLAM, Mapping, Localization
Abstract: Lidar-based SLAM systems perform well in a wide range of circumstances by relying on the geometry of the environment. However, even mature and reliable approaches struggle when the environment contains structureless areas such as long hallways. To allow the use of lidar-based SLAM in such environments, we propose to add reflector markers in specific locations that would otherwise be difficult. We present an algorithm to reliably detect these markers and two approaches to fuse the detected markers with geometry-based scan matching. The performance of the proposed methods is demonstrated on real-world datasets from several industrial environments.

Keywords: SLAM, Sensor Fusion, Data Sets for SLAM
Abstract: LiDAR odometry has attracted considerable attention as a robust localization method for autonomous robots operating in complex GNSS-denied environments. However, achieving reliable and efficient performance on heterogeneous platforms in large-scale environments remains an open challenge due to the limitations of onboard computation and memory resources needed for autonomous operation. In this work, we present LOCUS 2.0, a robust and computationally-efficient LiDAR odometry system for real-time underground 3D mapping. LOCUS 2.0 includes a novel normals-based GICP formulation that reduces the computation time of point cloud alignment, an adaptive voxelization strategy that maintains the desired computation load regardless of the environment’s geometry, and a sliding-window map approach that bounds the memory consumption and management. The proposed approach is shown to be suitable to be deployed on heterogeneous robotic platforms involved in large-scale explorations under severe computation and memory constraints. We demonstrate LOCUS 2.0, a key element of the CoSTAR team’s entry in the DARPA Subterranean Challenge, across various underground scenarios.

We release LOCUS 2.0 as an open-source library and also release a LiDAR-based odometry dataset in challenging and largescale underground environments. The dataset features a legged and wheeled platform in multiple environments including fog, dust, darkness, and geometrically degenerate surroundings with a total of 11 h of operations and 16 km of distance traveled.

Keywords: SLAM, Sensor Fusion
Abstract: Research in Simultaneous Localization and Mapping (SLAM) has made outstanding progress over the past years. SLAM systems are nowadays transitioning from academic to real world applications. However, this transition has posed new demanding challenges in terms of accuracy and robustness. To develop new SLAM systems that can address these challenges, new datasets containing cutting-edge hardware and realistic scenarios are required. We propose the Hilti SLAM Challenge Dataset. Our dataset contains indoor sequences of offices, labs, and construction environments and outdoor sequences of construction sites and parking areas. All these sequences are characterized by featureless areas and varying illumination conditions that are typical in real-world scenarios and pose great challenges to SLAM algorithms that have been developed in confined lab environments. Accurate sparse ground truth, at millimeter level, is provided for each sequence. The sensor platform used to record the data includes a number of visual, lidar, and inertial sensors, which are spatially and temporally calibrated. The purpose of this dataset is to foster the research in sensor fusion to develop SLAM algorithms that can be deployed in tasks where high accuracy and robustness are required, e.g., in construction environments. Many academic and industrial groups tested their SLAM systems on the proposed dataset in the Hilti SLAM Challenge. The results of the challenge, which are summarized in this paper, show that the proposed dataset is an important asset in the development of new SLAM algorithms that are ready to be deployed in the real-world.

Keywords: SLAM, Computer Vision for Medical Robotics
Abstract: Visual SLAM inside the human body will open the way to computer-assisted navigation in endoscopy. However, due to space limitations, medical endoscopes only provide monocular images, leading to systems lacking true scale. In this paper we exploit the controlled lighting in colonoscopy to achieve the first in-vivo 3D reconstruction of the human colon using photometric stereo on a calibrated monocular endoscope. Our method works in a real medical environment, providing both a suitable in-place calibration procedure and a depth estimation technique adapted to the colon's tubular geometry. We validate our method on simulated colonoscopies, obtaining a mean error of 7% in depth estimation, which is below 3 mm on average. Our qualitative results on the EndoMapper dataset show that the method is able to correctly estimate the colon shape in real human colonoscopies, paving the ground for true-scale monocular SLAM in endoscopy.

Keywords: SLAM, Sensor Fusion
Abstract: The emerging paradigm of Continuous-Time Simultaneous Localization And Mapping (CTSLAM) has become a competitive alternative to conventional discrete-time approaches in recent times and holds the additional promise of fusing multi-modal sensor setups in a truly generic manner, rendering its importance to robotic navigation and manipulation seminal. In this spirit, this work expands upon continuous-time concepts, evaluates their suitability in common stereo and stereo-inertial online configurations, and provides an extensible, generic, robust, and modular open-source implementation to the community. The presented experimental analysis records the performance of our approach in these setups against the state-of-the-art in discrete-time Simultaneous Localization And Mapping (SLAM) on established datasets, achieving competitive results, and provides a direct comparison between online discrete- and continuous-time approaches for the first time. Targeting the absence of open-sourced, continuous-time pipelines and their associated, oftentimes prohibitive, initial developmental overhead, our implementation is made public.

Keywords: Virtual Reality and Interfaces, Telerobotics and Teleoperation, Human-Centered Robotics
Abstract: The recent momentum in aerial manipulation has led to an interest in developing virtual reality interfaces for aerial physical interaction tasks with simple, intuitive, and reliable control and perception. However, this requires the use of expensive subsystems and there is still a research gap between interface design, user evaluations and the effect on aerial manipulation tasks. Here, we present a methodology for low-cost available drone systems with a Unity-based interface for immersive FPV teleoperation. We applied our approach in a flight track where a cluttered environment is used to simulate a demanding aerial manipulation task inspired by forestry drones and canopy sampling. Through objective measures of teleoperation performance and subjective questionnaires, we found that operators performed worse using the FPV interface and had higher perceived levels of cognitive load when compared to traditional interface design. Additional analysis of physiological measures highlighted that objective stress levels and cognitive load were also influenced by task duration and perceived performance, providing an insight into what interfaces could target to support teleoperator requirements during aerial manipulation tasks.

Keywords: Virtual Reality and Interfaces, Telerobotics and Teleoperation, Physical Human-Robot Interaction
Abstract: This paper presents an easy-to-deploy, virtual reality-based teleoperation system for controlling a robot arm. The proposed system is based on a consumer-grade virtual reality device (Oculus Quest 2) with a low-cost robot arm (a LoCoBot) to allow easy replication and set up. The proposed Work-from-Home Virtual Reality (WFH-VR) system allows the user to feel an intimate connection with the real remote robot arm. Virtual representations of the robot and objects to be manipulated in the real-world are presented in VR by streaming data pertaining to orientation and poses. The user studies suggest that 1) the proposed telerobotic system is effective under conditions both with and without network latency, whereas a method that simply streams video does not. This design enables the system implemented at an arbitrary distance from the actual work site. 2) The proposed system allows novices to perform manipulation tasks requiring higher dexterity than traditional keyboard controls can support, such as setting tableware. All results, hardware settings, and questionnaire feedback can be obtained at https://arg-nctu.github.io/projects/vr-robot-arm.html.

Keywords: Virtual Reality and Interfaces, Sensor Fusion, AI-Based Methods
Abstract: Most virtual reality (VR) applications use a commercial controller for interaction. However, a typical virtual reality controller (VRC) lacks positional precision and accuracy in millimeter-scale scenarios. This lack of precision and accuracy is caused by built-in sensors’ drift. Therefore, the tracking performance of a VRC needs to be enhanced for millimeter-scale scenarios. Herein, we introduce a novel way of enhancing the tracking performance of a commercial VRC in a millimeter-scale environment using a deep learning (DL) algorithm. Specifically, we use a long short-term memory (LSTM) model trained with data collected from a linear motor, an IMU sensor, and a VRC. We integrate the virtual environment developed in Unity software with the LSTM model running in Python. We designed three experimental conditions: the VRC, Kalman filter (KF), and LSTM modes. Furthermore, we evaluate tracking performances in the three conditions and two other experimental scenarios, namely stationary and dynamic. In the stationary experimental scenario, the system is left motionless for 10 s. By contrast, in the dynamic experimental scenarios, the linear stage moves the system by 12 mm along the X, Y, and Z axes. The experimental results indicate that the deep learning model outperforms the standard controller’s positional performance by 85.69 % and 92.14 % in static and dynamic situations, respectively.

Keywords: Virtual Reality and Interfaces, Sensor Fusion
Abstract: Recently, various types of hand shape recognition systems have been developed for human-machine interfaces. However, most wearable recognition systems cannot robustly handle the variations in attachment positions of the devices. Thus, we propose a hand shape recognition system using a wearable multi-joint wrist contour measuring device to realize robust and effective recognition of hand shape, regardless of attachment position variations. In particular, this device can measure the wrist contour and band flexion data to recognize hand shape. The wrist contour data are measured using photo-reflectors mounted inside the device, and the band flexion data are measured using photo-interrupters installed at the device joints. Additionally, the attachment position information is extracted from the wrist contour or band flexion data using dimensional compression or attachment position recognition to achieve robustness against the position variations. Subsequently, the extracted information is incorporated into the hand shape recognizer. The results of the recognition experiment demonstrate that the attachment position information extracted from the band flexion data using dimensional compression could effectively realize robust hand shape recognition, considering variations in the device attachment position.

Keywords: Motion Control, Medical Robots and Systems
Abstract: In this paper, we propose a novel torque controller for the implementation virtual remote center of motion. The controller allows the system to implement the required behavior and guarantees the satisfaction of the remote center of motion constraint. Exploiting the Udwadia-Kalaba equation for constrained dynamic systems, the controller is synthesized considering the dynamic effect the constraint produces on the manipulator, achieving more effective control with respect to kinematic strategies, and allowing the implementation of compliance behaviors. Simulations and experimental validation with a KUKA LWR 4+ with 7 degrees of freedom has been performed to check the performances of the proposed controller. Results show the effectiveness of the proposed controller with different control action, and the capability to interact with the environment by implementing compliant motion control.

Keywords: Virtual Reality and Interfaces, Human Performance Augmentation, Computer Vision for Medical Robotics
Abstract: Augmented reality (AR) has the potential to improve the immersion and efficiency of computer-assisted orthopaedic surgery (CAOS) by allowing surgeons to maintain focus on the operating site rather than external displays in the operating theatre. Successful deployment of AR to CAOS requires a calibration that can accurately calculate the spatial relationship between real and holographic objects. Several studies attempt this calibration through manual alignment or with additional fiducial markers in the surgical scene. We propose a calibration system that offers a direct method for the calibration of AR head-mounted displays (HMDs) with CAOS systems, by using infrared-reflective marker-arrays widely used in CAOS. In our fast, user-agnostic setup, a HoloLens 2 detected the pose of marker arrays using infrared response and time-of-flight depth obtained through sensors onboard the HMD. Registration with a commercially available CAOS system was achieved when an IR marker-array was visible to both devices. Study tests found relative-tracking mean errors of 2.03 mm and 1.12 degrees when calculating the relative pose between two static marker-arrays at short ranges. When using the calibration result to provide in-situ holographic guidance for a simulated wire-insertion task, a pre-clinical test reported mean errors of 2.07 mm and 1.54 degrees when compared to a pre-planned trajectory.

Keywords: Virtual Reality and Interfaces, Human Detection and Tracking, Touch in HRI
Abstract: Freehand gesture based interaction promises to enable rich interaction in applications such as augmented reality (AR), virtual reality (VR), Human-Robot Interaction (HRI), and Robotic Prosthetic devices. However, current sensing approaches are limited; camera-based solutions are constrained by optical occlusion, and devices that interpret muscle activity are unreliable. This work presents a novel wrist-worn sensing device that combines near-infrared (NIR) sensing through 20 active LED-photodiode pairs and 6 DOF inertial measurement unit (IMU) sensing to enable high-accuracy detection of surface touch and grasp interactions for applications in AR and robotic prosthetic devices. Two convolutional neural networks are used to map device inputs to detect touch events, and subsequently classify them by gesture type and direction. We evaluate the accuracy and temporal precision of our system for event detection and classification. Results from an in-lab user study of 12 participants show an average of 97% touch detection accuracy and 98% grasp detection accuracy.

Keywords: Virtual Reality and Interfaces, Force and Tactile Sensing, Design and Human Factors
Abstract: Research attention on natural user interfaces (NUIs) for drone flights are rising. Nevertheless, NUIs are highly diversified, and primarily evaluated by different physical environments leading to hard-to-compare performance between such solutions. We propose a virtual environment, namely VRFlightSim, enabling comparative evaluations with enriched drone flight details to address this issue. We first replicated a state-of-the-art (SOTA) interface and designed two tasks (crossing and pointing) in our virtual environment. Then, two user studies with 13 participants demonstrate the necessity of VRFlightSim and further highlight the potential of open-data interface designs.

Keywords: Education Robotics, Physical Human-Robot Interaction, Design and Human Factors
Abstract: This paper presents a customized programmable robotic system, TanTwin (Tangible Twin), designed to promote STEM education for K-12 children. Firstly, TanTwin is implemented based on a wheel-robot with standard LEGO bricks. With several deep neural networks, a child can convert a captured portrait of himself/herself into standard LEGO bricks, therefore he/she can build a tangible twin robot of himself/herself automatically. Besides, to adapt to the customized appearance, the corresponding visual element and content of the robotic system were also changed by a rule-based adaption algorithm. To demonstrate the effectiveness of TanTwin and to investigate whether tangible twin robots could contribute to children’s learning, we conducted a controlled experimental study to compare learning with a TanTwin and with a standard robot system through measuring students’ cognitive learning outcomes. The pre-/post- knowledge test results indicated that learning with a tangible twin robot leads to significantly better learning outcomes. Given the results, we validate our system and customization technology can promote STEM education.

Keywords: Tendon/Wire Mechanism, Sensor Fusion, Parallel Robots
Abstract: Suspended Cable-Driven Parallel Robots (SCDPR) have intriguing capabilities on large scales but still have open challenges in precisely estimating the end-effector pose. The cables exhibit a downward curved shape, also known as cable sag which needs to be accounted for in the pose estimation. The catenary equations can accurately describe this phenomenon but are only accurate in equilibrium conditions. Thus, pose estimation for large-scale SCDPR in dynamic motion is an open challenge. This work proposes a real-time pose estimation algorithm for dynamic trajectories of SCDPRs, which is accurate over large areas. We present a novel approach that considers cable sag to reduce the estimation error for large scales while also employing an Inertial Measurement Unit (IMU) to improve estimation accuracy for dynamic motion. Our approach reduces the RMSE to less than a third compared to standard methods not considering cable sag. Similarly, the inclusion of the IMU reduces the RMSE in dynamic situations by 40% compared to non-IMU aided approaches considering cable sag. Furthermore, we evaluate our Extended Kalman Filter (EKF) based algorithm on a real system with ground truth pose information.

Keywords: Tendon/Wire Mechanism, Redundant Robots, Compliance and Impedance Control
Abstract: This study focuses on replicating the musculoskeletal system of human arms for mimicking its movement. Muscle redundancy is critical for regulating the mechanical impedance of arms and legs. However, when implementing muscle redundancy on robots, making an ill-posed problem that cannot determine the muscle forces uniquely. In this paper, first, a method for controlling end-point stiffness in the muscle space for the joint and muscle redundant system is described. Next, the muscle model imitating the nonlinear viscosity characteristic of human muscles is introduced. Then, a method to control the joint viscosity by adjusting the internal forces of muscles adequately without affecting the stiffness control directly is proposed. Finally, numerical simulations are performed to investigate the effectiveness of the proposed method.

Keywords: Tendon/Wire Mechanism, Parallel Robots
Abstract: Cable-Driven Parallel Robots (CDPRs) have been proposed for a variety of applications such as material handling, rehabilitation, and instrumentation. However, the collision-free constraint of CDPRs limits the workspace of CDPRs and the feasible position of anchor points. To address the collision-free constraint of CDPRs, a data-driven kinematic control scheme is developed for CDPRs, enabling a CDPR to control its pose even if suffering collisions between a cable and the base or the end-effector. To deal with the collisions, the data-driven kinematic control scheme utilizes a motion model obtained based on data samples of the motion of the CDPR, rather than the Jacobian matrix of the CDPR, to map a control law in the task space to the time derivative of the length of cables in the joint space. To evaluate the effectiveness of the developed data-driven kinematic control scheme, experiments of controlling a suspended CDPR with two cables allowing collisions are conducted.

Keywords: Tendon/Wire Mechanism, Mechanism Design, Actuation and Joint Mechanisms
Abstract: Actuators that apply tension forces are widely applicable in robotics. In many applications of tensile actuators, a large stroke length, high force, and small, light device are important. For these requirements, the best current solution is a winch, which uses a rotating shaft to pull lightweight cable. However, most winches accumulate cable in a spool on their shaft which limits maximum stroke length and force at a miniature scale. An alternative is a capstan winch, in which the cable wraps around the shaft in a single-layered spiral before passing off the shaft. Although high-force and high- stroke versions exist, miniaturization has not been successfully demonstrated. We present the design, modeling, and characterization of a miniaturized capstan winch. The 16 g winch is capable of lifting 4.5 kg (280x body weight) a distance of 4.3 m (67x body length). We also demonstrate it actuating a jumping robot and pulling a remote-controlled car out of a ditch. Through its miniature design and high-force, high-stroke performance, our winch expands the potential capabilities of small-scale robots.

Keywords: Tendon/Wire Mechanism, Legged Robots, Mechanism Design
Abstract: Legged robots with high locomotive performance have been extensively studied, and various leg structures have been proposed. Especially, a leg structure that can achieve both continuous and high jumps is advantageous for moving around in a three-dimensional environment. In this study, we propose a parallel wire-driven leg structure, which has one DoF of linear motion and two DoFs of rotation and is controlled by six wires, as a structure that can achieve both continuous jumping and high jumping. The proposed structure can simultaneously achieve high controllability on each DoF, long acceleration distance and high power required for jumping. In order to verify the jumping performance of the parallel wire-driven leg structure, we have developed a parallel wire-driven monopedal robot, RAMIEL. RAMIEL is equipped with quasi-direct drive, high power wire winding mechanisms and a lightweight leg, and can achieve a maximum jumping height of 1.6 m and a maximum of seven continuous jumps.

Keywords: Optimization and Optimal Control, Motion Control, Tendon/Wire Mechanism
Abstract: The control of cable-driven robots is challenging due to the system’s non-linearity, actuation redundancy and the unilaterally bounded actuation constraints. To solve this problem, a workspace-based model predictive control (W-MPC) scheme is proposed which combines the online model predictive control with offline workspace analysis. Using the workspace, a set of convex constraints can be generated for a given reference trajectory. This can then be used to formulate a convex optimization problem for the online W-MPC. Meanwhile strict recursive feasibility and stability are obtained by taking advantage of the predictive feature of MPC. To demonstrate the effectiveness of the proposed W-MPC, simulation was performed on a 2-link planar cabled-riven robot and a spatial cable-driven parallel robot for both nominal and non-nominal scenarios. Hardware experiment was also carried out using a 3 degree-of-freedom planar cable robot. The results show that the controller is efficient and effective to motion tracking with the cable force constraints satisfied despite the existence of various model uncertainties.

Keywords: Tendon/Wire Mechanism, Flexible Robotics, Medical Robots and Systems
Abstract: Flexible robots with in-situ torsion can be used in laryngeal endoscopic surgery which can maintain the position and approach vector of the end-effector during the operation. However, the inherent errors would be produced by in-situ torsional motion which are different due to the various configuration of serpentine module in robot. In this paper, the kinematics model is established according to the structure of serpentine module. The dexterity analysis shows that the singular position is reduced and the angular velocity of dexterity is improved comparing with the robot without in-situ torsion function. The theoretical position errors caused by in-situ torsion is quantitatively analyzed by simulation. It is found that the maximum error is 5.19mm at the bending angle of 120°.In addition, the existence of joints in the robot arm also leads to the occurrence of rotation errors. The configuration and number of the joints are optimized to improve the accuracy. Finally, the experiments are carried out to verify the effectiveness of the proposed design and model. The results indicate that the flexible surgical robot have higher motion dexterity. And the inherent error during the in-situ torsion motion can be eliminated by structural optimization.

Keywords: Planning under Uncertainty, Failure Detection and Recovery, Aerial Systems: Applications
Abstract: Changes in model dynamics due to factors like actuator faults, platform aging, and unexpected disturbances can challenge an autonomous robot during real-world operations affecting its intended behavior and safety. Under such circumstances, it becomes critical to improve tracking performance, predict future states of the system, and replan to maintain safety and liveness conditions. In this letter, we propose a meta-learning-based framework to learn a model to predict the future system's states and their uncertainties under unforeseen and untrained conditions. Meta-learning is considered for this problem thanks to its ability to easily adapt to new tasks with a few data points gathered at runtime. We use the predictions from the meta-learned model to detect unsafe situations and proactively replan the system's trajectory when an unsafe situation is detected (e.g., a collision with an object). The proposed framework is applied and validated with both simulations and experiments on a faulty UAV performing an infrastructure inspection mission, demonstrating safety improvements.

Keywords: Planning under Uncertainty, Motion and Path Planning, Mobile Manipulation
Abstract: Robots often need to solve path planning problems where essential and discrete aspects of the environment are partially observable. This introduces a multi-modality, where the robot must be able to observe and infer the state of its environment. To tackle this problem, we introduce the Path- Tree Optimization (PTO) algorithm which plans a path-tree in belief-space. A path-tree is a tree-like motion with branching points where the robot receives an observation leading to a belief-state update. The robot takes different branches depend- ing on the observation received. The algorithm has three main steps. First, a rapidly-exploring random graph (RRG) on the state space is grown. Second, the RRG is expanded to a belief- space graph by querying the observation model. In a third step, dynamic programming is performed on the belief-space graph to extract a path-tree. The resulting path-tree combines exploration with exploitation i.e. it balances the need for gaining knowledge about the environment with the need for reaching the goal. We demonstrate the algorithm capabilities on navigation and mobile manipulation tasks, and show its advantage over a baseline using a task and motion planning approach (TAMP) both in terms of optimality and runtime.

Keywords: SLAM, Localization, Mapping
Abstract: Semantic simultaneous localization and mapping (SLAM) is a popular technology enabling indoor mobile robots to sufficiently perceive and interact with the environment. In this paper, we propose an object-aware semantic SLAM system, which consists of a quadric initialization method, an object-level data association method, and a multi-constraint optimization factor graph. To overcome the limitation of multi-view observations and the requirement of dense point clouds for objects, an efficient quadric initialization method based on object detection and surfel construction is proposed, which can efficiently initialize quadrics within fewer frames and with small viewing angles. The robust object-level joint data association method and the tightly coupled multi-constraint factor graph for quadrics optimization and joint bundle adjustment enable the accurate estimation of constructed quadrics and camera poses. Extensive experiments using public datasets show that the proposed system achieves competitive performance with respect to accuracy and robustness of object quadric estimation and camera localization compared with stateof-the-art methods.

Keywords: Planning under Uncertainty, Reinforcement Learning, Integrated Planning and Learning
Abstract: This paper addresses the problem of learning control policies for mobile robots, modeled as unknown Markov Decision Processes (MDPs), that are tasked with temporal logic missions, such as sequencing, coverage, or surveillance. The MDP captures uncertainty in the workspace structure and the outcomes of control decisions. The control objective is to synthesize a control policy that maximizes the probability of accomplishing a high-level task, specified as a Linear Temporal Logic (LTL) formula. To address this problem, we propose a novel accelerated model-based reinforcement learning (RL) algorithm for LTL control objectives that is capable of learning control policies significantly faster than related approaches. Its sample-efficiency relies on biasing exploration towards directions that may contribute to task satisfaction. This is accomplished by leveraging an automaton representation of the LTL task as well as a continuously learned MDP model. Finally, we provide comparative experiments that demonstrate the sample efficiency of the proposed method against recent RL methods for LTL objectives.

Keywords: Planning under Uncertainty, Optimization and Optimal Control, Model Learning for Control
Abstract: We present a sampling-based control approach that can generate smooth actions for general nonlinear systems without external smoothing algorithms. Model Predictive Path Integral (MPPI) control has been utilized in numerous robotic applications due to its appealing characteristics to solve non-convex optimization problems. However, the stochastic nature of sampling-based methods can cause significant chattering in the resulting commands. Chattering becomes more prominent in cases where the environment changes rapidly, possibly even causing the MPPI to diverge. To address this issue, we propose a method that seamlessly combines MPPI with an input-lifting strategy. In addition, we introduce a new action cost to smooth control sequence during trajectory rollouts while preserving the information theoretic interpretation of MPPI, which was derived from non-affine dynamics. We validate our method in two nonlinear control tasks with neural network dynamics: a pendulum swing-up task and a challenging autonomous driving task. The experimental results demonstrate that our method outperforms the MPPI baselines with additionally applied smoothing algorithms.

Keywords: Planning under Uncertainty, Motion and Path Planning, Planning, Scheduling and Coordination
Abstract: Path planning is a crucial algorithmic approach for designing robot behaviors. Sampling-based approaches, like rapidly exploring random trees (RRTs) or probabilistic roadmaps, are prominent algorithmic solutions for path planning problems. Despite its exponential convergence rate, RRT can only find suboptimal paths. On the other hand, RRT*, a widely-used extension to RRT, guarantees probabilistic completeness for finding optimal paths but suffers in practice from slow convergence in complex environments. Furthermore, real-world robotic environments are often partially observable or with poorly described dynamics, casting the application of RRT* in complex tasks suboptimal. This paper studies a novel algorithmic formulation of the popular Monte-Carlo tree search (MCTS) algorithm for robot path planning. Notably, we study Monte-Carlo Path Planning (MCPP) by analyzing and proving, on the one part, its exponential convergence rate to the optimal path in fully observable Markov decision processes (MDPs), and on the other part, its probabilistic completeness for finding feasible paths in partially observable MDPs (POMDP) assuming limited distance observability (proof sketch). Our algorithmic contribution allows us to employ recently proposed variants of MCTS with different exploration strategies for robot path planning. Experimental evaluation in simulated 2D and 3D environments with a 7 degrees of freedom (DOF) manipulator, as well as in a real-world robot path planning task, demonstrate the superiority of MCPP in POMDP tasks.

Keywords: Planning under Uncertainty
Abstract: Planning under uncertainty is a fundamental problem in robotics. Classical approaches rely on a metrical representation of the world and robot's states to infer the next course of action. While these approaches are considered accurate, they are often susceptible to metric errors and tend to be costly regarding memory and time consumption. However, in some cases, relying on qualitative geometric information alone is sufficient. Hence, the issues described above become an unnecessary burden. This work presents a novel qualitative Belief Space Planning (BSP) approach, highly suitable for platforms with low-cost sensors and particularly appealing in sparse environment scenarios. Our algorithm generalizes its predecessors by avoiding any deterministic assumptions. Moreover, it smoothly incorporates spatial information propagation techniques, known as compositions. We demonstrate our algorithm in simulations and the advantage of using compositions in particular.

Keywords: Task and Motion Planning, Motion and Path Planning, Manipulation Planning
Abstract: High-level autonomy requires discrete and continuous reasoning to decide both what actions to take and how to execute them. Integrated Task and Motion Planning (TMP) algorithms solve these hybrid problems jointly to consider constraints between the discrete symbolic actions (i.e., the task plan) and their continuous geometric realization (i.e., motion plans). This joint approach solves more difficult problems than approaches that address the task and motion subproblems independently.

TMP algorithms combine and extend results from both task and motion planning. TMP has mainly focused on computational performance and completeness and less on solution optimality. Optimal TMP is difficult because the independent optima of the subproblems may not be the optimal integrated solution, which can only be found by jointly optimizing both plans.

This paper presents Task and Motion Informed Trees (TMIT*), an optimal TMP algorithm that combines results from makespan-optimal task planning and almost-surely asymptotically optimal motion planning. TMIT* interleaves asymmetric forward and reverse searches to delay computationally expensive operations until necessary and perform an efficient informed search directly in the problem’s hybrid state space. This allows it to solve problems quickly and then converge towards the optimal solution with additional computational time, as demonstrated on the evaluated robotic-manipulation benchmark problems.

Keywords: Integrated Planning and Learning, Task and Motion Planning, Representation Learning
Abstract: The ability to plan for multi-step manipulation tasks in unseen situations is crucial for future home robots. But collecting sufficient experience data for end-to-end learning is often infeasible in the real world, as deploying robots in many environments can be prohibitively expensive. On the other hand, large-scale scene understanding datasets contain diverse and rich semantic and geometric information. But how to leverage such information for manipulation remains an open problem. In this paper, we propose a learning-to-plan method that can generalize to new object instances by leveraging object-level representations extracted from a synthetic scene understanding dataset. We evaluate our method with a suite of challenging multi-step manipulation tasks inspired by household activities and show that our model achieves measurably better success rate than state-of-the-art end-to-end approaches.

Keywords: Robot Safety, Autonomous Vehicle Navigation, Sensor-based Control
Abstract: A fully tested autonomous system works predictably under ideal or assumed environment. However, its behavior is not fully defined when some components malfunction or fail. In this paper, we consider automated guided vehicle (AGV), equipped with multiple sensors, executing a traversal task in a static unknown environment. We have analytically studied the system, computed a set of performance and safety metrics, and validated it with simulation results in Webots. We have also analyzed the effect on system performance under independent and correlated sensing errors. We have also performed sensitivity analysis to identify the most critical components in any given system; and this can be utilized to increase the reliability of the system and its conformance to safety objectives.

Keywords: Robot Safety, Human-Robot Collaboration, Service Robotics
Abstract: Vacuum-adhesion-based climbing robots have been developed to cater to the demands in the cleaning and inspection work of airplanes. A robot intended to clean and inspect an airplane faces a Risk of Adhesion (RoA) based on the robot and the surface conditions, such as worn-out suction cups. These sorts of underlying conditions are not easily noticeable for an operator of a robot and might lead to catastrophic events. Therefore, the ability of a robot to self-assess the RoA in a scenario and notify the operator is crucial for ensuring safety. Particularly, an aircraft inspection robot should have adhesion awareness. This paper proposes a novel method to self-assess the RoA of a vacuum-adhesion-based robot intended to clean and inspect airplanes. The RoA is assessed by a fuzzy inference system that analyzes the present pressure difference and the current duty setting of the vacuum pump of a robot. The robot operator can collaborate with the robot to take precautions based on the assessed RoA to ensure safety. The outcomes of the experiment conducted on an airplane skin validate the ability of the proposed method to assess the RoA associated with heterogeneous operating conditions effectively. Thus, the utilization of the proposed method would improve the safety of a vacuum-adhesion-based robot intended to clean and inspect airplanes.

Keywords: Robot Safety, Intelligent Transportation Systems
Abstract: Autonomous vehicles (AVs) rely on environment perception and behavior prediction to reason about agents in their surroundings. These perception systems must be robust to adverse weather such as rain, fog, and snow. However, validation of these systems is challenging due to their complexity and dependence on observation histories. This paper presents a method for characterizing failures of LiDAR-based perception systems for AVs in adverse weather conditions. We develop a methodology based in reinforcement learning to find likely failures in object tracking and trajectory prediction due to sequences of disturbances. We apply disturbances using a physics-based data augmentation technique for simulating LiDAR point clouds in adverse weather conditions. Experiments performed across a wide range of driving scenarios from a real-world driving dataset show that our proposed approach finds high likelihood failures with smaller input disturbances compared to baselines while remaining computationally tractable. Identified failures can inform future development of robust perception systems for AVs.

Keywords: Robot Safety, Methods and Tools for Robot System Design, Intelligent Transportation Systems
Abstract: Evaluating safety performance in a resource-efficient way is crucial for the development of autonomous systems. Simulation of parameterized scenarios is a popular testing strategy but parameter sweeps can be prohibitively expensive. To address this, we propose HiddenGems: a sample-efficient method for discovering the boundary between compliant and non-compliant behavior via active learning. Given a parameterized scenario, one or more compliance metrics, and a simulation oracle, HiddenGems maps the compliant and non-compliant domains of the scenario. The methodology enables critical test case identification, comparative analysis of different versions of the system under test, as well as verification of design objectives. We evaluate HiddenGems on a scenario with a jaywalker crossing in front of an autonomous vehicle and obtain compliance boundary estimates for collision, lane keep, and acceleration metrics individually and in combination, with 6 times fewer simulations than a parameter sweep. We also show how HiddenGems can be used to detect and rectify a failure mode for an unprotected turn with 86% fewer simulations.

Keywords: Robot Safety, Object Detection, Segmentation and Categorization, Deep Learning Methods
Abstract: Research on Deep Neural Networks (DNNs) has focused on improving performance and accuracy for real-world deployments, leading to new models, such as Spiking Neural Networks (SNNs), and optimization techniques, e.g., quantization and pruning for compressed networks. However, the deployment of these innovative models and optimization techniques introduces possible reliability issues, which is a pillar for DNNs to be widely used in safety-critical applications, e.g., autonomous driving. Moreover, scaling technology nodes have the associated risk of multiple faults happening at the same time, a possibility not addressed in state-of-the-art resiliency analyses.

Towards better reliability analysis for DNNs, we present enpheeph, a Fault Injection Framework for Spiking and Compressed DNNs. The enpheeph framework enables optimized execution on specialized hardware devices, e.g., GPUs, while providing complete customizability to investigate different fault models, emulating various reliability constraints and use-cases. Hence, the faults can be executed on SNNs as well as compressed networks with minimal-to-none modifications to the underlying code, a feat that is not achievable by other state-of-the-art tools.

To evaluate our enpheeph framework, we analyze the resiliency of different DNN and SNN models, with different compression techniques. By injecting a random and increasing number of faults, we show that DNNs can show a reduction in accuracy with a fault rate as low as 7 x 10 ^ (-7) faults per parameter, with an accuracy drop higher than 40%. Run-time overhead when executing enpheeph is less than 20% of the baseline execution time when executing 100 000 faults concurrently, at least 10x lower than state-of-the-art frameworks, making enpheeph future-proof for complex fault injection scenarios.

We release the source code of our enpheeph framework under an open-source license at https://github.com/Alexei95/enpheeph.

Keywords: Robot Safety, Optimization and Optimal Control
Abstract: This paper studies the safety control problem for mobile robots working in cluttered environments. A compact set is employed to represent the obstacles, and a direction-distance function is used to describe the obstacle-measurement model. The major contribution is a nontrivial modification of the quadratic programming (QP) approach for continuous safety control of integrator-modeled mobile robots. In particular, a refinement of the Moreau-Yosida method is proposed to regularize the measurement model while retaining feasibility and safety. The second contribution is the development of a new feasible set shaping technique with a positive basis for a QP-based continuous safety controller. Physical experiments are employed to verify the proposed method.

Keywords: Robot Safety, Probability and Statistical Methods, Reinforcement Learning
Abstract: While Deep Reinforcement Learning (DRL) provides transformational capabilities to the control of Robotics and Autonomous Systems (RAS), the black-box nature of DRL and uncertain deployment environments of RAS pose new challenges on its dependability. Although existing works impose constraints on the DRL policy to ensure successful completion of the mission, it is far from adequate to assess the DRL-driven RAS in a holistic way considering all dependability properties. In this paper, we formally define a set of dependability properties in temporal logic and construct a Discrete-Time Markov Chain (DTMC) to model the dynamics of risk/failures of a DRL-driven RAS interacting with the stochastic environment. We then conduct Probabilistic Model Checking (PMC) on the designed DTMC to verify those properties. Our experimental results show that the proposed method is effective as a holistic assessment framework while uncovering conflicts between the properties that may need trade-offs in training. Moreover, we find that the standard DRL training cannot improve dependability properties, thus requiring bespoke optimisation objectives. Finally, our method offers sensitivity analysis of dependability properties to disturbance levels from environments, providing insights for the assurance of real RAS.

Keywords: Robot Safety, Performance Evaluation and Benchmarking, Legged Robots
Abstract: The dynamic response of the legged robot locomotion is non-Lipschitz and can be stochastic due to environmental uncertainties. To test, validate, and characterize the safety performance of legged robots, existing solutions on observed and inferred risk can be incomplete and sampling inefficient. Some formal verification methods suffer from the model precision and other surrogate assumptions. In this paper, we propose a scenario sampling based testing framework that characterizes the overall safety performance of a legged robot by specifying (i) where (in terms of a set of states) the robot is potentially safe, and (ii) how safe the robot is within the specified set. The framework can also help certify the commercial deployment of the legged robot in real-world environment along with human and compare safety performance among legged robots with different mechanical structures and dynamic properties. The proposed framework is further deployed to evaluate a group of state-of-the-art legged robot locomotion controllers from various model-based, deep neural network involved, and reinforcement learning based methods in the literature. Among a series of intended work domains of the studied legged robots (e.g. tracking speed on sloped surface, with abrupt changes on demanded velocity, and against adversarial push-over disturbances), we show that the method can adequately capture the overall safety characterization and the subtle performance insights. Many of the observed safety outcomes, to the best of our knowledge, have never been reported by the existing work in the legged robot literature.

Keywords: Gesture, Posture and Facial Expressions, Computer Vision for Automation, RGB-D Perception
Abstract: Knowing the exact 3D location of workers and robots in a collaborative environment enables several real applications, such as the detection of unsafe situations or the study of mutual interactions for statistical and social purposes. In this paper, we propose a non-invasive and light-invariant framework based on depth devices and deep neural networks to estimate the 3D pose of robots from an external camera. The method can be applied to any robot without requiring hardware access to the internal states. We introduce a novel representation of the predicted pose, namely Semi-Perspective Decoupled Heatmaps (SPDH), to accurately compute 3D joint locations in world coordinates adapting efficient deep networks designed for the 2D Human Pose Estimation. The proposed approach, which takes as input a depth representation based on XYZ coordinates, can be trained on synthetic depth data and applied to real-world settings without the need for domain adaptation techniques. To this end, we present the SimBa dataset, based on both synthetic and real depth images, and use it for the experimental evaluation. Results show that the proposed approach, made of a specific depth map representation and the SPDH, overcomes the current state of the art.

Keywords: Human and Humanoid Motion Analysis and Synthesis, Human Detection and Tracking, Learning from Demonstration
Abstract: Human activities recognition is an important task for an intelligent robot, especially in the field of human-robot collaboration, it requires not only the label of sub-activities but also the temporal structure of the activity. In order to automatically recognize both the label and the temporal structure in sequence of human-object interaction, we propose a novel Pyramid Graph Convolutional Network (PGCN), which employs a pyramidal encoder-decoder architecture consisting of an attention based graph convolution network and a temporal pyramid pooling module for downsampling and upsampling interaction sequence on the temporal axis, respectively. The system represents the 2D or 3D spatial relation of human and objects from the detection results in video data as a graph. To learn the human-object relations, a new attention graph convolutional network is trained to extract condensed information from the graph representation. To segment action into sub-actions, a novel temporal pyramid pooling module is proposed, which upsamples compressed features back to the original time scale and classifies actions per frame. We explore various attention layers, namely spatial attention, temporal attention and channel attention, and combine different upsampling decoders to test the performance on action recognition and segmentation. We evaluate our model on two challenging datasets in the field of human-object interaction recognition, i.e. Bimanual Actions and IKEA Assembly datasets. We demonstrate that our classifier significantly improves both framewise action recognition and segmentation, e.g., F1 micro and F1@50 scores on Bimanual Actions dataset are improved by 4.3% and 8.5% respectively.

Keywords: Humanoid and Bipedal Locomotion, Humanoid Robot Systems, Legged Robots
Abstract: Legged locomotion in humans is governed by natural dynamics of the human body and neural control. One mechanism that is assumed to contribute to the high efficiency of human walking is the impulsive ankle push-off, which potentially powers the swing leg catapult. However, the mechanics of the human lower leg with its complex muscle-tendon units spanning over single and multiple joints is not yet understood. Legged robots allow testing the interaction between complex leg mechanics, control, and environment in real-world walking gait. We developed a 0.49 m tall, 2.2 kg anthropomorphic bipedal robot with Soleus and Gastrocnemius muscle-tendon units represented by linear springs, acting as mono- and biarticular elastic structures around the robot's ankle and knee joints. We tested the influence of three Soleus and Gastrocnemius spring-tendon configurations on the ankle power curves, the coordination of the ankle and knee joint movements, the total cost of transport, and walking speed. We controlled the robot with a feed-forward central pattern generator, leading to walking speeds between 0.35 m/s and 0.57 m/s at 1.0 Hz locomotion frequency, at 0.35 m leg length. We found differences between all three configurations; the Soleus spring-tendon modulates the robot's speed and energy efficiency likely by ankle power amplification, while the Gastrocnemius spring-tendon changes the movement coordination between ankle and knee joints during push-off.

Keywords: Human and Humanoid Motion Analysis and Synthesis, Dynamics
Abstract: Dynamic motions of humans and robots are widely driven by posture-dependent nonlinear interactions between their degrees of freedom. However, these dynamical effects remain mostly overlooked when studying the mechanisms of human movement generation. Inspired by recent works, we hypothesize that human motions are planned as sequences of geodesic synergies, and thus correspond to coordinated joint movements achieved with piecewise minimum energy. The underlying computational model is built on Riemannian geometry to account for the inertial characteristics of the body. Through the analysis of various human arm motions, we find that our model segments motions into geodesic synergies, and successfully predicts observed arm postures, hand trajectories, as well as their respective velocity profiles. Moreover, we show that our analysis can further be exploited to transfer arm motions to robots by reproducing individual human synergies as geodesic paths in the robot configuration space.

Keywords: Human and Humanoid Motion Analysis and Synthesis, Learning from Demonstration, Human-Robot Collaboration
Abstract: Manipulability ellipsoids efficiently capture the human pose and reveal information about the task at hand. Their use in task-dependent robot teaching – particularly their transfer from a teacher to a learner – can advance emulation of human-like motion. Although in recent literature focus is shifted towards manipulability transfer between two robots, the adaptation to the capabilities of the other kinematic system is to date not addressed and research in transfer from human to robot is still in its infancy. This work presents a novel manipulability domain adaptation method for the transfer of manipulability information to the domain of another kinematic system. As manipulability matrices/ellipsoids are symmetric positive-definite (SPD) they can be viewed as points on the Riemannian manifold of SPD matrices. We are the first to address the problem of manipulability transfer from the perspective of point cloud registration. We propose a manifold-aware Iterative Closest Point algorithm (ICP) with parallel transport initialization. Furthermore, we introduce a correspondence matching heuristic for manipulability ellipsoids based on inherent geometric features. We confirm our method in simulation experiments with 2-DoF manipulators as well as 7-DoF models representing the human-arm kinematics.

Keywords: Humanoid and Bipedal Locomotion, Biomimetics, Legged Robots
Abstract: Humans are able to negotiate downstep behaviors---both planned and unplanned---with remarkable agility and ease. The goal of this paper is to systematically study the translation of this human behavior to bipedal walking robots, even if the morphology is inherently different. Concretely, we begin with human data wherein planned and unplanned downsteps are taken. We analyze this data from the perspective of reduced-order modelling of the human, encoding the center of mass (CoM) kinematics and contact forces, which allows for the translation of these behaviors into the corresponding reduced-order model of a bipedal robot. We embed the resulting behaviors into the full-order dynamics of a bipedal robot via nonlinear optimization-based controllers. The end result is the demonstration of planned and unplanned downsteps in simulation on an underactuated walking robot.

Keywords: Grasping, Multifingered Hands, Deep Learning in Grasping and Manipulation
Abstract: The human hand capabilities are paramount for highly dexterous manipulation interactions. Unfortunately, the limitations of current technologies make replicating such capabilities unfeasible. Although several works have focused on directly attempting to create robot hands able to mimic human ones closely, few of them have attempted to create generalizable platforms, where robotic hand mechanisms can be iteratively selected and customized to different tasks. In order to build highly dexterous robotic hands in the future, it is crucial to understand not only human manipulation, but also develop methods to leverage robotic mechanisms limitations to mimic human hand interactions accurately. In this letter, we propose an end-to-end framework capable of generating underactuated tendon routings that allow a generic robot hand model to reproduce desired observed human grasp motion synergies accurately. Our contributions are threefold: (1) an end to end framework to generate task-oriented robot hand tendon routings, with the potential to implement desired synergies, (2) a novel grammar based representation of robot hand tendon routings, and (3) a schematic visualization of robot hand tendon routings. The latter two contributions have the potential to embed and compare properties among robot hands. Our results in simulation show that the proposed method produces tendon routing mechanisms that are able to closely mimic the joint trajectories of human subjects performing the same experimental tasks, while achieving dynamically stable grasping postures.

Keywords: Human Performance Augmentation, Physically Assistive Devices, Human-Robot Collaboration
Abstract: Walking with load is a common task in daily life and disaster rescue. Long-term load carriage may cause irreversible damage to the human body. Although remarkable progress has been made in the field of wearable robots, it is still far from avoiding interference to human legs, which will lead to energy consumption. In this paper, a novel wearable robot, Centaur, for assisting load carriage has been proposed. The Centaur system consists of two rigid robotic legs of two degrees-of-freedom (DOFs) to transfer load weight to the ground. Different from exoskeletons, the robotic legs of the Centaur are placed behind the human rather than attached to human limbs, which can provide a larger support polygon and avoid additional interference to the wearer. Additionally, the Centaur can attain the locomotion stability of the quadruped while maintaining the motion agility of the biped itself. This paper also presents an interactive motion control strategy based on the human-robot interaction force. This control strategy incorporates legged robotics walking controller and real-time walking trajectory planning to realize the cooperative walking with human beings. Finally, experiments of human walking with load carriage have been conducted on flat terrain to verify the concept of the Centaur system. The result demonstrates that the Centaur system can effectively reduce 70.03% of load weight during the single stance phase, which indicates that the Centaur system provides a new solution for assisting human walking with load-carriage.

Keywords: Human Performance Augmentation, Modeling and Simulating Humans, Optimization and Optimal Control
Abstract: This study aims to construct a running simulator based on a motion generation and control system that enables the description of motion strategies using the spring-loaded inverted pendulum (SLIP) model. The problems of stability and robustness encountered in the running simulation with the SLIP model are elucidated, and stable running is achieved by controlling the stiffness and the attitude angle dynamically at touchdown, as well as human energy adjustment that is introduced to consider the active motion strategy. As a result, passive and active control by humans can be expressed, and a framework that can express the changes in running due to motion strategies is constructed. Finally, we discuss the possibility of describing and elucidating the motion strategies.

Keywords: Human Detection and Tracking, Sensor Fusion, RGB-D Perception
Abstract: We tackle the problem of tracking the human lower body as an initial step toward an automatic motion assessment system for clinical mobility evaluation, using a multimodal system that combines Inertial Measurement Unit (IMU) data, RGB images, and point cloud depth measurements. This system applies the factor graph representation to an optimization problem that provides 3-D skeleton joint estimations. In this paper, we focus on improving the temporal consistency of the estimated human trajectories to greatly extend the range of operability of the depth sensor. More specifically, we introduce a new factor graph factor based on Koopman theory that embeds the nonlinear dynamics of several lower-limb movement activities. This factor performs a two-step process: first, a custom activity recognition module based on spatial temporal graph convolutional networks recognizes the walking activity; then, a Koopman pose prediction of the subsequent skeleton is used as an a priori estimation to drive the optimization problem toward more consistent results. We tested the performance of this module on datasets composed of multiple clinical lower-limb mobility tests, and we show that our approach reduces outliers on the skeleton form by almost 1 m, while preserving natural walking trajectories at depths up to more than 10 m.

Keywords: Computer Vision for Medical Robotics, Surgical Robotics: Laparoscopy, RGB-D Perception
Abstract: The semantic segmentation of surgical scenes is a prerequisite for task automation in robot assisted interventions. We propose LapSeg3D, a novel DNN-based approach for the voxel-wise annotation of point clouds representing surgical scenes. As the manual annotation of training data is highly time consuming, we introduce a semi-autonomous clustering-based pipeline for the annotation of the gallbladder, which is used to generate segmented labels for the DNN. When evaluated against manually annotated data, LapSeg3D achieves an F1 score of 0.94 for gallbladder segmentation on various datasets of ex-vivo porcine livers. We show LapSeg3D to generalize accurately across different gallbladders and datasets recorded with different RGB-D camera systems.

Keywords: Computer Vision for Medical Robotics, Social HRI, Human-Centered Robotics
Abstract: Egocentric vision is an emerging topic, which has demonstrated great potential in assistive healthcare scenarios, ranging from human-centric behavior analysis to personal social assistance. Within this field, due to the heterogeneity of visual perception from first-person views, egocentric pose estimation is one of the most significant prerequisites for enabling various downstream applications. However, existing methods for egocentric pose estimation mainly focus on predicting the pose represented in the camera coordinates from a single image, which ignores the latent cues in the temporal domain and results in less accuracy. In this paper, we propose Ego+X, an egocentric vision based system for 3D canonical pose estimation and human-centric social interaction characterization. Our system is composed of two head-mounted egocentric cameras, where one is faced downwards and the other looks outwards. By leveraging the global context provided by visual SLAM, we first propose Ego-Glo for spatial-accurate and temporal-consistent egocentric 3D pose estimation in the canonical coordinate system. With the help of an egocentric camera looking outwards, we then propose Ego-Soc by extending Ego-Glo to various social interaction tasks, e.g., object detection and human-human interaction. Quantitative and qualitative experiments have been conducted to demonstrate the effectiveness of our proposed Ego+X.

Keywords: Computer Vision for Medical Robotics, Localization
Abstract: Monocular SLAM in deformable scenes will open the way to multiple medical applications like computer- assisted navigation in endoscopy, automatic drug delivery or autonomous robotic surgery. In this paper we propose a novel method to simultaneously track the camera pose and the 3D scene deformation, without any assumption about environment topology or shape. The method uses an illumination-invariant photometric method to track image features and estimates camera motion and deformation combining reprojection error with spatial and temporal regularization of deformations. Our results in simulated colonoscopies show the method’s accuracy and robustness in complex scenes under increasing levels of deformation. Our qualitative results in human colonoscopies from Endomapper dataset show that the method is able to successfully cope with the challenges of real endoscopies: deformations, low texture and strong illumination changes. We also compare with previous tracking methods in simpler scenarios from Hamlyn dataset where we obtain competitive performance, without needing any topological assumption.

Keywords: Computer Vision for Medical Robotics, Visual Tracking, Medical Robots and Systems
Abstract: Suture needle localization is necessary for autonomous suturing. Previous approaches in autonomous suturing often relied on fiducial markers rather than markerless detection schemes for localizing a suture needle due to the inconsistency of markerless detections. However, fiducial markers are not practical for real-world applications and can often be occluded from environmental factors in surgery (e.g., blood). Therefore in this work, we present a robust tracking approach for estimating the 6D pose of a suture needle when using inconsistent detections. We define observation models based on suture needles' geometry that captures the uncertainty of the detections and fuse them temporally in a probabilistic fashion. In our experiments, we compare different permutations of the observation models in the suture needle localization task to show their effectiveness. Our proposed method outperforms previous approaches in localizing a suture needle. We also demonstrate the proposed tracking method in an autonomous suture needle regrasping task and ex vivo environments.

Keywords: Omnidirectional Vision, RGB-D Perception
Abstract: Depth estimation from panoramic imagery has received minimal attention in contrast to standard perspective imagery, which constitutes the majority of the literature on the key research topic. The vast - and frequently complete - field of view provided by such panoramic photographs makes them appealing for a variety of applications, including robots, autonomous vehicles, and virtual reality. Consumer-level camera systems capable of capturing such images are likewise growing more affordable, and may be desirable complements to autonomous systems' sensor packages. They do, however, introduce significant distortions and violate some assumptions regarding perspective view images. Additionally, many state-of-the-art algorithms are not designed for its projection model, and their depth estimation performance tends to degrade when being applied to panoramic imagery.

This paper presents a novel technique for adapting view synthesis-based depth estimation models to omnidirectional vision. Specifically, we: 1) integrate a ``virtual'' spherical camera model into the training pipeline, facilitating the model training, 2) exploit spherical convolutional layers to perform convolution operations on equirectangular images, handling the severe distortion, and 3) propose an optical flow-based masking scheme to mitigate the effect of unwanted pixels during training. Our qualitative and quantitative results demonstrate that these simple yet efficient designs result in significantly improved depth estimations when compared to previous approaches.

Keywords: Visual Servoing, Deep Learning for Visual Perception
Abstract: An increasing number of nonspecialist robotic users demand easy-to-use machines. In the context of visual servoing, the removal of explicit image processing is becoming a trend, allowing an easy application of this technique. This work presents a deep learning approach for solving the perception problem within the visual servoing scheme. An artificial neural network is trained using the supervision coming from the knowledge of the controller and the visual features motion model. In this way, it is possible to give a geometrical interpretation to the estimated visual features, which can be used in the analytical law of the visual servoing. The approach keeps perception and control decoupled, conferring flexibility and interpretability on the whole framework. Simulated and real experiments with a robotic manipulator validate our approach.

Keywords: Visual Tracking, Deep Learning for Visual Perception
Abstract: Tracking and reconstructing 3D objects from cluttered scenes are the key components for computer vision, robotics and autonomous driving systems. While recent progress in implicit function has shown encouraging results on high-quality 3D shape reconstruction, it is still very challenging to generalize to cluttered and partially observable LiDAR data. In this paper, we propose to leverage the continuity in video data. We introduce a novel and unified framework which utilizes a neural implicit function to simultaneously track and reconstruct 3D objects in the wild. Our approach adapts the DeepSDF model (i.e., an instantiation of the implicit function) in the video online, iteratively improving the shape reconstruction while in return improving the tracking, and vice versa. We experiment with both Waymo and KITTI datasets, and show significant improvements over state-of-the-art methods for both tracking and shape reconstruction tasks.

Keywords: Visual Tracking, Computer Vision for Transportation, Computer Vision for Automation
Abstract: In the recent literature, on the one hand, many 3D multi-object tracking (MOT) works have focused on tracking accuracy and neglected computation speed, commonly by designing rather complex cost functions and feature extractors. On the other hand, some methods have focused too much on computation speed at the expense of tracking accuracy. In view of these issues, this paper proposes a robust and fast camera-LiDAR fusion-based MOT method that achieves a good trade-off between accuracy and speed. Relying on the characteristics of camera and LiDAR sensors, an effective deep association mechanism is designed and embedded in the proposed MOT method. This association mechanism realizes tracking of an object in a 2D domain when the object is far away and only detected by the camera, and updating of the 2D trajectory with 3D information obtained when the object appears in the LiDAR field of view to achieve a smooth fusion of 2D and 3D trajectories. Extensive experiments based on the KITTI dataset indicate that our proposed method presents obvious advantages over the state-of-the-art MOT methods in terms of both tracking accuracy and processing speed. Our code is made publicly available for the benefit of the community.

Keywords: Visual Tracking, Sensor-based Control
Abstract: Event-based sensors have recently drawn increasing interest in robotic perception due to their lower latency, higher dynamic range, and lower bandwidth requirements compared to standard CMOS-based imagers. These properties make them ideal tools for real-time perception tasks in highly dynamic environments. In this work, we demonstrate an application where event cameras excel: accurately estimating the impact location of fast-moving objects. We introduce a lightweight event representation called Binary Event History Image (BEHI) to encode event data at low latency, as well as a learning-based approach that allows real-time inference of a confidence-enabled control signal to the robot. To validate our approach, we present an experimental catching system in which we catch fast-flying ping-pong balls. We show that the system is capable of achieving a success rate of 81% in catching balls targeted at different locations, with a velocity of up to 13 m/s even on compute-constrained embedded platforms such as the Nvidia Jetson NX.

Keywords: Mapping, RGB-D Perception, Motion and Path Planning
Abstract: Creating accurate maps of complex, unknown environments is of utmost importance for truly autonomous navigation robot. However, building these maps online is far from trivial, especially when dealing with large amounts of raw sensor readings on a computation and energy constrained mobile system, such as a small drone. While numerous approaches tackling this problem have emerged in recent years, the mapping accuracy is often sacrificed as systematic approximation errors are tolerated for efficiency's sake. Motivated by these challenges, we propose Voxfield, a mapping framework that can generate maps online with higher accuracy and lower computational burden than the state of the art. Built upon the novel formulation of non-projective truncated signed distance fields (TSDFs), our approach produces more accurate and complete maps, suitable for surface reconstruction. Additionally, it enables efficient generation of euclidean signed distance fields (ESDFs), useful e.g. for path planning, that does not suffer from typical approximation errors. Through a series of experiments with public datasets, both real-world and synthetic, we demonstrate that our method beats the state of the art in map coverage, accuracy and computational time. Moreover, we show that Voxfield can be utilized as a back-end in recent multi-resolution mapping frameworks, producing high quality maps even in large-scale experiments. Finally, we validate our method by running it onboard a quadrotor, showing it can generate accurate ESDF maps usable for real-time path planning and obstacle avoidance.

Keywords: Mapping, SLAM
Abstract: Safe motion planning in robotics requires planning into space which has been verified to be free of obstacles. However, obtaining such environment representations using lidars is challenging by virtue of the sparsity of their depth mea- surements. We present a learning-aided 3D lidar reconstruction framework that upsamples sparse lidar depth measurements with the aid of overlapping camera images so as to generate denser reconstructions with more definitively free space than can be achieved with the raw lidar measurements alone. We use a neural network with an encoder-decoder structure to predict dense depth images along with depth uncertainty estimates which are fused using a volumetric mapping system. We conduct experiments on real-world outdoor datasets captured using a handheld sensing device and a legged robot. Using input data from a 16-beam lidar mapping a building network, our experiments showed that the amount of estimated free space was increased by more than 40% with our approach. We also show that our approach trained on a synthetic dataset generalises well to real-world outdoor scenes without additional fine-tuning. Finally, we demonstrate how motion planning tasks can benefit from these denser reconstructions.

Keywords: Mapping, Vision-Based Navigation, Localization
Abstract: This paper presents an accurate and scalable method for fiducial tag localization on a 3D prior environmental map. The proposed method comprises three steps: 1) visual odometry-based landmark SLAM for estimating the relative poses between fiducial tags, 2) geometrical matching-based global tag-map registration via maximum clique finding, and 3) tag pose refinement based on direct camera-map alignment with normalized information distance. Through simulation-based evaluations, the proposed method achieved a 98 % global tag-map registration success rate and an average tag pose estimation accuracy of a few centimeters. Experimental results in a real environment demonstrated that it enables to localize over 110 fiducial tags placed in an environment in 25 minutes for data recording and post-processing.

Keywords: Mapping, Semantic Scene Understanding, RGB-D Perception
Abstract: The equations of motion governing mobile robots are dependent on terrain properties such as the coefficient of friction, and contact model parameters. Estimating these properties is thus essential for robotic navigation. Ideally any map estimating terrain properties should run in real time, mitigate sensor noise, and provide probability distributions of the aforementioned properties, thus enabling risk-mitigating navigation and planning. This paper addresses these needs and proposes a Bayesian inference framework for semantic mapping which recursively estimates both the terrain surface profile and a probability distribution for terrain properties using data from a single RGB-D camera. The proposed framework is evaluated in simulation against other semantic mapping methods and is shown to outperform these state-of-the-art methods in terms of correctly estimating simulated ground-truth terrain properties when evaluated using a precision-recall curve and the Kullback-Leibler divergence test. Additionally, the proposed method is deployed on a physical legged robotic platform in both indoor and outdoor environments, and we show our method correctly predicts terrain properties in both cases. The proposed framework runs in real-time and includes a ROS interface for easy integration.

Keywords: Mapping, Autonomous Vehicle Navigation, Localization
Abstract: Deploying fully autonomous vehicles has been a subject of intense research in both industry and academia. However, the majority of these efforts have relied heavily on High Definition (HD) prior maps. These are necessary to provide the planning and control modules a rich model of the operating environment. While this approach has shown success, it drastically limits both the scale and scope of these deployments as creating and maintaining HD maps for very large areas can be prohibitive. In this work, we present a new method for building the HD map online by starting with a Standard Definition (SD) prior map such as a navigational road map, and incorporating onboard sensors to infer the local HD map. We evaluate our method extensively on 100 sequences of real-world vehicle data and demonstrate that it can infer a highly structured HD map-like model of the world accurately using only SD prior maps and onboard sensors.

Keywords: Mapping, Localization
Abstract: High-Definition (HD) maps are needed for robust navigation of autonomous vehicles, limited by the on-board storage capacity. To solve this, we propose a novel framework, Environment-Aware Normal Distributions Transform (EA-NDT), that significantly improves compression of standard NDT map representation. The compressed representation of EA-NDT is based on semantic-aided clustering of point clouds resulting in more optimal cells compared to grid cells of standard NDT. To evaluate EA-NDT, we present an open-source implementation that extracts planar and cylindrical primitive features from a point cloud and further divides them into smaller cells to represent the data as an EA-NDT HD map. We collected an open suburban environment dataset and evaluated EA-NDT HD map representation against the standard NDT representation. Compared to the standard NDT, EA-NDT achieved consistently at least 1.5× higher map compression while maintaining the same descriptive capability. Moreover, we showed that EA-NDT is capable of producing maps with significantly higher descriptivity score when using the same number of cells than the standard NDT.

Keywords: Mapping
Abstract: Identifying the environment's structure, i.e., to detect core components as rooms and walls, can facilitate several tasks fundamental for the successful operation of indoor autonomous mobile robots, including semantic environment understanding. These robots often rely on 2D occupancy maps for core tasks such as localisation, motion and task planning. However, reliable identification of structure and room segmentation from 2D occupancy maps is still an open problem due to clutter (e.g., furniture and movable object), occlusions, and partial coverage. We propose a method for the RObust StructurE identification and ROom SEgmentation (ROSE^2) of 2D occupancy maps, which may be cluttered and incomplete. ROSE^2 identifies the main directions of walls and is resilient to clutter and partial observations, allowing to extract a clean, abstract geometrical floor-plan-like description of the environment, which is used to segment, i.e., to identify rooms in, the original occupancy grid map. ROSE^2 is tested in several real-world publicly-available cluttered maps obtained in different conditions. The results show how it can robustly identify the environment structure in 2D occupancy maps suffering from clutter and partial observations, while significantly improving room segmentation accuracy. Thanks to the combination of clutter removal and robust room segmentation ROSE^2 consistently achieves higher performance than the state-of-the-art methods, against which it is compared.

Keywords: Mapping, SLAM, Deep Learning for Visual Perception
Abstract: Feature-based visual simultaneous localization and mapping (SLAM) methods only estimate the depth of extracted features, generating a sparse depth map. To solve this sparsity problem, depth completion tasks that estimate a dense depth from a sparse depth have gained significant importance in recent years. Existing methodologies that use sparse depth from visual SLAM mainly employ point features. However, point features have limitations in preserving structural regularities owing to textureless environments and sparsity problems. To deal with these issues, we perform depth completion with visual SLAM using line features, which can better contain structural regularities than point features. The proposed methodology creates a convex hull region by performing constrained Delaunay triangulation with depth interpolation using line features. However, the generated depth includes low-frequency information and is discontinuous at the convex hull boundary. Therefore, we propose a mesh depth refinement (MDR) module to address this problem. The MDR module effectively transfers the high-frequency details of an input image to the interpolated depth and plays a vital role in bridging the conventional and deep learning-based approaches. The Struct-MDC outperforms other state-of-the-art algorithms on public and our custom datasets, and even outperforms supervised methodologies for some metrics. In addition, the effectiveness of the proposed MDR module is verified by a rigorous ablation study.

Keywords: Data Sets for Robotic Vision, Deep Learning for Visual Perception, Mapping
Abstract: This work addresses a gap in semantic scene completion (SSC) data by creating a novel outdoor data set with accurate and complete dynamic scenes. Our data set is formed from randomly sampled views of the world at each time step, which supervises generalizability to complete scenes without occlusions or traces. We create SSC baselines from state-of-the-art open source networks and construct a benchmark real-time dense local semantic mapping algorithm, MotionSC, by leveraging recent 3D deep learning architectures to enhance SSC with temporal information. Our network shows that the proposed data set can quantify and supervise accurate scene completion in the presence of dynamic objects, which can lead to the development of improved dynamic mapping algorithms. All software is available at https://github.com/UMich-CURLY/3DMapping.

Keywords: Soft Robot Materials and Design, Biologically-Inspired Robots, Soft Sensors and Actuators
Abstract: This work presents a humanoid soft robotics finger design with rigid skeletons and proprioceptive sensors. This 4-DOFs dexterous finger has soft joints and rigid phalanxes, which is about the size of human hand. To enhance the overall stiffness and for human-like behavior and configuration, rigid-soft actuators which we called quasi-joint is introduced. Although their lengths are shortened in this design, the soft actuators can still bend over 90°, exhibiting joint-like flexion and abduction/adduction. Thus IP joints and MCP joint are realized. EGaIn soft sensors are embedded into the structure for bending detection. In addition, multi-step molding fabrication method is introduced for this complex multi-material structure. This rigid-soft finger is a preliminary work and modular part of a highly dexterous humanoid soft robotic hand.

Keywords: Soft Robot Materials and Design, Mechanism Design, Climbing Robots
Abstract: There are many spaces inaccessible to humans where robots could help deliver sensors and equipment. Many of these spaces contain three-dimensional passageways and uneven terrain that pose challenges for robot design and control. Everting toroidal robots, which move via simultaneous eversion and inversion of their body material, are promising for navigation in these types of spaces. We present a novel soft everting toroidal robot that propels itself using a motorized device inside an air-filled membrane. Our robot requires only a single control signal to move, can conform to its environment, and can climb vertically with a motor torque that is independent of the force used to brace the robot against its environment. We derive and validate models of the forces involved in its motion, and we demonstrate the robot's ability to navigate a maze and climb a pipe.

Keywords: Soft Robot Materials and Design, Kinematics
Abstract: As soft, continuum robots see increasing areas of application, many scenarios have arisen where it is necessary to consider the geometric shape of the robot. The current approaches to robot kinematics, such as the piecewise constant-curvature (PCC) model, are effective in representing simple overall robot geometry and estimating the end-effector state, but they are less intuitive for planing robots that involve complex geometries. In this work, we propose a solution to the geometric design problem by a two-part approach: a free-form spline defines a "shape curve" that describes the overall geometry of the robot, and then a "kinematic curve" composed of shapes that are feasible to replicate with continuum robots is fitted to the shape curve. As an implementation of this approach, we specifically explore the application of piecewise cubic Bezier curves in designing the shape curve of the robot, and pairs of arcs to construct the kinematic curves. Finally, the approach is applied to a tip-extension "vine" robot that is designed and fabricated to "grow" along a designed path and access the top surface of an obstacle.

Keywords: Soft Robot Materials and Design, Grippers and Other End-Effectors, Soft Sensors and Actuators
Abstract: Soft fingers with multiple segments can perform various grasping modes when each segment is individually controlled, although this requires a number of inputs and leads to a complicated structure. In this paper, we propose a multi-segment soft finger capable of generating dual modes only using a single input channel. We use the snap-through behavior of a soft spherical shell with a slit as a flow valve between two finger segments. Experimental results showed that geometrical designs and material properties of the valve determine its critical pressure (i.e., the pressure causing buckling of the shell) and affect the proposed soft fingers' shape deformation and mode transition. We finally characterized an antipodal gripper with two proposed fingers in terms of the acquisition region and grasp robustness. The gripper could achieve not only a broad range of precision grasp by fingers with a passive distal segment but also robust stability of power grasp by fingers with an active distal segment, without adding extra input channels.

Keywords: Soft Robot Materials and Design, Hydraulic/Pneumatic Actuators, Soft Robot Applications
Abstract: The inherent compliance of soft mobile robots makes them a safe option for interaction with humans and fragile environments. To expand their possibilities around our daily life, it is desirable for them to have enhanced traversal ability with safe structure and movement. To meet this requirement, we propose a new type of soft mobile robot that moves while selecting rotational, wave, and climbing locomotion. The proposed robot, named DECSO, forms a closed chain of modular segment actuators that individually generate contraction and bidirectional bending motion by applying negative pressure. In this paper, after presenting methods for generating the above three kinds of locomotion, the characteristics and modeling of the segment actuator are illustrated. Furthermore, the experimental results show that a 4-segment prototype achieved smooth rotational movement in the horizontal plane at a velocity of 75 mm/s, while a 6-segment prototype was able not only to move forward by travelling waves, but also to cross over the human body and obstacles of 62% of its own height by rotational movement. These findings suggest a new possibility for soft mobile robots that can safely touch and move around the human body.

Keywords: Soft Robot Materials and Design, Soft Robot Applications
Abstract: Grasping and manipulation are complex and demanding tasks, especially when executed in dynamic and unstructured environments. Typically, such tasks are executed by rigid articulated end-effectors, with a plethora of actuators that need sophisticated sensing and complex control laws to execute them efficiently. Soft robotics offers an alternative that allows for simplified execution of these demanding tasks, enabling the creation of robust, efficient, lightweight, and affordable solutions that are easy to control and operate. In this work, we introduce a new class of soft, kirigami-based robotic grippers, study their post-contact behavior and investigate different cut patterns for their development. We follow an experimental approach in which several designs are proposed and employed in a series of grasping and force exertion tests to compare their capabilities and post-contact behavior. The results of such experiments indicate a clear relationship between degree of reconfiguration and grasping force, and provide key insights into the effect of the cut patterns in the performance of the designs. These findings are then used in the design process of an improved version of multi-layer, disposable kirigami grippers that are fabricated employing simple 3D printed layers or laser cut PET films and silicone rubber using the concept of Hybrid Deposition Manufacturing (HDM). A series of experimental results demonstrate that the proposed design and manufacturing methods can enable the creation of soft, kirigami-based grippers with superior grasping capabilities that can handle delicate, contaminated, and everyday life objects and can even be disposed off in an automated way (e.g., after handling hazardous materials, such as medical waste).

Keywords: Soft Robot Materials and Design, Soft Robot Applications, Modeling, Control, and Learning for Soft Robots
Abstract: Recent advances in soft robotics in academia have led to the adoption of soft grippers in industrial settings. Due to their soft bending actuators, these grippers can handle delicate objects with great care. However, due to their flexibility, the actuators are prone to out-of-plane deformations upon asymmetric loading. These undesired deformations lead to reduced grasp performance and may cause instability or failure of the grip. While the state-of-the-art contributions describe complex designs to limit those deformations, this work focuses on a complementary path investigating the material distribution. In this paper, a novel bending actuator is developed with improved out-of-plane deformation resistance by optimizing the material distribution in multi-material designs composed of two polymers with different mechanical properties. This is made possible by the strong interfacial strength of Diels-Alder chemical bonds in the used polymers, which have a self-healing capability. A Solid Isotropic Material with Penalization (SIMP) topology optimization is performed to increase the out-of-plane resistance. The actuator is simulated using FEA COMSOL in which the (hyper) elastic materials are simulated by Mooney-Rivlin models, fitted on experimental uniaxial tensile test data. This multi-material actuator and a reference single material actuator were manufactured and modeled. Via experimental characterization and validation in FEA simulations, it is shown that the actuator performance, characterized by the in-plane performance and out-plane resistance, can be increased by an optimized multi-material composition, without changing the geometrical shape of the actuator.

Keywords: Soft Robot Materials and Design, Hydraulic/Pneumatic Actuators, Soft Sensors and Actuators
Abstract: This study presents the usage of Braided Actuators with Nested Structures to improve the stroke characteristics of the McKibben actuators. Thin McKibben actuators can contract 23 % of their original length when pressurized. Usage of braiding and nested structures to improve contraction ratio has proven successful. In this study, we use a combined telescopic Nested Structure and Braided Actuators to increase the contraction ratio of the actuators. The Nested Braided Actuator (NBA) performance was compared to Single Actuator (SA), Braided Actuator (BA), and Nested Actuator (NA). The results at 350 kPa indicate that, NBA had the highest contraction ratio of 45.5 %, followed by NA (39.38 %), BA (29.57 %), and SA (23.41 %). At 350 kPa, the contraction force per actuator NBA was 36.29 N, NA exerted 41.33 N, BA exerted 30.41 N and SA exerted 43.8 N. The actuator's performance showed that it was capable of high stroke with only a 20 – 30 % loss in contraction force compared to the SA.

Keywords: Soft Robot Materials and Design, Hydraulic/Pneumatic Actuators, Marine Robotics
Abstract: The optimal stiffness for soft swimming robots depends on swimming speed, which means no single stiffness can maximise efficiency in all swimming conditions. Tunable-stiffness would produce an increased range of high-efficiency swimming speeds for robots with flexible propulsors and enable soft control surfaces for steering underwater vehicles. We propose and demonstrate a method for tunable soft robotic stiffness using inflatable rubber tubes to stiffen a silicone foil through pressure and second moment of area change. We achieved double the effective stiffness of the system for an input pressure change from 0 to 0.8 bar and 2 J energy input. We achieved a resonant amplitude gain of 5 to 7 times the input amplitude and tripled the high-gain frequency range compared to a foil with fixed stiffness. These results show that changing second moment of area is an energy effective approach to tunable-stiffness robots.

Keywords: Localization, Marine Robotics, Biologically-Inspired Robots
Abstract: Having accurate localization capabilities is one of the fundamental requirements of autonomous robots. For underwater vehicles, the choices for effective localization are limited due to limitations of GPS use in water and poor environmental visibility that makes camera-based methods ineffective. Popular inertial navigation methods for underwater localization using Doppler-velocity log sensors, sonar, high-end inertial navigation systems, or acoustic positioning systems require bulky expensive hardware which are incompatible with low-cost, bio-inspired underwater robots. In this paper, we introduce an approach for underwater robot localization inspired by GPS methods known as acoustic pseudoranging. Our method allows us to localize multiple bio-inspired robots equipped with commonly available micro electro-mechanical systems microphones. This is achieved by estimating the time difference of arrival of acoustic signals sent simultaneously through four speakers with a known constellation geometry. We also leverage the same acoustic framework to perform one-way communication with multiple robots to execute some primitive motions. To our knowledge, this is the first application of the approach for the on-board localization of small bio-inspired robots in water. Hardware schematics and the accompanying code are released to aid further development in the field.

Keywords: Localization, Surgical Robotics: Steerable Catheters/Needles, Surgical Robotics: Planning
Abstract: Remote magnetic navigation offers an ideal platform for automated catheter navigation. Magnetically guided catheters show great dexterity and can reach locations that are otherwise challenging to access. By automating aspects of catheterization procedures, we can simplify and expedite the procedure to allow surgeons to focus on other critical tasks during the surgery. In this article, we describe an automation strategy that is based on the center line of extracted and registered vascular geometries. Position feedback is accomplished with a Hall-effect sensor embedded near the distal end of the catheter. Sensor measurements are compared to the magnetic field predicted by a linear model of the electromagnetic navigation system. By defining specific magnetic field gradients and applying the known vascular geometry, the magnetic fields can be utilized for the simultaneous navigation and localization of the catheter. This eliminates the need for other external, dedicated mapping systems, and the use of fluoroscopy imaging is minimized. The concept is tested in 2d vascular models and the accuracy of the localization is assessed with overhead camera tracking.

Keywords: Localization, Wheeled Robots, Sensor Fusion
Abstract: A novel robust and slip-aware speed estimation framework is developed and experimentally verified for mobile robot navigation by designing proprioceptive robust observers at each wheel. The observer for each corner is proved to be consistent, in the sense that it can provide an upper bound of the mean square estimation error (MSE) timely. Under proper conditions, the MSE is proved to be uniformly bounded. A covariance intersection fusion method is used to fuse the wheel-level estimates, such that the updated estimate remains consistent. The estimated slips at each wheel are then used for a robust consensus to improve the reliability of speed estimation in harsh and combined-slip scenarios. As confirmed by indoor and outdoor experiments under different surface conditions, the developed framework addresses state estimation challenges for mobile robots that experience uneven torque distribution or large slip. The novel proprioceptive observer can also be integrated with existing tightly-coupled visual-inertial navigation systems.

Keywords: Localization, Autonomous Vehicle Navigation
Abstract: VPR is a fundamental task for autonomous navigation as it enables a robot to localize itself in the workspace when a known location is detected. Although accuracy is an essential requirement for a VPR technique, computational and energy efficiency are not less important for real-world applications. CNN-based techniques archive state-of-the-art VPR performance but are computationally intensive and energy demanding. Binary neural networks (BNN) have been recently proposed to address VPR efficiently. Although a typical BNN is an order of magnitude more efficient than a CNN, its processing time and energy usage can be further improved. In a typical BNN, the first convolution is not completely binarized for the sake of accuracy. Consequently, the first layer is the slowest network stage, requiring a large share of the entire computational effort. This paper presents a class of BNNs for VPR that combines depthwise separable factorization and binarization to replace the first convolutional layer to improve computational and energy efficiency. Our best model achieves higher VPR performance while spending considerably less time and energy to process an image than a BNN using a non-binary convolution as a first stage.

Keywords: Localization, SLAM
Abstract: This paper proposes a highly accurate trajectory estimation method for outdoor mobile robots using global navigation satellite system (GNSS) time differences of carrier phase (TDCP) measurements. By using GNSS TDCP, the relative 3D position can be estimated with millimeter precision. However, when a phenomenon called cycle slip occurs, wherein the carrier phase measurement jumps and becomes discontinuous, it is impossible to accurately estimate the relative position using TDCP. Although previous studies have eliminated the effect of cycle slip using a robust optimization technique, it was difficult to completely eliminate the effect of outliers. In this paper, we propose a method to detect GNSS carrier phase cycle slip, estimate the amount of cycle slip, and modify the observed TDCP to calculate the relative position using the factor graph optimization framework. The estimated relative position acts as a loop closure in graph optimization and contributes to the reduction in the integration error of the relative position. Experiments with an unmanned aerial vehicle showed that by modifying the cycle slip using the proposed method, the vehicle trajectory could be estimated with an accuracy of 5–30 cm using only a single GNSS receiver, without using any other external data or sensors.

Keywords: Localization, Vision-Based Navigation
Abstract: Robocentric visual-inertial odometry (R-VIO) in our recent work [1] models the probabilistic state estimation problem with respect to a moving local (body) frame, which is contrary to a fixed global (world) frame as in the world-centric formulation, thus avoiding the observability mismatch issue and achieving better estimation consistency. To further improve efficiency and robustness in order to be amenable for the resource-constrained applications, in this paper, we propose a novel information-based estimator, termed R-VIO2. In particular, the numerical stability and computational efficiency are significantly boosted by using the i) square-root expression and ii) incremental QR-based update combined with back substitution. Moreover, the spatial transformation and time offset between visual and inertial sensors are jointly calibrated online to robustify the estimator performance in the presence of unknown parameter errors. The proposed R-VIO2 has been extensively tested on public benchmark dataset as well as in a large-scale real-world experiment, and shown to achieve very competitive accuracy and superior time efficiency against the state-of-the-art visual-inertial navigation methods.

Keywords: Localization, Range Sensing, Intelligent Transportation Systems
Abstract: We present an extensive comparison between three topometric localization systems: radar-only, lidar-only, and a cross-modal radar-to-lidar system across varying seasonal and weather conditions using the Boreas dataset. Contrary to our expectations, our experiments showed that our lidar-only pipeline achieved the best localization accuracy even during a snowstorm. Our results seem to suggest that the sensitivity of lidar localization to moderate precipitation has been exaggerated in prior works. However, our radar-only pipeline was able to achieve competitive accuracy with a much smaller map. Furthermore, radar localization and radar sensors still have room to improve and may yet prove valuable in extreme weather or as a redundant backup system. Code for this project can be found at: https://github.com/utiasASRL/vtr3

Keywords: Localization, Sensor Fusion, Marine Robotics
Abstract: For autonomous systems, the smoothness of estimated trajectories is essential to ensure robust and performant vehicle control. Unfortunately, approaches for state estimation using Kalman filters are sensitive to measurement outliers that can diverge dramatically. We propose a state-estimation algorithm using factor graph optimization (FGO) in continuous-time for GNSS/INS navigation systems to track this problem. The focus is on the smoothness of the estimated trajectory in environments where the GNSS observations become temporarily unreliable. Therefore, an inland shipping scenario with bridge crossings is selected as an application example. To estimate the trajectory in continuous-time, we integrate a White-Noise-On-Acceleration (WNOA) motion prior factor based on Gaussian process regression between successive states. Our results show a 30% improvement in accuracy and increased smoothness of the estimated trajectory with FGO compared to Kalman filtering.

Keywords: Localization, Range Sensing, Multi-Robot Systems
Abstract: Accurate and real-time position estimates are crucial for mobile robots. This work focuses on ranging-based positioning systems, which rely on distance measurements between known points, called anchors, and a tag to localize.

The topology of the network formed by the anchors strongly influences the tag's localizability, i.e., its ability to be accurately localized.

Here, the tag and some anchors are supposed to be carried by robots, which allows enhancing the positioning accuracy by planing the anchors' motions.

We leverage Bayesian Cram'er-Rao Lower Bounds (CRLBs) on the estimates' covariance in order to quantify the tag's localizability. This class of CRLBs can capture prior information on the tag's position and take it into account when deploying the anchors.

We propose a method to decrease a potential function based on the Bayesian CRLB in order to maintain the localizability of the tag while having some prior knowledge about its position distribution. Then, we present a new experiment highlighting the link between the localizability potential and the precision expected in practice. Finally, two real-time anchor motion planners are demonstrated with ranging measurements in the presence or absence of prior information about the tag's position.

Keywords: Reinforcement Learning, Robot Safety, Motion and Path Planning
Abstract: Reinforcement learning (RL) is capable of sophisticated motion planning and control for robots in uncertain environments. However, state-of-the-art deep RL approaches typically lack safety guarantees, especially when the robot and environment models are unknown. To justify widespread deployment, robots must respect safety constraints without sacrificing performance. Thus, we propose a Black-box Reachability-based Safety Layer (BRSL) with three main components: (1) data-driven reachability analysis for a black-box robot model, (2) a trajectory rollout planner that predicts future actions and observations using an ensemble of neural networks trained online, and (3) a differentiable polytope collision check between the reachable set and obstacles that enables correcting unsafe actions. In simulation, BRSL outperforms other state-of-the-art safe RL methods on a Turtlebot 3, a quadrotor, a trajectory-tracking point mass, and a hexarotor in wind with an unsafe set adjacent to the area of highest reward.

Keywords: Reinforcement Learning, Incremental Learning, Deep Learning in Grasping and Manipulation
Abstract: Deep reinforcement learning has shown its effectiveness in various applications, providing a promising direction for solving tasks with high complexity. However, naively applying classical RL for learning a complex long-horizon task with a single control policy is prohibitively inefficient. Thus, policy modularization tackles this problem by learning a set of modules that are mapped to primitive and properly orchestrating them. In this study, we further expand the discussion by incorporating simultaneous activation of the skills and structuring them into multiple hierarchies in a recursive fashion. Moreover, we sought to devise an algorithm that can properly orchestrate the skills with different action spaces via multiplicative Gaussian distributions, which highly increase the reusability. By exploiting the modularity, interpretability can also be achieved by observing the modules that are used in the new task if each of the skills is known. We demonstrate how the proposed scheme can be employed in practice by solving a pick and place task with a 6 DoF manipulator, and examine the effects of each property from ablation studies.

Keywords: Reinforcement Learning, Legged Robots
Abstract: Developing robust vision-guided controllers for quadrupedal robots in complex environments, with various obstacles, dynamical surroundings and uneven terrains, is very challenging. While Reinforcement Learning (RL) provides a promising paradigm for agile locomotion skills with vision inputs in simulation, it is still very challenging to deploy the RL policy in the real world. Our key insight is that aside from the discrepancy in the domain gap, in visual appearance between the simulation and the real world, the latency from the control pipeline is also a major cause of difficulty. In this paper, we propose Multi-Modal Delay Randomization (MMDR) to address this issue when training RL agents. Specifically, we simulate the latency of real hardware by using past observations, sampled with randomized periods, for both proprioception and vision. We train the RL policy for end-to-end control in a physical simulator without any predefined controller or reference motion, and directly deploy it on the real A1 quadruped robot running in the wild. We evaluate our method in different outdoor environments with complex terrains and obstacles. We demonstrate the robot can smoothly maneuver at a high speed, avoid the obstacles, and show significant improvement over the baselines.

Keywords: Reinforcement Learning, Motion and Path Planning, Formal Methods in Robotics and Automation
Abstract: Reinforcement learning (RL) based approaches have enabled robots to perform various tasks. However, most existing RL algorithms focus on learning a particular task, without considering generalization to new tasks. To address this issue, by combining meta learning and reinforcement learning, we develop a meta Q-learning of multi-task (MQMT) framework where the robot effectively learns a meta model from a diverse set of training tasks and then generalizes the learned model to a new set of tasks that have never been encountered during training using only a small amount of additional data. Particularly, the multiple tasks are specified by co-safe linear temporal logic specification. As a semantics-preserving rewriting operation, LTL progression is exploited to decompose training tasks into learnable sub-goals, which not only enables simultaneous learning of multiple tasks, but also facilitates reward design by converting non-Markovian reward process to Markovian ones. Reward shaping is further incorporated into the reward design to relax the sparse reward issue to improve reinforcement learning. The simulation and experiment results demonstrate the effectiveness of the MQMT framework.

Keywords: Machine Learning for Robot Control, Reinforcement Learning, Motion and Path Planning
Abstract: Model-free Deep Reinforcement Learning (DRL) controllers have demonstrated promising results on various challenging non-linear control tasks. While a model-free DRL algorithm can solve unknown dynamics and high-dimensional problems, it lacks safety assurance. Although safety constraints can be encoded as part of a reward function, there still exists a large gap between an RL controller trained with this modified reward and a safe controller. In contrast, instead of implicitly encoding safety constraints with rewards, we explicitly co-learn a Twin Neural Lyapunov Function (TNLF) with the control policy in the DRL training loop and use the learned TNLF to build a runtime monitor. Combined with the path generated from a planner, the monitor chooses appropriate waypoints that guide the learned controller to provide collision-free control trajectories. Our approach inherits the scalability advantages from DRL while enhancing safety guarantees. Our experimental evaluation demonstrates the effectiveness of our approach compared to DRL with augmented rewards and constrained DRL methods over a range of high-dimensional safety-sensitive navigation tasks.

Keywords: Reinforcement Learning, Robot Safety, Collision Avoidance
Abstract: This paper aims to solve a safe reinforcement learning (RL) problem with risk measure-based constraints. As risk measures, such as conditional value at risk (CVaR), focus on the tail distribution of cost signals, constraining risk measures can effectively prevent a failure in the worst case. An on-policy safe RL method, called TRC, deals with a CVaR-constrained RL problem using a trust region method and can generate policies with almost zero constraint violations with high returns. However, to achieve outstanding performance in complex environments and satisfy safety constraints quickly, RL methods are required to be sample efficient. To this end, we propose an off-policy safe RL method with CVaR constraints, called off-policy TRC. If off-policy data from replay buffers is directly used to train TRC, the estimation error caused by the distributional shift results in performance degradation. To resolve this issue, we propose novel surrogate functions, in which the effect of the distributional shift can be reduced, and introduce an adaptive trust-region constraint to ensure a policy not to deviate far from replay buffers. buffer. The proposed method has been evaluated in simulation and real-world environments and satisfied safety constraints within a few steps while achieving high returns even in complex robotic tasks.

Keywords: Reinforcement Learning, Compliance and Impedance Control, Machine Learning for Robot Control
Abstract: Reinforcement Learning (RL) has the potential of solving complex continuous control tasks, with direct applications to robotics. Nevertheless, current state-of-the-art methods require a vast amount of interaction experience and are generally not safe or feasible to learn directly on a physical robot. We address this challenge by a framework for learning latent action spaces for RL agents from demonstrated trajectories. We extend this framework by connecting it to a variable impedance Cartesian space controller, allowing us to learn contact-rich tasks safely and efficiently. Our method learns from trajectories that incorporate both positional, but also crucially impedance-space information. We evaluate our method on a number of peg-in-hole task variants with a Franka Panda arm and demonstrate that learning variable impedance actions for RL in Cartesian space can be deployed directly on the real robot, without resorting to learning in simulation.

Keywords: Reinforcement Learning, Planning under Uncertainty, Aerial Systems: Perception and Autonomy
Abstract: This paper proposes a deep reinforcement learning method for mobile sensors to estimate the properties of the source of the hazardous gas release. The problem of estimating the properties of the released gas is generally termed as the source term estimation (STE) problem. Since the sensor measurements from atmospheric gas dispersion are sparse, intermittent, and time-varying due to the turbulence and the sensor noise, STE is considered to be a challenging problem. The particle filter is adopted to estimate the source term under such stochastic noise conditions. The deep deterministic policy gradient (DDPG) is also employed to find the best source search policy in terms of successful estimation and traveled distance. Through ablation studies, we demonstrate that the use of the Gaussian mixture model, which clusters the potential source positions from the particle filter, as an input to the DDPG and the gated recurrent unit functioning as a memory in DDPG help to improve the STE performance. Besides, simulation results in randomized source term conditions and previously-unseen environments show the superior STE performance of the proposed algorithm compared with the existing information-theoretic STE algorithm.

Keywords: Reinforcement Learning, Perception for Grasping and Manipulation, Force and Tactile Sensing
Abstract: Robotic manipulation is challenging when both the objects being manipulated and the tactile sensors are deformable. In this work, we addressed the interplay between the manipulation of deformable objects, tactile sensing, and model-free reinforcement learning on a real robot. We showed how a real robot can learn to manipulate a deformable, thin-shell object via feedback from deformable, multimodal tactile sensors. We addressed the learning of a page flipping task using a two-stage approach. For the first stage, we learned nominal page flipping trajectories for two page sizes by constructing a reward function that quantifies functional task performance from the perspective of tactile sensing. For the second stage, we learned adapted trajectories using tactile-driven perceptual coupling, with an intuitive assumption that, while the page flipping trajectories for different task contexts (page sizes) might differ, similar tactile feedback should be expected from functional trajectories for each context. We also investigated the quality of information encoded by two different representations of tactile sensing data: one based on the artificial apical tuft of bio-inspired tactile sensors, and another based on PCA eigenvalues. The results and effectiveness of our learning framework were demonstrated on a real 7-DOF robot arm and gripper outfitted with tactile sensors.

Keywords: AI-Based Methods, Dynamics
Abstract: Aiming at the problem that it is difficult to calculate the force of permanent magnets in the magnetic field, this paper proposes a nonlinear mechanical model of linear array magnetic field based on radial basis function neural network (RBFNN). Combined with the linear Halbach array adsorption module of the wall-climbing robot, the three-dimensional geometric magnetic fields of four typical linear array permanent magnets were constructed, and the theoretical models of the interaction between the magnetic fields were given respectively. Further, the finite element simulation calculation of the magnetic force was carried out using COMSOL Multiphysics software. According to the parametric scanning results of the orthogonal test, a nonlinear intelligent prediction model of the force between magnetic fields with local loss sensitivity is established by using the RBFNN numerical fitting method. The average deviation of the network test set is 1.19, and the standard deviation is 0.80. The intelligent prediction model has strong general performance, faster convergence speed and stronger flexibility, which provides a theoretical basis for the interaction and control of array magnetic fields.

Keywords: Wheeled Robots, Legged Robots, Climbing Robots
Abstract: This paper presents the design, analysis, and performance evaluation of an omnidirectional transformable wheel-leg robot called OmniWheg. We design a novel mechanism consisting of a separable omni-wheel and 4-bar linkages, allowing the robot to transform between omni-wheeled and legged modes smoothly. In wheeled mode, the robot can move in all directions and efficiently adjust the relative position of its wheels, while it can overcome common obstacles in legged mode, such as stairs and steps. Unlike other articles studying whegs, this implementation with omnidirectional wheels allows the correction of misalignments between right and left wheels before traversing obstacles, which effectively improves the success rate and simplifies the preparation process before the wheel-leg transformation. We describe the design concept, mechanism, and the dynamic characteristic of the wheel-leg structure. We then evaluate its performance in various scenarios, including passing obstacles, climbing steps of different heights, and turning/moving omnidirectionally. Our results confirm that this mobile platform can overcome common indoor obstacles and move flexibly on the flat ground with the new transformable wheel-leg mechanism, while keeping a high degree of stability.

Keywords: Climbing Robots, Legged Robots, Grippers and Other End-Effectors
Abstract: This paper introduces SCALER, a quadrupedal robot that demonstrates climbing on bouldering walls, overhangs, ceilings and trotting on the ground. SCALER is one of the first high-degrees of freedom four-limbed robots that can free-climb under the Earth's gravity and one of the most mechanically efficient quadrupeds on the ground. Where other state-of-the-art climbers specialize in climbing, SCALER promises practical free-climbing with payload textit{and} ground locomotion, which realizes true versatile mobility. A new climbing gait, SKATE gait, increases the payload by utilizing the SCALER body linkage mechanism. SCALER achieves a maximum normalized locomotion speed of 1.87 /s, or 0.56 m/s on the ground and 1.0 /min, or 0.35 m/min in bouldering wall climbing. Payload capacity reaches 233 % of the SCALER weight on the ground and 35 % on the vertical wall. Our GOAT gripper, a mechanically adaptable underactuated two-finger gripper, successfully grasps convex and non-convex objects and supports SCALER.

Keywords: Climbing Robots, Grippers and Other End-Effectors, Additive Manufacturing
Abstract: Microspine grippers allow robots to ascend steep rocky slopes and cliff faces, enabling scientific exploration of exposed strata on Earth and other solar system bodies. Historically, the Shape Deposition Manufacturing (SDM) process has been used to fabricate multi-material suspensions for load-sharing among multiple microspines. We instead apply the Hybrid Deposition Manufacturing (HDM) process to microspine fabrication, and we further propose a novel 3D-printed microspine suspension design that can be manufactured via Fused Deposition Manufacturing (FDM) alone, using a single flexible material with an embedded fishhook. We use a model of microspine stiffness that allows designers to compensate for order-of-magnitude changes in material tensile modulus by adjusting geometric parameters of the design. The stiffness model and the FDM microspine design are validated through tensile testing, and mechanical properties of the HDM and FDM designs are compared against a standard SDM microspine design. We demonstrate that the FDM process can produce microspines with equivalent normal and axial stiffness and superior maximum load and fatigue response to SDM microspines, and discuss additional advantages of the FDM process for rapid prototyping and broader accessibility.

Keywords: Engineering for Robotic Systems, Simulation and Animation, Motion Control
Abstract: Spherical robots have many desirable traits when designing mass efficient systems interacted with unstructured terrain. In this paper, we propose a hybrid bionic spherical robot based on the morphological properties of sea urchins and the movement characteristics of tumbleweeds. This robot enables it to move freely in harsh terrain with the distributed high-aspect-ratio telescopic units, where our control strategy is based on the a central pattern generator and combines foot pressure measurements and actuator state information. In particular, these data from multiple foot pressure sensors also are used to compute the robots' center of mass, with evaluating locomotion state during rolling maneuvers. The experimental results show the ability of our design to provide reliable pose estimates, also overcoming a challenge in the field of mobile robotics, including switch directions flexibly in harsh terrain due to their anisotropy.

Keywords: Space Robotics and Automation
Abstract: A spherical robotic probe has several advantages in rough environments and has therefore raised interest for application in planetary exploration. A sphere is well-suited to protect high-sensitive payloads, however, the locomotion system for planetary surfaces raises several challenges. This paper presents a novel locomotion system consisting of linear actuators which are usable in a multi-functional fashion. Apart from pushing and bringing leverage for locomotion the extendable rods enable a tripod mode for improved sensing. The developed solutions offer a mathematical-physical system description, simple algorithms for the control of locomotion and balancing as well as general calculations for determining the maximum achievable performance parameters of such a robot. The first built prototype shows the basic suitability of the system and reveals directions for further research.

Keywords: Wheeled Robots, Optimization and Optimal Control, Motion and Path Planning
Abstract: Designing a cost function for nonlinear model predictive control (MPC) with a sparse/binary stage cost is challenging. This paper proposes a novel MPC approach with a scheduled quadratic stage cost function that approximates the true stage cost in order to optimally control a nonlinear system with a sparse/binary stage cost. The cost function parameter is optimally scheduled by a parameter scheduling policy obtained by solving a Markov decision process (MDP) constructed from sampled trajectories from any nonlinear MPC solver. The proposed approach is implemented into a differential drive wheeled mobile robot (WMR) designed for smart warehousing via the robot operating system (ROS) framework. The simulation and experimental results successfully demonstrate the effectiveness of our MPC approach in cases of the point stabilization problem of a differential drive WMR.

Keywords: Climbing Robots, Micro/Nano Robots
Abstract: Surface-to-surface transitions, vertical and inverted locomotion, and high payload capacity are important requirements for a climbing robot. Despite many previous approaches, the miniaturization of climbing robots that satisfy these requirements is still a big challenge. In this paper, we present an insect-scale wheeled climbing robot that employs a low-cost dry adhesive technology. The design, inspired by caterpillars, consists of 2 main links that are connected by a pivot joint. The prototype robot measures a length of 40 mm and a width of 10 mm, and it weighs 1.7 g. On a horizontal surface, the robot moves with a speed of 12.3 mm/s and can drag a load weight of 10 g (six times its weight) at a speed of 4 mm/s. The high torque-to-weight ratio achieved by two micro-geared ultrasonic motors permits vertical and inverted locomotion while carrying a high payload capacity (120% of the robot’s weight). The locomotion strategy for surface-to-surface transitions, i.e., movements at right angle corners, is substantiated by the functionality of the pivot joint. To satisfy the design requirements, many dry adhesive tapes are evaluated, and the climbing robot with an appropriate adhesion force is demonstrated.

Keywords: Product Design, Development and Prototyping, Dynamics, Optimization and Optimal Control
Abstract: Locomotion through rolling is attractive compared to other forms of locomotion thanks to uniform designs, high degree of mobility, dynamic stability, and self-recovery from collision. Despite previous efforts to design rolling soft systems, pneumatic and other soft actuators are often limited in terms of high-speed dynamics, system integration, and/or functionalities. Furthermore, mathematical description of the rolling dynamics for this type of robot and how the models can be used for speed control are often not mentioned. This article introduces a cylindrical-shaped shell-bulging rolling soft wheel that employs an array of 16 folded HASEL actuators as a mean for improved rolling performance. The actuators represent the soft components with discrete forces that propel the wheel, whereas the wheel’s frame is rigid but allows for smooth, continuous change in position and speed. We discuss the interplay between the electrical and mechanical design choices, the modeling of the wheel’s hybrid (continuous and discrete) dynamic behavior, and the implementation of a model predictive controller (MPC) for the robot’s speed. With the balance of several design factors, we show the wheel’s ability to carry integrated hardware with a maximum rolling speed at 0.7 m/s (or 2.2 body lengths per second), despite its total weight of 979 g, allowing the wheel to outperform the existing rolling soft wheels with comparable weights and sizes. We also show that the MPC enables the wheel to accelerate and leverage its inherent braking capability to reach desired speeds—a critical function that did not exist in previous rolling soft systems.

Keywords: Motion and Path Planning, Medical Robots and Systems, Optimization and Optimal Control
Abstract: Within the context of Robotic Minimally Invasive Surgery (R-MIS), we propose a novel linear model predictive controller formulation for the coordination of multiple autonomous robotic arms. The controller is synthesized by formulating a linear approximation of non-linear constraints, which allows the controller to be both computationally faster and better performing due to the increased prediction horizon allowed within the real-time control requirements for the proposed surgical application. The solution is validated under the expected constraints of a surgical scenario in which multiple laparoscopic tools must move and coordinate in a shared environment.

Keywords: Motion and Path Planning, Range Sensing, Autonomous Agents
Abstract: In this paper, we propose a new sampling-based path planning approach, focusing on the challenges linked to autonomous exploration. Our method relies on the definition of a disk graph of free-space bubbles, from which we derive a biased sampling function that expands the graph towards known free space for maximal navigability and frontiers discovery. The proposed method demonstrates an exploratory behavior similar to Rapidly-exploring Random Trees, while retaining the connectivity and flexibility of a graph-based planner. We demonstrate the interest of our method by first comparing its path planning capabilities against state-of-the-art approaches, before discussing exploration-specific aspects, namely replanning capabilities and incremental construction of the graph. A simple frontiers-driven exploration controller derived from our planning method is also demonstrated using the Pioneer platform.

Keywords: Motion and Path Planning, Field Robots
Abstract: This paper introduces a new method for robot motion planning and navigation in uneven environments through a surfel representation of underlying point clouds. The proposed method addresses the shortcomings of state-of-the-art navigation methods by incorporating both kinematic and physical constraints of a robot with standard motion planning algorithms (e.g., those from the Open Motion Planning Library), thus enabling efficient sampling-based planners for challenging uneven terrain navigation on raw point cloud maps. Unlike techniques based on Digital Elevation Maps (DEMs), our novel surfel-based state-space formulation and implementation are based on raw point cloud maps, allowing for the modeling of overlapping surfaces such as bridges, piers, and tunnels. Experimental results demonstrate the robustness of the proposed method for robot navigation in real and simulated unstructured environments. The proposed approach also optimizes planners' performances by boosting their success rates up to 5x for challenging unstructured terrain planning and navigation, thanks to our surfel-based approach's robot constraint-aware sampling strategy. Finally, we provide an open-source implementation of the proposed method to benefit the robotics community.

Keywords: Motion and Path Planning, Legged Robots, Deep Learning Methods
Abstract: Despite the progress in legged robotic locomotion, autonomous navigation in unknown environments remains an open problem. Ideally, the navigation system utilizes the full potential of the robots' locomotion capabilities while operating within safety limits under uncertainty. The robot must sense and analyze the traversability of the surrounding terrain, which depends on the hardware, locomotion control, and terrain properties. It may contain information about the risk, energy, or time consumption needed to traverse the terrain. To avoid hand-crafted traversability cost functions we propose to collect traversability information about the robot and locomotion policy by simulating the traversal over randomly generated terrains using a physics simulator. Thousand of robots are simulated in parallel controlled by the same locomotion policy used in reality to acquire 57 years of real-world locomotion experience equivalent. For deployment on the real robot, a sparse convolutional network is trained to predict the simulated traversability cost, which is tailored to the deployed locomotion policy, from an entirely geometric representation of the environment in the form of a 3D voxel-occupancy map. This representation avoids the need for commonly used elevation maps, which are error-prone in the presence of overhanging obstacles and multi-floor or low-ceiling scenarios. The effectiveness of the proposed traversability prediction network is demonstrated for path planning for the legged robot ANYmal in various indoor and natural environments.

Keywords: Motion and Path Planning, Industrial Robots, Manipulation Planning
Abstract: This work presents an online trajectory generation algorithm using a sinusoidal jerk profile. The generator takes initial acceleration, velocity and position as input, and plans a multi-segment trajectory to a goal position under jerk, acceleration, and velocity limits. By analyzing the critical constraints and conditions, the corresponding closed-form solution for the time factors and trajectory profiles are derived. The proposed algorithm was first derived in Mathematica and then converted into a C++ implementation. Finally, the algorithm was utilized and demonstrated in ROS & Gazebo using a UR3 robot. Both the Mathematica and C++ implementations can be accessed at https://github.com/Haoran-Zhao/Jerk-continuous-online-trajectory-generator-with-constraints.git

Keywords: Motion and Path Planning, Robotics and Automation in Construction, Imitation Learning
Abstract: Automated excavation is promising to improve the safety and efficiency of excavators, and trajectory planning is one of the most important techniques. In this paper, we propose a two-stage method that integrates data-driven imitation learning and model-based trajectory optimization to generate optimal trajectories for autonomous excavators. We firstly train a deep neural network using demonstration data to mimic the operation patterns of human experts under various terrain states including their geometry shape and material type. Then, for a particular terrain, we use a stochastic trajectory optimization method to improve the trajectory generated by the neural network to guarantee kinematics feasibility, improve smoothness, satisfy hard constraints, and achieve desired excavation volumes. We test the proposed algorithm on a Franka robot arm. The experimental results show that the proposed two-stage algorithm by combing expert knowledge and model optimization can increase the excavation weights by up to 24.77% meanwhile with low variance.

Keywords: Motion and Path Planning, Service Robotics, AI-Based Methods
Abstract: Area coverage is crucial for robotics applications such as cleaning, painting, exploration, and inspections. Hinged reconfigurable robots have been introduced for these application domains to improve the area coverage performance. However, the existing coverage algorithms of hinged reconfigurable robots require improvements in the aspects; consideration of beyond a limited set of reconfigurable shapes, coordinated reconfiguration and navigation, and online decision-making. Therefore, this paper proposes a novel online Complete Coverage Path Planning (CCPP) method for a hinged reconfigurable robot. The proposed CCPP method is designed with two sub-methods, the Global Coverage Path Planning (GCPP) and Local Coverage Path Planning (LCPP). The GCPP method has been implemented, adapting a Glasius Bio-inspired Neural Network (GBNN) that performs online path planning considering a fixed shape for the robot. Obstacle regions that the GCPP would not adequately cover due to access constraints are covered by the LCPP method that considers concurrent reconfiguration and navigation of the robot. A genetic algorithm determines the reconfiguration parameters that ascertain collision-free coverage and access of obstacle regions. Experimental results validate that the proposed online CCPP method is effective in ascertaining the complete area coverage in heterogeneous environments, including dynamic workspaces. Furthermore, the deployment of the LCPP method can considerably improve the coverage.

Keywords: Motion and Path Planning, Autonomous Vehicle Navigation, Deep Learning Methods
Abstract: With the Autonomous Vehicle (AV) industry shifting towards machine-learned approaches for motion planning, the performance of self-driving systems is starting to rely heavily on large quantities of expert driving demonstrations. However, collecting this demonstration data typically involves expensive HD sensor suites (LiDAR + RADAR + cameras), which quickly becomes financially infeasible at the scales required. This motivates the use of commodity sensors like cameras for data collection, which are an order of magnitude cheaper than HD sensor suites, but offer lower fidelity. Leveraging these sensors for training an AV motion planner opens a financially viable path to observe the 'long tail' of driving events.

As our main contribution we show it is possible to train a high-performance motion planner using commodity vision data which outperforms planners trained on HD-sensor data for a fraction of the cost. To the best of our knowledge, we are the first to demonstrate this using real-world data. We compare the performance of the autonomy system on these two different sensor configurations, and show that we can compensate for the lower sensor fidelity by means of increased quantity: a planner trained on 100h of commodity vision data outperforms the one with 25h of expensive HD data. We also share the engineering challenges we had to tackle to make this work.

Keywords: Motion and Path Planning, Field Robots
Abstract: Informative path planning is an important and challenging problem in robotics that remains to be solved in a manner that allows for wide-spread implementation and real-world practical adoption. Among various reasons for this, one is the lack of approaches that allow for informative path planning in high-dimensional spaces and non-trivial sensor constraints. In this work we present a sampling-based approach that allows us to tackle the challenges of large and high-dimensional search spaces. This is done by performing informed sampling in the high-dimensional continuous space and incorporating potential information gain along edges in the reward estimation. This method rapidly generates a global path that maximizes information gain for the given path budget constraints. We discuss the details of our implementation for an example use case of searching for multiple objects of interest in a large search space using a fixed-wing UAV with a forward-facing camera. We compare our approach to a sampling-based planner baseline and demonstrate how our contributions allow our approach to consistently out-perform the baseline by 18.0%. With this we thus present a practical and generalizable informative path planning framework that can be used for very large environments, limited budgets, and high dimensional search spaces, such as robots with motion constraints or high-dimensional configuration spaces.

Keywords: Biomimetics, Learning from Experience, Tendon/Wire Mechanism
Abstract: While the musculoskeletal humanoid has various biomimetic benefits, its complex modeling is difficult, and many learning control methods have been developed. However, for the actual robot, the hysteresis of its joint angle tracking is still an obstacle, and realizing target posture quickly and accurately has been difficult. Therefore, we develop a feedback control method considering the hysteresis. To solve the problem in feedback controls caused by the closed-link structure of the musculoskeletal body, we update a neural network representing the relationship between the error of joint angles and the change in target muscle lengths online, and realize target joint angles accurately in a few trials. We compare the performance of several configurations with various network structures and loss definitions, and verify the effectiveness of this study on an actual musculoskeletal humanoid, Musashi.

Keywords: Biomimetics, Soft Sensors and Actuators, Tendon/Wire Mechanism
Abstract: In this study, seated walk, a movement of walking while sitting on a chair with casters, is realized on a musculoskeletal humanoid from human teaching. The body is balanced by using buttock-contact sensors implemented on the planar interskeletal structure of the human mimetic musculoskeletal robot. Also, we develop a constrained teaching method in which one-dimensional control command, its transition, and a transition condition are described for each state in advance, and a threshold value for each transition condition such as joint angles and foot contact sensor values is determined based on human teaching. Complex behaviors can be easily generated from simple inputs. In the musculoskeletal humanoid MusashiOLegs, forward, backward, and rotational movements of seated walk are realized.

Keywords: Biologically-Inspired Robots, Biomimetics, Actuation and Joint Mechanisms
Abstract: Control of an articulated spine is important for humanoids' dynamic and balanced motion. Although there have been many spinal structures for humanoids, their actuation is still limited due to the usage of geared-motors for joints. This paper introduces position control of a distributed electromechanical spine in a vertical plane. The spine dynamics model is approximated as an open chain. Gravitational and spring torques are compensated for the control. Moreover, torque-to-current conversion for the actuator is developed. Experimental results show the implemented control of the electromechanical spine for undulatory motions.

Keywords: Grasping, Biologically-Inspired Robots, Actuation and Joint Mechanisms
Abstract: It is known that the surface microstructure, which mimics the surface of a gecko's foot, exerts a large gripping force with a tiny contact force. By applying this structure to the fingertips of a two-fingered parallel gripper, stable grasping can be achieved independent of the wetting and frictional state of the contact surface. However, when releasing an object, the adhesive force of the microstructure is large, which hinders the release of the object. In this study, we developed a releasing mechanism using a conveyor mechanism, focusing on the characteristic of the micro-protrusion structure that easily peels off in the direction of rotation. This mechanism is driven in conjunction with the gripper's grasping and releasing motions, and experiments have confirmed that it can stably release the object.

Keywords: Biomimetics, Soft Robot Applications, Dynamics
Abstract: In this paper, we describe simulations and experiments on the occurrence of cavitation with an instantaneous force generation mechanism based on the motion of the mantis shrimp. The aim of this study is to strengthen the striking force of the developed mechanism by realizing the mechanism of cavitation occurrence. This paper presents a motion and cavitation model for the developed mechanism and provides the results of the simulations and experiments considering the occurrence of cavitation. In previous studies, mantis shrimp have been shown to have two consecutive impacts on a hard object with cavitation in water. However, no robot can imitate the striking mechanism of the mantis shrimp and replicate two consecutive impacts. In this study, the mechanism was designed and developed based on a mantis shrimp model. From the experiment, it was observed that the developed mechanism caused cavitation only at the point of impact, rather than during arm swing. This result suggested that the striking force was strengthened, providing insight into the striking mechanism of the mantis shrimp.

Keywords: Soft Robot Applications, Soft Robot Materials and Design, Flexible Robotics
Abstract: Storing elastic energy in a soft body and releasing it instantly enables ultrafast movement beyond muscle capability, which small animals (mainly arthropods) realize. We applied this mechanism to a soft robot to achieve locomotion on top of a tubular surface such as a branch (i.e., upside-down brachiation). To achieve arboreal locomotion, the robot must have strong grippers to support its body and minimize the duration of time when one gripper is off the branch. By using a simulation model and prototype, we demonstrated that storing and releasing elastic energy in a soft body satisfies these requirements, allowing us to fabricate a lightweight robot. The prototype completed one step of the locomotion process in 0.22 secs, making it 2.3 times faster than the original speed of the mounted motor. In addition, we confirmed that the robot climbs 0- to 90-degree branches.

Keywords: Biomimetics, Learning from Experience, Bioinspired Robot Learning
Abstract: The musculoskeletal humanoid is difficult to modelize due to the flexibility and redundancy of its body, whose state can change over time, and so balance control of its legs is challenging. There are some cases where ordinary PID controls may cause instability. In this study, to solve these problems, we propose a method of learning a correlation model among the joint angle, muscle tension, and muscle length of the ankle and the zero moment point to perform balance control. In addition, information on the changing body state is embedded in the model using parametric bias, and the model estimates and adapts to the current body state by learning this information online. This makes it possible to adapt to changes in upper body posture that are not directly taken into account in the model, since it is difficult to learn the complete dynamics of the whole body considering the amount of data and computation. The model can also adapt to changes in body state, such as the change in footwear and change in the joint origin due to recalibration. The effectiveness of this method is verified by a simulation and by using an actual musculoskeletal humanoid, Musashi.

Keywords: Biologically-Inspired Robots, Soft Robot Applications, Hydraulic/Pneumatic Actuators
Abstract: Some marine animals, such as sea stars, have developed versatile adhesive appendages that can be used for a variety of behaviors including sticking to surfaces, manipulating objects, and locomoting. These appendages couple reversible adhesion capabilities with muscular structures that can do work on the world while still being soft enough to conform to various surfaces and to squeeze through tight spaces. We took inspiration from the hydraulic tube feet of sea star to create modules consisting of soft active suction discs and soft pneumatic linear actuators. Tube feet convert fluid motion into linear actuation using soft tubes that are constrained radially. In this work, we study the tradeoff between the stiffness of such radial constraints, and the ability of the linear actuators to apply axial forces. The adhesive force of the suction disc is dependent on both the pressure differential between the suction cavity and the environment and the deformation within the body of the disc while it is being loaded. We show that the tube foot modules are capable of creating locomotion over flat surfaces, up small steps, and in a confined space due to the redundancy caused by an array of actuators. We also show that active suction not only enables the use of softer tube foot actuators, but sensing the active suction line provides feedback to increase the efficiency of locomotion over obstacles.

Keywords: Biologically-Inspired Robots, Legged Robots, Biomimetics
Abstract: From insects to larger mammals, legged animals can be seen easily traversing a wide variety of challenging environments, by carefully selecting, reaching, and exploiting high-quality contacts with the terrain. In contrast, existing robotic foothold planning methods remain computationally expensive, often relying on exhaustive search and/or (often offline) optimization methods, thus limiting their adoption for real-life robotic deployments. In this work, we propose a low-cost, bio-inspired foothold planning method for legged robots, which replicates the mechanism of the central nervous system of legged mammals. We develop a low-level joint-space CPG model along with a high-level vision-based controller that can inexpensively predict future foothold locations and locally optimize them based on a potential field based approach. Specifically, by reasoning about the quality of ground contacts and the robot's stability through the high-level vision-based controller, our CPG model smoothly and iteratively updates relevant locomotion parameters to both optimize foothold locations and body pose, directly in the joint space of the robot for easier implementation. We experimentally validate our control model on a modular hexapod on various locomotion tasks in obstacle-rich environments as well as on stair climbing. Our results show that our method enables stabler and steadier locomotion, yielding higher-quality feedback from onboard sensors by minimizing the effect of slippage and unexpected impacts.

Keywords: Distributed Robot Systems, Search and Rescue Robots, Autonomous Vehicle Navigation
Abstract: Actively searching for targets using a multi-agent system in an unknown environment poses a two-pronged problem, where on the one hand we need agents to cover as much of the environment as possible and on the other have a higher density of agents where there are potential targets to maximize detection performance. This paper proposes a fully distributed solution for an ad hoc network of agents to cooperatively search an unknown environment and actively track found targets. The solution combines a distributed pheromone-based coverage control strategy with a distributed target selection mechanism.

Keywords: Distributed Robot Systems, Multi-Robot Systems, Sensor Fusion
Abstract: This paper presents a method for Bayesian multi-robot peer-to-peer data fusion where any pair of autonomous robots hold non-identical, but overlapping parts of a global joint probability distribution, representing real world inference tasks (e.g., mapping, tracking). It is shown that in dynamic stochastic systems, filtering, which corresponds to marginalization of past variables, results in direct and hidden dependencies between variables not mutually monitored by the robots, which might lead to an overconfident fused estimate. The paper makes both theoretical and practical contributions by providing (i) a rigorous analysis of the origin of the dependencies and (ii) a conservative filtering algorithm for heterogeneous data fusion in dynamic systems that can be integrated with existing fusion algorithms. This work uses factor graphs as both the analysis tool and the inference engine. Each robot in the network maintains a local factor graph and communicates only relevant parts of it (a sub-graph) to its neighboring robot. We discuss the applicability to various multi-robot robotic applications and demonstrate the performance using a multi-robot multi-target tracking simulation, showing that the proposed algorithm produces conservative estimates at each robot.

Keywords: Distributed Robot Systems, Swarm Robotics, Assembly
Abstract: Multi-robot systems have been shown to build large-scale, blueprint-specific structures using distributed, environmentally-mediated coordination. Little attention, however, has been devoted to error propagation and mitigation. In this paper, we introduce a detailed simulation of TERMES, a prototypical construction system, in which robots have realistic error profiles. We use this simulator and 32 randomly generated 250-brick blueprints to show that action errors can have significant long-term effects. We study the spatio-temporal error distribution and introduce and characterize the efficacy of a simple decay-based error correction mechanism. Although inefficient, this type of error correction is promising because it can be performed by robots with the same limited sensory capabilities as those who place bricks. To limit the impact on the construction rate, we also examine decay mechanisms informed by spatial and temporal error distributions. The incorporation of decay in our building process increases the probability of successful completion by ~4, at the expense of ~1/4 decrease in construction rate.

Keywords: Multi-Modal Perception for HRI, Deep Learning Methods, Human-Robot Collaboration
Abstract: We present Perceive-Represent-Generate (PRG), a novel three-stage framework that maps perceptual information of different modalities (e.g., visual or sound), corresponding to a series of instructions, to a sequence of movements to be executed by a robot. In the first stage, we perceive and preprocess the given inputs, isolating individual commands from the complete instruction provided by a human user. In the second stage we encode the individual commands into a multimodal latent space, employing a deep generative model. Finally, in the third stage we convert the latent samples into individual trajectories and combine them into a single dynamic movement primitive, allowing its execution by a robotic manipulator. We evaluate our pipeline in the context of a novel robotic handwriting task, where the robot receives as input a word through different perceptual modalities (e.g., image, sound), and generates the corresponding motion trajectory to write it, creating coherent and high-quality handwritten words.

Keywords: Distributed Robot Systems, Cooperating Robots, Reactive and Sensor-Based Planning
Abstract: In target tracking with mobile multi-sensor systems, sensor deployment impacts the observation capabilities and the resulting state estimation quality. Based on a partially observable Markov decision process (POMDP) formulation comprised of the observable sensor dynamics, unobservable target states, and accompanying observation laws, we present a distributed information-driven solution approach to the multi-agent target tracking problem, namely, sequential multi-agent nominal belief-state optimization (SMA-NBO). SMA-NBO seeks to minimize the expected tracking error via receding horizon control including a heuristic expected cost-to-go (HECTG). SMA-NBO incorporates a computationally efficient approximation of the target belief-state over the horizon. The agent-by-agent decision-making is capable of leveraging on-board (edge) compute for selecting (sub-optimal) target-tracking maneuvers exhibiting non-myopic cooperative fleet behavior. The optimization problem explicitly incorporates semantic information defining target occlusions from a world model. To illustrate the efficacy of our approach, a random occlusion forest environment is simulated. SMA-NBO is compared to other baseline approaches. The simulation results show SMA-NBO 1) maintains tracking performance and reduces the computational cost by replacing the calculation of the expected target trajectory with a single sample trajectory based on maximum a posteriori estimation; 2) generates cooperative fleet decision by sequentially optimizing single-agent policy with efficient usage of other agents' policy of intent; 3) aptly incorporates the multiple weighted trace penalty (MWTP) HECTG, which improves tracking performance with a computationally efficient heuristic.

Keywords: Natural Machine Motion, Legged Robots, Passive Walking
Abstract: Several template models have been developed to facilitate the analysis of limit-cycles for quadrupedal locomotion. The parameters in the model are usually fixed; however, biology shows that animals change their leg stiffness according to the locomotion velocity, and this adaptability invariably affects the stability of the gait. This paper provides an analysis of the influence of this variable leg stiffness on the stability of different quadrupedal gaits. The analysis exploits a simplified quadrupedal model with compliant legs and shoulder joints represented as torsional springs. This model can reproduce the most common quadrupedal gaits observed in nature. The stability of such emerging gaits is then checked. Afterward, an optimization process is used to search for the system parameters that guarantee maximum gait stability. Our study shows that using the highest feasible leg swing frequency and adopting a leg stiffness that increases with the speed of locomotion noticeably improves the gait stability over a wide range of horizontal velocities while reducing the oscillations of the trunk. This insight can be applied in the design of novel elastic quadrupedal robots, where variable stiffness actuators could be employed to improve the overall locomotion behavior.

Keywords: Multi-Modal Perception for HRI, SLAM, Integrated Planning and Learning
Abstract: Recent years have witnessed an emerging paradigm shift toward embodied artificial intelligence, in which an agent must learn to solve challenging tasks by interacting with its environment. There are several challenges in solving embodied multimodal tasks, including long-horizon planning, vision-and-language grounding, and efficient exploration. We focus on a critical bottleneck, namely the performance of planning and navigation. To tackle this challenge, we propose a Neural SLAM approach that, for the first time, utilizes several modalities for exploration, predicts an affordance-aware semantic map, and plans over it at the same time. This significantly improves exploration efficiency, leads to robust long- horizon planning, and enables effective vision-and-language grounding. With the proposed Affordance-aware Multimodal Neural SLAM (AMSLAM) approach, we obtain more than 40% improvement over prior published work on the ALFRED benchmark and set a new state-of-the-art generalization performance at a success rate of 23.48% on the test unseen scenes.

Keywords: Multi-Modal Perception for HRI, Search and Rescue Robots
Abstract: There has been a plethora of work towards improving robot perception and navigation, yet their application in hazardous environments, like during a fire or an earthquake, is still at a nascent stage. We hypothesize two key challenges here: first, it is difficult to replicate such scenarios in the real world, which is necessary for training and testing purposes. Second, current systems are not fully able to take advantage of the rich multi-modal data available in such hazardous environments. To address the first challenge, we propose to harness the enormous amount of visual content available in the form of movies and TV shows, and develop a dataset that can represent hazardous environments encountered in the real world. The data is annotated with high-level danger ratings for realistic disaster images, and corresponding keywords are provided that summarize the content of the scene. In response to the second challenge, we propose a multi-modal danger estimation pipeline for collaborative human-robot escape scenarios. Our Bayesian framework improves danger estimation by fusing information from robot's camera sensor and language inputs from the human. Furthermore, we augment the estimation module with a risk-aware planner that helps in identifying safer paths out of the dangerous environment. Through extensive simulations, we exhibit the advantages of our multi-modal perception framework that gets translated into tangible benefits such as higher success rate in a collaborative human-robot mission.

Keywords: Distributed Robot Systems, Biomimetics, Flexible Robotics
Abstract: In this study, we propose a method to reduce the traveling load by designing a distributed robot arrangement that considers the shape of a

pipeline for a cabled in-pipe mobile robot. The objective of this study

is to establish a method to reduce the friction acting on the wires of a self-propelled in-pipe mobile robot and develop a robotic system that

can inspect long-distance pipes. This study contributes the purpose of this research in that it proposes a model for estimating the tension acting on the wires of a mobile robot in a pipe and describes a method for reducing the traveling load using a distributed arrangement of robots. The friction acting on the wires in the bent pipe becomes a load for the robots moving in the pipe while towing the wires. Therefore, it is difficult for these robots to inspect complex, long-distance pipelines. Furthermore, we clarified the factors that cause friction forces in long-distance pipes and modeled and quantified

the friction forces acting on the wires in the pipes so that each factor could be addressed. Using this model, we propose a distributed deployment method, in which multiple mobile robots are deployed. By driving two robots separated by a distance, we confirmed that two robots could move at 2.5 mm/s in a pipeline that could not be moved by a single robot. Moreover, we proposed a method for controlling the robot according to the load acting on it and confirmed that the robot equipped with this control could inspect a narrow pipe with an inner diameter of 108 mm, where a load of approximately 340 N acted on the back end of the robot, with an efficiency that was approximately 1.7 times higher. This result has the potential to enable the configuration

and control of robots, which would be difficult to

Keywords: Flexible Robotics, Surgical Robotics: Planning, Surgical Robotics: Steerable Catheters/Needles
Abstract: The robotic flexible endoscope is developed rapidly in the field of surgery robots due to its high flexibility and safety. However, some inherent features, e.g., high nonlinearity, material creep, complex dynamic hysteresis behaviors, and the unknown coupling effects between bending and twisting motions, can lead to the significant degradation on three-dimensional (3-D) positioning performance of the endoscope. Aiming at these challenges, this paper built a practical multi-motion hysteresis phenomenon model for the bending and twisting motions of the robotic flexible endoscope with consideration of the coupling effects. Then, the time-optimal synchronous terminal motion planner is first proposed for the 3-D motions of the robotic endoscope to decouple the coupling effects in an intuitive separate control scheme. Finally, a series of hardware experiments are conducted on a robotic flexible ureteroscope platform. The accuracy of the proposed model and the trajectory-planning-based decoupling strategy is comprehensively validated. Particularly, the experimental results with the proposed trajectory planner show the satisfactory performance of vibration suppression and over-shoot suppression.

Keywords: Surgical Robotics: Steerable Catheters/Needles, Medical Robots and Systems, Surgical Robotics: Planning
Abstract: Structural intervention cardiology (SIC) interventions are crucial procedures for correcting heart valves, wall, and muscle form defects. However, the possibility of embolization or perforation, as well as the lack of transparent vision and autonomous surgical equipment, make it difficult for the clinician. In this paper, we propose a robot-assisted tendon-driven catheter and machine learning-based path planner to overcome these challenges. Firstly, an analytical inverse kinematic model is constructed to convert the tip location in the Cartesian space to the tendons’ displacement. Then inverse reinforcement learning algorithm is employed to calculate the the optimal path to avoid possible collisions between the catheter tip and the atrial wall. Moreover, a closed-loop feedback controller is adopted to improve positioning accuracy in a direct distal position measurement manner. Simulation and experiments are designed and conducted to demonstrate the feasibility and performance of the proposed system.

Keywords: Medical Robots and Systems, Flexible Robotics, Visual Tracking
Abstract: In image-guided robotic surgery, there exist different endoscopes coupled either with specialized surgical robots (SSRs) or general industrial robots (GIRs). In general, SSRs mechanically respect the remote-center-of-motion (RCM) constraints with directly and explicitly controllable degrees-of-freedom (DoFs), whereas GIRs meet the constraints at the algorithmic level in an implicit manner, with a loss of direct control of some RCM DoFs. Conventionally, different robotic endoscopes are treated as different monolithic systems. Then, kinematic models and control schemes are separately established to automate different robotic endoscopes. This means that a similar analysis must be followed for each system, which is tedious and time-consuming. This paper introduces a modular method to analyze the individual kinematics of the robotic holder, the endoscope shaft and the surgical endoscope with RCM constraints explicitly handled. For achieving automation, the integrated kinematics of a generic robotic endoscope is determined by combining the kinematics of the modular elements. Considering that cameras are intrinsically embedded sensors, a visual servo control scheme applicable to different robotic endoscopes is formulated to incorporate four explicit (virtual) DoFs characterizing the RCM constraints in the control law. Simulations and experiments performed using three different robotic endoscopes validate the effectiveness and practicality of the kinematic modeling and control scheme for automatic field-of-view adjustment.

Keywords: Medical Robots and Systems, Software-Hardware Integration for Robot Systems
Abstract: This paper introduces an integrated real-time intraoperative surgical guidance system, in which an endoscope camera of da Vinci surgical robot and a transrectal ultrasound (TRUS) transducer are co-registered using photoacoustic markers that are detected in both fluorescence (FL) and photoacoustic (PA) imaging. The co-registered system enables the TRUS transducer to track the laser spot illuminated by a pulsed-laser-diode attached to the surgical instrument, providing both FL and PA images of the surgical region-of-interest (ROI). As a result, the generated photoacoustic marker is visualized and localized in the da Vinci endoscopic FL images, and the corresponding tracking can be conducted by rotating the TRUS transducer to display the PA image of the marker. A quantitative evaluation revealed that the average registration and tracking errors were 0.84 mm and 1.16°, respectively. This study shows that the co-registered photoacoustic marker tracking can be effectively deployed intraoperatively using TRUS+PA imaging providing functional guidance of the surgical ROI.

Keywords: Flexible Robotics, Surgical Robotics: Steerable Catheters/Needles, Visual Servoing
Abstract: Magnetically-actuated flexible endoscopes (MAFE) have been well used in minimally-invasive surgery because they can be steered by a magnetic field thus more flexible than traditional endoscopes. Model-free and uncalibrated visual feedback control makes it possible to manipulate MAFE with a magnetic field without external tracking systems. Because no extra sensor is required to obtain position and posture information, the size of MAFE can be made smaller. However, the traditional control method focuses on 2DoF control, which lacks control over the posture of the end of MAFE. In this way, the friction between MAFE and tissue may cause injury during the advancement of the endoscope. In this paper, we propose algorithms to enhance the pose control of MAFE to 4DoF and 5DoF based on model-free and uncalibrated visual-feedback control. Experiments in structured environments verify that the control algorithms can realize 4DoF manual navigation and 5DoF automatic navigation.

Keywords: Medical Robots and Systems, Surgical Robotics: Laparoscopy, Flexible Robotics
Abstract: Existing robotic endoscopes for laparoscopic surgery, predominantly rigid or limited in dexterity, occupy a large motion space1. The large occupied motion space necessitates large incisions and reduces the motion space for surgeons to simultaneously operate other surgical instruments. Meanwhile, surgeons only have limited view adjustment capability to avoid occlusion and they often have to lift/push some organs to observe occluded target lesions in some operations such as cholecystectomy. The situation gets worse when the operations are on obese patients. In this paper, we develop a novel dexterous robotic flexible endoscope (DRFE), which is comprised of a concentric cable-driven structure and a 2-DoF articulated joint attached to the end of DRFE, for laparoscopic surgery. The proposed design occupies much less motion space both inside and outside human body as compared to conventional robotic flexible endoscopes. When used in surgery, the part of the endoscope outside the body can remain still, which reduces the risk of expanding the incision and simplifies the structure of the remote center mechanism. Simulation and experimental studies are performed to validate the effectiveness of the proposed device in the improvement of vision occlusion and usability. Initial results reveal that the DRFE is highly dexterous and accurate in observing lesions with vision occlusion.

Keywords: Surgical Robotics: Steerable Catheters/Needles, Medical Robots and Systems, Flexible Robotics
Abstract: Minimally invasive surgeries (MIS) or natural orifice transluminal endoscopic surgeries (NOTES) such as the transurethral resection of bladder tumor (TURBT) require the surgical robot to be miniaturized to perform surgical procedures in confined spaces. However, the surgical robot’s tiny size poses problems in its fabrication and shape sensing. In this paper, a miniature continuum surgical robot is proposed with a unique laminated structure which can be fabricated through a 2D lamination process and converted into 3D through folding. This multi-material laminated structure also facilitates the integration of tiny piezoelectric transducers on the robot’s surface as beacons to generate ultrasonic waves for shape detection. A novel beacon total focusing method (b-TFM) algorithm is developed to process the received ultrasonic data and create a high-quality ultrasonic image from which the shape of the continuum robot can be extracted. The proposed robot and the ultrasonic shape detection method are validated through simulations and experiments. The error in the open-loop trajectory control is less than 4 mm without compensation, and the error in the ultrasonic shape detection is less than 1 mm. This confirms the possibility of improving the trajectory control accuracy by using the detected shape as a feedback for closed-loop control.

Keywords: Medical Robots and Systems, Human-Robot Collaboration, Representation Learning
Abstract: In this study, we present an intuitive machine learning-based approach to evaluate and interpret surgical skills level of a participant working with robotic platforms. The proposed method is domain-adapted, i.e., jointly utilizes an end-to-end learning approach for smoothness detection and domain knowledge-based metrics such as fluidity and economy of motion for extracting skills-related features within a given trajectory. An advantage of our approach compared to similar stochastic or deep learning models is its intuitive and transparent manner for extraction and visualization of skills-related features within the data. We illustrate the performance of our proposed method on trials of the JIGSAWS data set as well as our own experimental data gathered from Phantom Premium 1.5A Haptic Device. This approach utilized tSNE technique and provides visualized low-dimensional representation for different trials that highlights nuanced information within the executive task and returns unusual or faulty trials as outliers far away from their normal skill or participant clusters. This information regarding the input trajectory can be used for evaluation and education applications such as learning curve analysis in surgical assessment and training programs.

Keywords: Calibration and Identification, Deep Learning in Grasping and Manipulation, Kinematics
Abstract: Peg transfer is a well-known surgical training task in the Fundamentals of Laparoscopic Surgery (FLS). While human sur-geons teleoperate robots such as the da Vinci to perform this task with high speed and accuracy, it is challenging to automate. This paper presents a novel system and control method using a da Vinci Research Kit (dVRK) surgical robot and a Zivid depth sensor, and a human subjects study comparing performance on three variants of the peg-transfer task: unilateral, bilateral without handovers, and bilateral with handovers. The system combines 3D printing, depth sensing, and deep learning for calibration with a new analytic inverse kinematics model and time-minimized motion controller. In a controlled study of 3384 peg transfer trials performed by the system, an expert surgical resident, and 9 volunteers, results suggest that the system achieves accuracy on par with the experienced surgical resident and is significantly faster and more consistent than the surgical resident and volunteers. The system also exhibits the highest consistency and lowest collision rate. To our knowledge, this is the first autonomous system to achieve “superhuman” performance on a standardized surgical task. All data is available at https://sites.google.com/view/surgicalpegtransfer.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, Recognition
Abstract: Effectively disassembling and recovering materials from Waste Electrical and Electronic Equipment (WEEE) is a critical step in moving global supply chains from carbon-intensive, mined materials to recycled and renewable ones. Traditional recycling processes rely on shredding and sorting waste streams, but for WEEE, which is comprised of numerous dissimilar materials in a small volume, we explore targeted disassembly of numerous objects for improved material recovery. Since many WEEE objects can look very similar due to common features, but their material composition and internal component layout can vary, it is critical to have an accurate classifier for subsequent disassembly steps for accurate material separation and recovery. This work introduces RGB-X, a multi-modal image classification approach, that innovatively utilizes key features from external RGB images with those generated from X-ray images to very accurately classify electronic objects. More specifically, this work develops Iterative Class Activation Mapping (iCAM), a novel network architecture that explicitly focuses on the finer-details in the multi-modal feature maps that are needed for accurate electronic object classification. Electronic objects lack large and well annotated X-ray datasets for training due to expense and need of expert guidance. To overcome this constraint, we present a novel way of creating a synthetic dataset using domain randomization applied to the X-ray domain. The combined RGBX approach gives us an accuracy of 98.6% on 10 generations of modern smartphones, which is greater than their individual accuracies of 89.1% (RGB) and 97.9% (X-ray) independently. We provide experimental results3 to corroborate our results.