Keywords: Computer Vision for Transportation
Abstract: Raquel Urtasun is the Chief Scientist of Uber ATG and the Head of Uber ATG Toronto. She is also an Associate Professor in the Department of Computer Science at the University of Toronto, a Canada Research Chair in Machine Learning and Computer Vision and a co-founder of the Vector Institute for AI. She received her Ph.D. degree from the Computer Science department at Ecole Polytechnique Federal de Lausanne (EPFL) in 2006 and did her postdoc at MIT and UC Berkeley. She is a world leading expert in AI for self-driving cars. Her research interests include machine learning, computer vision, robotics and remote sensing. Her lab was selected as an NVIDIA NVAIL lab. She is a recipient of an NSERC EWR Steacie Award, an NVIDIA Pioneers of AI Award, a Ministry of Education and Innovation Early Researcher Award, three Google Faculty Research Awards, an Amazon Faculty Research Award, a Connaught New Researcher Award, a Fallona Family Research Award and two Best Paper Runner up Prize awarded at CVPR in 2013 and 2017 respectively.

Keywords: Marine Robotics, Deep Learning in Robotics and Automation, Computer Vision for Other Robotic Applications
Abstract: This paper presents an approach for autonomous underwater robots to visually detect and identify divers. The proposed approach enables an autonomous underwater robot to detect multiple divers in a visual scene and distinguish between them. Such methods are useful for robots to identify a human leader, for example, in multi-human/robot teams where only designated individuals are allowed to command or lead a team of robots. Initial diver identification is performed using the Faster R-CNN algorithm with a region proposal network which produces bounding boxes around the divers' locations. Subsequently, a suite of spatial and frequency domain descriptors are extracted from the bounding boxes to create a feature vector. A K-Means clustering algorithm, with "k" set to the number of detected bounding boxes, thereafter identifies the detected divers based on these feature vectors. We evaluate the performance of the proposed approach on video footage of divers swimming in front of a mobile robot and demonstrate its accuracy.

Keywords: Marine Robotics, Field Robots, Localization
Abstract: Navigation and control are a largely unsolved problems for micro autonomous underwater vehicles (uAUVs). The main challenges are due to the lack of accurate underwater localization systems, which fit on-board of uAUVs. In this work, we present an integrated navigation and control architecture consisting of a low-cost embedded localization module and an underwater way-point tracking controller, which fulfills the requirements of uAUVs. The performance of the navigation and control system is benchmarked in two different experimental scenarios.

Keywords: Energy and Environment-Aware Automation, Marine Robotics, Motion and Path Planning
Abstract: This paper considers the problem of online optimal trajectory design under time-varying environments. Of particular interest is the design of energy-efficient trajectories under strong and uncertain disturbances in ocean environments and time-varying goal location. We formulate the problem within the constrained online convex optimization formalism, and a modified online gradient descent algorithm is motivated. The mobility constraints are met using a carefully chosen step-size, and the proposed algorithm is shown to incur sublinear regret. Different from the state-of-the-art algorithms that entail planning and re-planning the full trajectory using forecast data at each time instant, the proposed algorithm is entirely online and relies mostly on the current ocean velocity measurements at the vehicle locations. The trade-off between excess delay incurred in reaching the goal and the overall energy consumption is examined via numerical tests carried out on real data obtained from the regional ocean modelling system. As compared to the state-of-the-art algorithms, the proposed algorithm is not only energy-efficient but also several orders of magnitude computationally efficient.

Keywords: Marine Robotics, Planning, Scheduling and Coordination
Abstract: Operation of Autonomous Underwater Vehicles (AUVs) in large spatiotemporal missions is currently challenged by onboard energy resources requiring manned support. With current methods, AUVs are programmed to return to a static charging station based on a threshold in their energy level. Although, this approach has shown success in extending the operational life, it becomes impractical due to interruption of AUV operation and loss of energy needed to return to charging station. It is also not practical for large networks due to shortage of charging stations. We introduce mobile onsite power delivery, which will fundamentally change the range and duration of underwater operations. This paper presents a mission planning method to generate mobile charger trajectories, given pre-defined working AUV trajectories, considering environmental constraints such as currents and obstacles. The problem is formulated as a Multiple Generalized Traveling Salesman Problem (MGTSP) that is then transformed into a Traveling Salesman Problem (TSP). Energy cost in dynamic currents is integrated with a path planning algorithm using a grid-based environment model. A scheduling strategy extends the problem over multiple charging cycles. Simulation results show that the planning method significantly improves mission success and energy expenditure. Field experiments in Lake Superior validate feasibility of the planned trajectories for long-term marine missions.

Keywords: Field Robots, Marine Robotics, Human Detection and Tracking
Abstract: This paper explores the design and development of a class of robust diver detection algorithms for autonomous diver following applications. By considering the operational challenges for underwater visual tracking in diverse real-world settings, we formulate a set of desired features of a generic diver-following algorithm. We attempt to accommodate these features and maximize general tracking performance by exploiting the state-of-the-art deep object detection models. We fine-tune the building blocks of these models with a goal of balancing the trade-off between robustness and efficiency in an on-board setting under real-time constraints. Subsequently, we design an architecturally simple Convolutional Neural Network (CNN)-based diver detection model that is much faster than the state-of-the-art deep models yet provides comparable detection performances. In addition, we validate the performance and effectiveness of the proposed model through a number of diver-following experiments in closed-water and open-water environments.

Keywords: Biologically-Inspired Robots, Marine Robotics, Soft Material Robotics
Abstract: Underwater robots permit remote access to over 70% of the Earth’s surface that is covered in water for a variety of scientific, environmental, tactical, or industrial purposes. Many practical applications for robots in this setting include sensing, monitoring, exploration, reconnaissance, or inspection tasks. In the interest of expanding this activity and opportunity within aquatic environments, this paper describes a swimming robot with a simple, robust, and scalable design. RoboScallop is inspired by the locomotion of bivalve scallops, utilizing two articulating rigid shells and an elastic membrane to produce water jet propulsion. A one-DoF, reciprocating mechanism enclosed within the robot shells is used to generate pulsating thrust, and the performance of this novel swimming method is evaluated by characterization of the robot jet force and swimming speed. This is the first time jet propulsion is demonstrated for a robot swimming in normal, Newtonian fluid using a bivalve morphology. We found the robot metrics to be comparable to its biological counterpart but free from metabolic limitations which prevent sustained free swimming in living species. Leveraging this locomotion principle provides unique benefits over other existing underwater propulsion techniques, including robustness, scalability, resistance to entanglement, and possible implicit water treatment capabilities, toward further development of a new class of self-contained, hybrid-stiffness underwater robots.

Keywords: Mapping, Range Sensing, Perception for Grasping and Manipulation
Abstract: The representation of the environment strongly affects how robots can move and interact with it. This paper presents an online approach for continuous mapping using Gaussian Process Implicit Surfaces (GPIS). Compared with grid-based methods, GPIS better utilizes sparse measurements to represent the world seamlessly. It provides direct access to the signed-distance function (SDF) and its derivatives which are invaluable for other robotic tasks and it incorporates uncertainty in the sensor measurements. Our approach incrementally and efficiently updates GPIS by employing a regressor on observations and a spatial tree structure. The effectiveness of the suggested approach is demonstrated using simulations and real world 2D/3D data.

Keywords: Mapping, Semantic Scene Understanding
Abstract: Dense 3D visual mapping estimates as many as possible pixel depths, for each image. This results in very dense point clouds that often contain redundant and noisy information, especially for surfaces that are roughly planar, for instance, the ground or the walls in the scene. In this paper we leverage on semantic image segmentation to discriminate which regions of the scene require simplification and which should be kept at high level of details. We propose four different point cloud simplification methods which decimate the perceived point cloud by relying on class-specific local and global statistics still maintaining more points in the proximity of class boundaries to preserve the infra-class edges and discontinuities. 3D dense model is obtained by fusing the point clouds in a 3D Delaunay Triangulation to deal with variable point cloud density. In the experimental evaluation we have shown that, by leveraging on semantics, it is possible to simplify the model and diminish the noise affecting the point clouds.

Keywords: Mapping
Abstract: In several applications, autonomous mobile robots benefit from knowing the structure of the indoor environments where they operate. This knowledge can be extracted from the metric maps built (e.g., using SLAM algorithms) from the data perceived by the robots' sensors. The layout is a way to represent the structure of an indoor environment with geometrical primitives. Most of the current methods for reconstructing the layout from a metric map represent the parts of the environment that have been fully observed. In this paper, we propose an approach that predicts the layout of rooms which are only partially known in a 2D metric grid map. The prediction is made according to the global structure of the environment, as identified from its known parts. Experiments show that our approach is able to effectively predict the layout of several indoor environments that have been observed to different degrees.

Keywords: Mapping, SLAM, Range Sensing
Abstract: In this work, we develop a new surface reconstruction pipeline that combines monocular camera images and LiDAR measurements from a moving sensor rig to reconstruct dense 3D mesh models of indoor scenes. For surface reconstruction, the 3D LiDAR and camera are widely deployed for gathering geometric information from environments. Current state-of-the-art multi-view stereo or LiDAR-only reconstruction methods cannot reconstruct indoor environments accurately due to shortcomings of each sensor type. In our approach, LiDAR measurements are integrated into a multi-view stereo pipeline for point cloud densification and tetrahedralization. In addition to that, a graph cut algorithm is utilized to generate a watertight surface mesh. Because our proposed method leverages the complementary nature of these two sensors, the accuracy and completeness of the output model are improved. The experimental results on real world data show that our method significantly outperforms both the state-of-the-art camera-only and LiDAR-only reconstruction methods in accuracy and completeness.

Keywords: Mapping, Planning, Scheduling and Coordination, Motion and Path Planning
Abstract: Information-based mapping algorithms are critical to robot exploration tasks in several applications ranging from disaster response to space exploration. Unfortunately, most existing information-based mapping algorithms are plagued by the computational difficulty of evaluating the Shannon mutual information between potential future sensor measurements and the map. This has lead researchers to develop approximate methods, such as Cauchy-Schwarz Quadratic Mutual Information (CSQMI). In this paper, we propose a new algorithm, called Fast Shannon Mutual Information (FSMI), which is significantly faster than existing methods at computing the exact Shannon mutual information. The key insight behind FSMI is recognizing that the integral over the sensor beam can be evaluated analytically, removing an expensive numerical integration. In addition, we provide a number of approximation techniques for FSMI, which significantly improve computation time. Equipped with these approximation techniques, the FSMI algorithm is more than three orders of magnitude faster than the existing computation for Shannon mutual information; it also outperforms the CSQMI algorithm significantly, being roughly twice as fast, in our experiments.

Keywords: Mapping, Sensor Fusion, Aerial Systems: Perception and Autonomy
Abstract: In this paper, we propose a novel dense surfel mapping system that scales well in different environments with only CPU computation. Using a sparse SLAM system to estimate camera poses, the proposed mapping system can fuse intensity images and depth images into a globally consistent model. The system is carefully designed so that it can build from room-scale environments to urban-scale environments using depth images from RGB-D cameras, stereo cameras or even a monocular camera. First, superpixels extracted from both intensity and depth images are used to model surfels in the system. superpixel-based surfels make our method both run-time efficient and memory efficient. Second, surfels are further organized according to the pose graph of the SLAM system to achieve O(1) fusion time regardless of the scale of reconstructed models. Third, a fast map deformation using the optimized pose graph enables the map to achieve global consistency in real-time. The proposed surfel mapping system is compared with other state-of-the-art methods on synthetic datasets. The performances of urban-scale and room-scale reconstruction are demonstrated using the KITTI dataset and autonomous aggressive flights, respectively. The code is available for the benefit of the community.

Keywords: Cognitive Human-Robot Interaction, Semantic Scene Understanding
Abstract: The speed and accuracy with which robots are able to interpret natural language is fundamental to realizing effective human-robot interaction. A great deal of attention has been paid to developing models and approximate inference algorithms that improve the efficiency of language understanding. However, existing methods still attempt to reason over a representation of the environment that is flat and unnecessarily detailed, which limits scalability. An open problem is then to develop methods capable of producing the most compact environment model sufficient for accurate and efficient natural language understanding. We propose a model that leverages environment-related information encoded within instructions to identify the subset of observations and perceptual classifiers necessary to perceive a succinct, instruction-specific environment representation. The framework uses three probabilistic graphical models trained from a corpus of annotated instructions to infer salient scene semantics, perceptual classifiers, and grounded symbols. Experimental results on two robots operating in different environments demonstrate that by exploiting the content and the structure of the instructions, our method learns compact environment representations that significantly improve the efficiency of natural language symbol grounding.

Keywords: Social Human-Robot Interaction, Learning and Adaptive Systems
Abstract: Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically---continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.

Keywords: Visual Learning, Task Planning, Cognitive Human-Robot Interaction
Abstract: High-level human instructions often correspond to behaviors with multiple implicit steps. In order for robots to be useful in the real world, they must be able to to reason over both motions and intermediate goals implied by human instructions. In this work, we propose a framework for learning representations that convert from a natural-language command to a sequence of intermediate goals for execution on a robot. A key feature of this framework is prospection, training an agent not just to correctly execute the prescribed command, but to predict a horizon of consequences of an action before taking it. We demonstrate the fidelity of plans generated by our framework when interpreting real, crowd-sourced natural language commands for a robot in simulated scenes.

Keywords: Virtual Reality and Interfaces, Cognitive Human-Robot Interaction, Aerial Systems: Perception and Autonomy
Abstract: In this paper, we present an interface that uses natural language grounding within an MR environment to solve high-level task and navigational instructions given to an autonomous drone. To the best of our knowledge, this is the first work to perform fully autonomous language grounding in an MR setting for a robot. Given a map, our interface first grounds natural language commands to reward specifications within a Markov Decision Process (MDP) framework. Then, it passes the reward specification to an MDP solver. Finally, the drone performs the desired operations in the real world while planning and localizing itself. Our approach uses MR to provide a set of known virtual landmarks, enabling the drone to understand commands referring to objects without being equipped with object detectors for multiple novel objects or a predefined environment model. We conducted an exploratory user study to assess users’ experience of our MR interface with and without natural language, as compared to a web interface. We found that users were able to command the drone more quickly via both MR interfaces as compared to the web interface, with roughly equal system usability scores across all three interfaces.

Keywords: Social Human-Robot Interaction, Human-Centered Automation
Abstract: Scene generation is an important step of robotic drawing. Recent works have shown success in scene generation conditioned on text using a variety of approaches, with which the generated scenes cannot be revised after its generation. To allow modification on generated scenes, we model the scene generation process as a discrete event system. Instead of training text-to-pixel mappings using large datasets, the proposed approach uses object instances retrieved from the Internet to synthesize scenes. Evaluated on 128 experiments using MSCOCO evaluation dataset, the result shows the scene generation performance has been increased by 197%, 22.3%, and 55.7% compared with the state of the art approach on three standard metrics (CIDEr, ROUGH-L, METEOR), respectively. Human evaluation conducted on Amazon Mechanical Turk shows over 80% of generated scenes are considered to have higher recognizability and better alignment with natural language descriptions than baseline works.

Keywords: AI-Based Methods, Task Planning, Manipulation Planning
Abstract: We present an algorithm for combining natural language processing (NLP) and fast robot motion planning to automatically generate robot movements. Our formulation uses a novel concept called Dynamic Constraint Mapping to transform complex, attribute-based natural language instructions into appropriate cost functions and parametric constraints for optimization-based motion planning. We generate a factor graph from natural language instructions called the Dynamic Grounding Graph (DGG), which takes latent parameters into account. The coefficients of this factor graph are learned based on conditional random fields (CRFs) and are used to dynamically generate the constraints for motion planning. We map the cost function directly to the motion parameters of the planner and compute smooth trajectories in dynamic scenes. We highlight the performance of our approach in a simulated environment and via a human interacting with a 7-DOF Fetch robot using intricate language commands including negation, orientation specification, and distance constraints.

Keywords: Motion and Path Planning
Abstract: The paper proposes a path planning algorithm for cluttered environment and maze. The proposed planning algorithm exploits the merit of convex optimization while forming the convex navigable free-spaces, ensuring safety of the vehicle. The contiguous convex free-spaces are iteratively computed from a random-walk based seed generation method to create a contiguous navigable geometry. Inside this contiguous navigable geometry an undirected graph is then created, whose each node and edge belong to at least one convex region which boils down the path planning problem into a graph search problem. In addition, the proposed multiple query planning algorithm can merge the user provided feasible initial and goal configuration with the existing undirected graph in each plan, without deteriorating the planning performance in terms of run-time and path length. Simulation and experimental results confirm the superiority of the proposed planning algorithm jointly in terms of both path length and run-time by a significant margin.

Keywords: Motion and Path Planning, Dynamics, Nonholonomic Motion Planning
Abstract: The Rapidly-exploring Random Tree (RRT) algorithm has been one of the most prevalent and popular motion-planning techniques for two decades now. Surprisingly, in spite of its centrality, there has been an active debate under which conditions RRT is probabilistically complete. We provide two new proofs of probabilistic completeness (PC) of RRT with a reduced set of assumptions. The first one for the purely geometric setting, where we only require that the solution path has a certain clearance from the obstacles. For the kinodynamic case with forward propagation of random controls and duration, we only consider in addition mild Lipschitz-continuity conditions. These proofs fill a gap in the study of RRT itself. They also lay sound foundations for a variety of more recent and alternative sampling-based methods, whose PC property relies on that of RRT.

Keywords: Motion and Path Planning, Optimization and Optimal Control, Hybrid Logical/Dynamical Planning and Verification
Abstract: In this paper we propose a method to improve the accuracy of trajectory optimization for dynamic robots with intermittent contact by using orthogonal collocation. Until recently, most trajectory optimization methods for systems with contacts employ mode-scheduling, which requires an a priori knowledge of the contact order and thus cannot produce complex or non-intuitive behaviors. Contact-implicit trajectory optimization methods offer a solution to this by allowing the optimization to make or break contacts as needed, but thus far have suffered from poor accuracy. Here, we combine methods from direct collocation using higher order orthogonal polynomials with contact-implicit optimization to generate trajectories with significantly improved accuracy. The key insight is to increase the order of the polynomial representation while maintaining the physics assumption that impact occurs over the duration of one finite element.

Keywords: Motion and Path Planning, Energy and Environment-Aware Automation, Field Robots
Abstract: This paper tackles the problem of energy-efficient coverage path planning for exploring general surfaces by an autonomous vehicle. Efficient algorithms are developed to generate paths on freeform 3D surfaces according to a special design pattern as peak-aware smooth Fermat spiral for this purpose. By using the exact boundary-sourced geodesic distances, how to generate Fermat spiral paths is first introduced to cover a general surface. Then, heuristics for energy-efficiency are incorporated to add peak points of a height-field as sources for geodesic computation. Lastly, the paths are further smoothed on the given surface to avoid sharp turns. The paths generated by our method can significantly reduce the cost caused by gravity and turning. Physical experiments have been taken on different terrain surfaces to demonstrate the effectiveness of our approach.

Keywords: Motion and Path Planning, Industrial Robots, Motion Control
Abstract: Time-Optimal Path Parameterization (TOPP) is a well-studied problem in robotics and has a wide range of applications. There are two main families of methods to address TOPP: Numerical Integration (NI) and Convex Optimization (CO). NI-based methods are fast but difficult to implement and suffer from robustness issues, while CO-based approaches are more robust but at the same time significantly slower. Here we propose a new approach to TOPP based on Reachability Analysis (RA). The key insight is to recursively compute reachable and controllable sets at discretized positions on the path by solving small Linear Programs (LPs). The resulting algorithm is faster than NI-based methods and as robust as CO-based ones (100% success rate), as confirmed by extensive numerical evaluations. Moreover, the proposed approach offers unique additional benefits: Admissible Velocity Propagation and robustness to parametric uncertainty can be derived from it in a simple and natural way.

Keywords: Motion and Path Planning, Surveillance Systems, Visual Tracking
Abstract: In this paper, we address a class of visibility-based pursuit-evasion game in which a mobile observer tries to maintain a line-of-sight (LOS) with a mobile target in an environment containing obstacles. The observer knows the current position of the target as long as the target is in the observer’s LOS. At first, we address this problem in an environment containing a single corner. We formulate the game as an optimal control problem of maximizing the time for which the observer can keep the reachability set of the target in its field-of-view. Using Pontryagin’s principle, we show that the primitives for optimal motion of the observer are straight lines ( ST ) and spiral-like curves ( C ). Next, we present the synthesis of the optimal trajectories from any given initial position of the observer. We show that the optimal path of the observer belongs to the class { ST,C-ST,ST-C-ST} . Given any initial position of the target, we present a partition of the workspace around a corner based on the optimal control policy of the observer.

Keywords: Learning from Demonstration, Learning and Adaptive Systems, Physical Human-Robot Interaction
Abstract: Our goal is to enable robots to learn cost functions from user guidance. Often it is difficult or impossible for users to provide full demonstrations, so corrections have emerged as an easier guidance channel. However, when robots learn cost functions from corrections rather than demonstrations, they have to extrapolate a small amount of information – the change of a waypoint along the way – to the rest of the trajectory. We cast this extrapolation problem as online function approximation, which exposes different ways in which the robot can interpret what trajectory the person intended, depending on the function space used for the approximation. Our simulation results and user study suggest that using function spaces with non-Euclidean norms can better capture what users intend, particularly if environments are uncluttered. This, in turn, can lead to the robot learning a more accurate cost function and improves the user’s subjective perceptions of the robot.

Keywords: Learning from Demonstration, Learning and Adaptive Systems
Abstract: In this paper, we focus on generating complex robotic trajectories by merging sequential motion primitives. A robotic trajectory is a time series of positions and orientations ending at a desired target. Hence, we first discuss the generation of converging pose trajectories via dynamical systems, providing a rigorous stability analysis. Then, we present approaches to merge motion primitives which represent both the position and the orientation part of the motion. Developed approaches preserve the shape of each learned movement and allow for continuous transitions among succeeding motion primitives. Presented methodologies are theoretically described and experimentally evaluated, showing that it is possible to generate a smooth pose trajectory out of multiple motion primitives.

Keywords: Learning from Demonstration, Learning and Adaptive Systems, Force and Tactile Sensing
Abstract: The recent generation of compliant robots enables kinesthetic teaching of novel skills by human demonstration. This enables strategies to transfer tasks to the robot in a more intuitive way than conventional programming interfaces. Programming physical interactions can be achieved by manually guiding the robot to learn the behavior from the motion and force data. To let the robot react to changes in the environment, force sensing can be used to identify constraints and act accordingly. While autonomous exploration strategies in the whole workspace are time consuming, we propose a way to learn these schemes from human demonstrations in an object targeted manner. The presented teaching strategy and the learning framework allow to generate adaptive robot behaviors relying on the robot's sense of touch in a systematically changing environment. A generated behavior consists of a hierarchical representation of skills, where haptic exploration skills are used to touch the environment with the end effector, and relative manipulation skills, which are parameterized according to previous exploration events. The effectiveness of the approach has been proven in a manipulation task, where the adaptive task structure is able to generalize to unseen object locations. The robot autonomously manipulates objects without relying on visual feedback.

Keywords: Learning from Demonstration, Learning and Adaptive Systems, Physical Human-Robot Interaction
Abstract: A major goal of learning from demonstration is task generalization via observation of a teacher. In this paper, we propose a novel framework for learning motion from a single demonstration. Our approach reconstructs the demonstrated trajectory's phase space curve via a linear piece-wise regression method. We approximate dynamics of trajectory segments with linear time invariant equations, each yielding closed form solutions. We show convergence to desired phase space states via an energy-based analysis. The robustness of the model is evaluated on a robot for a sequential trajectory task. Additionally, we show the advantages that the phase space model has over the dynamic motion primitive for a kinematic based task.

Keywords: Learning from Demonstration, Learning and Adaptive Systems
Abstract: We propose a method to generate an order for learning and transferring motor skills based on motion complexity, then evaluate the order to learn motor skills of a task and transfer them to another task as a form of reinforcement learning (RL). Here, motion complexity refers to the complexity calculated from multiple motion trajectories of a task. To do this, multiple human demonstrations are extracted and clustered to calculate motion complexity and identify the motor skills involved in a task. The motion trajectories of the task are then used to calculate the motion complexity considering temporal entropy and spatial entropy. Finally, both orders [Simple-to-Complex] and [Complex-to-Simple] are generated to learn and transfer motor skills based on the motion complexities of multiple tasks. To evaluate these orders, two tasks [Drawing] and [Fitting] are performed using an actual robotic arm. To verify the learning and transfer processes, we apply our method to three different figures as well as to pegs and holes of three different shapes and analyze the experimental results. In addition, we provide guidelines for using the [Simple-to-Complex] and [Complex-to-Simple] orders in RL.

Keywords: Learning from Demonstration, Learning and Adaptive Systems, Humanoid Robots
Abstract: Bimanual operations in humanoids offer the possibility to carry out more than one manipulation task at the same time, which in turn introduces the problem of task prioritization. We address this problem from a learning from demonstration perspective, by extending the task-parameterized Gaussian mixture model to Jacobian and null space structures. The proposed approach is tested on bimanual skills but can be applied in any scenario where the prioritization between potentially conflicting tasks needs to be learned. We evaluate the proposed framework in: two different tasks with humanoids requiring the learning of priorities and a loco-manipulation scenario, showing that the approach can be exploited to learn the prioritization of multiple tasks in parallel.

Keywords: Semantic Scene Understanding, Deep Learning in Robotics and Automation, Performance Evaluation and Benchmarking
Abstract: We present a method for improving segmentation tasks on images affected by adherent rain drops and streaks. We introduce a novel stereo dataset recorded using a system that allows one lens to be affected by real water droplets while keeping the other lens clear. We train a denoising generator using this dataset and show that it is effective at removing the effect of real water droplets, in the context of image reconstruction and road marking segmentation. To further test our de-noising approach, we describe a method of adding computer-generated adherent water droplets and streaks to any images, and use this technique as a proxy to demonstrate the effectiveness of our model in the context of general semantic segmentation. We benchmark our results using the CamVid road marking segmentation dataset, Cityscapes semantic segmentation datasets and our own real-rain dataset, and show significant improvement on all tasks.

Keywords: Object Detection, Segmentation and Categorization, Semantic Scene Understanding, Deep Learning in Robotics and Automation
Abstract: The ability to interpret a scene is an important capability for a robot that is supposed to interact with its environment. The knowledge of what is in front of the robot is, for example, relevant for navigation, manipulation, or planning. Semantic segmentation labels each pixel of an image with a class label and thus provides a detailed semantic annotation of the surroundings to the robot. Convolutional neural networks (CNNs) are popular methods for addressing this type of problem. The available software for training and the integration of CNNs for real robots, however, is quite fragmented and often difficult to use for non-experts, despite the availability of several high-quality open-source frameworks for neural network implementation and training. In this paper, we propose a tool called Bonnet, which addresses this fragmentation problem by building a higher abstraction that is specific for the semantic segmentation task. It provides a modular approach to simplify the training of a semantic segmentation CNN independently of the used dataset and the intended task. Furthermore, we also address the deployment on a real robotic platform. Thus, we do not propose a new CNN approach in this paper. Instead, we provide a stable and easy-to-use tool to make this technology more approachable in the context of autonomous systems. In this sense, we aim at closing a gap between computer vision research and its use in robotics research.

Keywords: Visual Learning, Semantic Scene Understanding, SLAM
Abstract: Deployment of deep learning models in robotics as sensory information extractors can be a daunting task to handle, even using generic GPU cards. Here, we address three of its most prominent hurdles, namely, i) the adaptation of a single model to perform multiple tasks at once (in this work, we consider depth estimation and semantic segmentation crucial for acquiring geometric and semantic understanding of the scene), while ii) doing it in real-time, and iii) using asymmetric datasets with uneven numbers of annotations per each modality. To overcome the first two issues, we adapt a recently proposed real-time semantic segmentation network, making changes to further reduce the number of floating point operations. To approach the third issue, we embrace a simple solution based on hard knowledge distillation under the assumption of having access to a powerful `teacher' network. We showcase how our system can be easily extended to handle more tasks, and more datasets, all at once, performing depth estimation and segmentation both indoors and outdoors with a single model. Quantitatively, we achieve results equivalent to (or better than) current state-of-the-art approaches with one forward pass costing just 13ms and 6.5 GFLOPs on 640x480 inputs. This efficiency allows us to directly incorporate the raw predictions of our network into the SemanticFusion framework for dense 3D semantic reconstruction of the scene.

Keywords: Semantic Scene Understanding, Visual-Based Navigation, SLAM
Abstract: We propose a system for visual simultaneous localization and mapping (SLAM) that combines traditional local appearance-based features with semantically meaningful object landmarks to achieve both accurate local tracking and highly view-invariant object-driven relocalization. Our mapping process uses a sampling-based approach to efficiently infer the 3D pose of object landmarks from 2D bounding box object detections. These 3D landmarks then serve as a view-invariant representation which we leverage to achieve camera relocalization even when the viewing angle changes by more than 125 degrees. This level of view-invariance cannot be attained by local appearance-based features (e.g. SIFT) since the same set of surfaces are not even visible when the viewpoint changes significantly. Our experiments show that even when existing methods fail completely for viewpoint changes of more than 70 degrees, our method continues to achieve a relocalization rate of around 90%, with a mean rotational error of around 8 degrees.

Keywords: Automation at Micro-Nano Scales, Automation in Life Sciences: Biotechnology, Pharmaceutical and Health Care, Visual Tracking
Abstract: Automatic targeting of plant cells to perform tasks like extraction of chloroplast is often desired in the study of plant biology. Hence, this paper proposes an improved cell segmentation method combined with a robust tracking algorithm for vision-guided micromanipulation in plant cells. The objective of this work is to develop an automatic plant cell detection and localization technique to complete the automated workflow for plant cell manipulation. The complex structural properties of plant cells make both segmentation of cells and visual tracking of the microneedle immensely challenging, unlike single animal cell applications. Thus, an improved version of watershed segmentation with adaptive thresholding is proposed to detect the plant cells without the need for staining of the cells or additional tedious preparations. To manipulate the needle to reach the identified centroid of the cells, tracking of the needle tip is required. Visual and motion information from two data sources namely, template tracking and projected manipulator trajectory are combined using score-based normalized weighted averaging to continuously track the microneedle. Experimental results validate the effectiveness of the proposed method by detecting plant cell centroids accurately, tracking the microneedle constantly and reaching the plant cell of interest despite the presence of visual disturbances.

Keywords: SLAM
Abstract: Simultaneous Localization And Mapping (SLAM) is a fundamental problem in mobile robotics. While sparse point-based SLAM methods provide accurate camera localization, the generated maps lack semantic information. On the other hand, state of the art object detection methods provide rich information about entities present in the scene from a single image. This work incorporates a real-time deep-learned object detector to the monocular SLAM framework for representing generic objects as quadrics that permit detections to be seamlessly integrated while allowing the real-time performance. Finer reconstruction of an object, learned by a CNN network, is also incorporated and provides a shape prior for the quadric leading further refinement. To capture the dominant structure of the scene, additional planar landmarks are detected by a CNN-based plane detector and modelled as independent landmarks in the map. Extensive experiments support our proposed inclusion of semantic objects and planar structures directly in the bundle-adjustment of SLAM - textit{Semantic SLAM} - that enriches the reconstructed map semantically, while significantly improving the camera localization.

Keywords: SLAM, RGB-D Perception, Object Detection, Segmentation and Categorization
Abstract: We present a real-time dense mapping system which uses the predicted 2D semantic labels for optimizing the geometric quality of reconstruction. With a combination of CNN for 2D labeling and a SLAM system for camera trajectory estimation, recent approaches have succeeded in incrementally fusing and labeling 3D scenes. However, the geometric quality of the reconstruction can be further improved by incorporating such semantic prediction results, which is not sufficiently exploited by existing methods. In this paper, we propose to use semantic information to improve two crucial modules in the reconstruction pipeline, namely tracking and loop detection, for obtaining mutual benefits in geometric reconstruction and semantic recognition. Specifically for tracking, we use a novel probabilistic projective association approach to efficiently pick out candidate correspondences, where the confidence of these correspondences is quantified concerning similarities on all available short-term invariant features. For the loop detection, we incorporate these semantic labels into the original encoding through Randomized Ferns to generate a more comprehensive representation for retrieving candidate loop frames. Evaluations on a publicly available synthetic dataset have shown the effectiveness of our approach that considers such semantic hints as a reliable feature for achieving higher geometric quality.

Keywords: SLAM, Deep Learning in Robotics and Automation, Visual Learning
Abstract: Place recognition and loop closure detection are challenging for long-term visual navigation tasks. SeqSLAM is considered to be one of the most successful approaches to achieving long-term localization under varying environmental conditions and changing viewpoints. It depends on a brute-force, time-consuming sequential matching method. We propose MRS-VPR, a multi-resolution, sampling-based place recognition method, which can significantly improve the matching efficiency and accuracy in sequential matching. The novelty of this method lies in the coarse-to-fine searching pipeline and a particle filter-based global sampling scheme, that can balance the matching efficiency and accuracy in the long-term navigation task. Moreover, our model works much better than SeqSLAM when the testing sequence has a much smaller scale than the reference sequence. Our experiments demonstrate that the proposed method is efficient in locating short temporary trajectories within long-term reference ones without losing accuracy compared to SeqSLAM.

Keywords: SLAM, RGB-D Perception, Probability and Statistical Methods
Abstract: When registering 3-D point clouds it is expected that some points in one cloud do not have corresponding points in the other cloud. These non-correspondences are likely to occur near one another, as surface regions visible from one sensor pose are obscured or out of frame for another. In this work, a hidden Markov random field model is used to capture this prior within the framework of the iterative closest point algorithm. The EM algorithm is used to estimate the distribution parameters and learn the hidden component memberships. Experiments are presented demonstrating that this method outperforms several other outlier rejection methods when the point clouds have low or moderate overlap.

Keywords: SLAM, Localization
Abstract: Registering models is an essential building block of many robotic applications. In case of 3D data, the models to be aligned usually consist of point clouds. In this work we propose a formalism to represent in a uniform manner scenes consisting of high-level geometric primitives, including lines and planes. Additionally, we derive both an iterative and a direct method to determine the transformation between heterogeneous scenes (solver). We analyzed the convergence behavior of this solver on synthetic data. Furthermore, we conducted comparative experiments on a full registration pipeline that operates on raw data, implemented on top of our solver. To this extent we used public benchmark datasets and we compared against state-of-the-art approaches. Finally, we provide an implementation of our solver together with scripts to ease the reproduction of the results presented in this work.

Keywords: SLAM, Mapping, Multi-Robot Systems
Abstract: Pose graph optimization is a common problem in robotics and associated fields. Most commonly, pose graph optimization is performed by finding

the set of pose estimates which are the ``most likely'' for given set of measurements. In some situations, arbitrarily large errors in pose graph initialization are unavoidable and can cause these pose-based methods to diverge or fail. This paper details the parameterization of

the classic pose graph problem in a relative context, optimizing directly over relative edge constraints between vertices in the pose graph and not on the poses themselves. Unlike previous literature on relative optimization, this paper details relative optimization over an

entire pose graph, instead of a subset of edges, resulting in increased

robustness to arbitrarily large errors than the classic pose-based optimization or prior relative. Several small-scale simulation comparison studies and a single and multi-agent hardware experiment are

presented with results pointing to Relative Edge Optimization as a strong candidate to solving real-world pose graph optimization problems

that contain arbitrarily large initialization errors that have proven problematic to the global approach thus far.

Keywords: AI-Based Methods, Autonomous Agents, Planning, Scheduling and Coordination
Abstract: We present a general policy formulation for partially observable Markov decision processes (POMDPs) called controller family policies that may be used as a framework to facilitate the design of new policy forms. We prove how modern approximate policy forms: point-based, finite state controller (FSC), and belief compression, are instances of this family of generalized controller policies. Our analysis provides a deeper understanding of the POMDP model and suggests novel ways to design POMDP solutions that can combine the benefits of different state-of-the-art methods. We illustrate this capability by creating a new customized POMDP policy form called the belief-integrated FSC (BI-FSC) tailored to overcome the shortcomings of a state-of-the-art algorithm that uses non-linear programming (NLP). Specifically, experiments show that for NLP the BI-FSC offers improved performance over a vanilla FSC-based policy form on benchmark domains. Furthermore, we demonstrate the BI-FSC’s execution on a real robot navigating in a maze environment. Results confirm the value of using the controller family policy as a framework to design customized policies in POMDP robotic solutions.

Keywords: AI-Based Methods, Learning and Adaptive Systems, Optimization and Optimal Control
Abstract: This paper presents a novel model-free Reinforcement Learning algorithm for learning behavior in continuous action, state, and goal spaces. The algorithm approximates optimal value functions using non-parametric estimators. It is able to efficiently learn to reach multiple arbitrary goals in deterministic and nondeterministic environments. To improve generalization in the goal space, we propose a novel sample augmentation technique. Using these methods robots learn faster and overall better controllers. We benchmark the performance using simulation and a real world voltage control robot that learns to control a Cartesian task space without direct sensori access.

Keywords: AI-Based Methods, Path Planning for Multiple Mobile Robots or Agents, Optimization and Optimal Control
Abstract: Belief space planning (BSP) is a fundamental problem in robotics. Determining an optimal action quickly grows intractable as it involves calculating the expected accumulated cost (reward), where the expectation accounts for all future measurement realizations. State of the art approaches therefore resort to simplifying assumptions and approximations to reduce computational complexity. Importantly, while in robotics re-planning is essential, these approaches calculate each planning session from scratch. In this work we contribute a novel approach, iX-BSP, that is based on the key insight that calculations in consecutive planning sessions are similar in nature and can be thus re-used. Our approach performs incremental calculation of the expectation by appropriately re-using computations already performed in a precursory planing session while accounting for the information obtained in inference between the two planning sessions. The formulation of our approach considers general distributions and accounts for data association aspects. We evaluate iX-BSP in statistical simulation and show incremental expectation calculations significantly reduce runtime without impacting performance.

Keywords: AI-Based Methods, Legged Robots, Force and Tactile Sensing
Abstract: Mobile robots are becoming very popular in real-world outdoors applications, where there are many challenges in robot control and perception. One of the most critical problems is to characterise the terrain traversed by the robot. This knowledge is indispensable for optimal terrain negotiation. Currently, most approaches are performing terrain classification from vision, but there is not enough research on terrain identification from a direct interaction of the robot with the environment. In our work, we proposed new methods for classification of force/torque data from an interaction of the legged robot foot with the ground, gathered during the walking process. We provided machine learning methods for terrain classification from raw force/torque signals for which we achieved 93% accuracy on a challenging dataset with 160 minutes of recorded fixed-length steps. We also worked on a dataset where the assumption of a fixed-length step is not valid. In this case, the final result is around 80% of accuracy. The most important fact is that the data in both cases was recorded while the robot was walking, no particular movements or controlled environment were needed. Additionally, we also proposed a clustering method which allows us to learn about the class membership based on the recorded data only, without any human supervision.

Keywords: AI-Based Methods
Abstract: A core capability of robots is to reason about multiple objects under uncertainty. Partially Observable Markov Decision Processes (POMDPs) provide a means of reasoning under uncertainty for sequential decision making, but are computationally intractable in large domains. In this paper, we propose Object-Oriented POMDPs (OO-POMDPs), which represent the state and observation spaces in terms of classes and objects. The structure afforded by OO-POMDPs support a factorization of the agent's belief into independent object distributions, which enables the size of the belief to scale linearly versus exponentially in the number of objects. We formulate a novel Multi-Object Search (MOS) task as an OO-POMDP for mobile robotics domains in which the agent must find the locations of multiple objects. Our solution exploits the structure of OO-POMDPs by featuring human language to selectively update the belief at task onset. Using this structure, we develop a new algorithm for efficiently solving OO-POMDPs: Object-Oriented Partially Observable Monte-Carlo Planning (OO-POMCP). We show that OO-POMCP with grounded language commands is sufficient for solving challenging MOS tasks both in simulation and on a physical mobile robot.

Keywords: AI-Based Methods, RGB-D Perception, Range Sensing
Abstract: In this work, we propose the Depth Generation Network (DGN) to address the problem of dense depth estimation by exploiting the variational method and the deep-learning technique. In particular, we focus on improving the feasibility of depth estimation under complex scenarios given stereo RGB images, where the stereo pairs and/or depth ground-truth captured by real sensors may be deteriorated; the stereo setting parameters may be unavailable or unreliable, hence hamper efforts to establish the correspondence between image pairs via supervision learning or epipolar geometric cues. Instead of relying on real data, we supervise the training of our model using synthetic depth maps generated by the simulator, which deliver complex scenes and reliable data with ease. Two non-trivial challenges, i.e., (i) attaining reasonable amount yet realistic samples for training, and (ii) developing a model that adapts to both synthetic and real scenes arise, whereas in this work we mainly deal with the later one yet leveraging state-of-the-art Falling Things (FAT) dataset to overcome the first. Experiments on FAT and KITTI datasets demonstrate that our model estimates relative dense depth in fine details, potentially generalizable to real scenes without knowing the stereo geometric and optic settings.

Keywords: Perception for Grasping and Manipulation, RGB-D Perception, Computer Vision for Automation
Abstract: Template matching has been shown to accurately estimate the pose of a new object given a limited number of samples. However, pose estimation of occluded objects is still challenging. Furthermore, many robot application domains encounter texture-less objects for which depth images are more suitable than color images. In this paper, we propose a novel framework, Multi-Task Template Matching (MTTM), that finds the nearest template of a target object from a depth image while predicting segmentation masks and a pose transformation between the template and a detected object in the scene using the same feature map of the object region. The proposed feature comparison network computes segmentation masks and pose predictions by comparing feature maps of templates and cropped features of a scene. The segmentation result from this network improves the robustness of the pose estimation by excluding points that do not belong to the object. Experimental results show that MTTM outperforms baseline methods for segmentation and pose estimation of occluded objects despite using only depth images.

Keywords: Perception for Grasping and Manipulation, Dual Arm Manipulation
Abstract: In this paper we present a novel method of categorizing naturalistic human arm motions during activities of daily living using clustering techniques. While many current approaches attempt to define all arm motions using heuristic interpretation, or a combination of several abstract motion primitives, our unsupervised approach generates a hierarchical description of natural human motion with well recognized groups. Reliable recommendation of a subset of motions for task achievement is beneficial to various fields, such as robotic and semi-autonomous prosthetic device applications. The proposed method makes use of well-known techniques such as dynamic time warping (DTW) to obtain a divergence measure between motion segments, DTW barycenter averaging (DBA) to get a motion average, and Ward’s distance criterion to build the hierarchical tree. The clusters that emerge summarize the variety of recorded motions into the following general tasks: reach-to-front, transfer-box, drinking from vessel, on-table motion, turning a key or door knob, and reach-to-back pocket. The clustering methodology is justified by comparing against an alternative measure of divergence using Bezier coefficients and K-medoids clustering.

Keywords: Perception for Grasping and Manipulation
Abstract: Robots working in human environments often encounter a wide range of articulated objects, such as tools, cabinets, and other jointed objects. Such articulated objects can take an infinite number of possible poses, as a point in a potentially high-dimensional continuous space. A robot must perceive this continuous pose in order to manipulate the object to a desired pose. This problem of perception and manipulation of articulated objects remains a challenge due to its high dimensionality and multi-modal uncertainty. In this paper, we propose a factored approach to estimate the poses of articulated objects using an efficient nonparametric belief propagation algorithm. We consider inputs as geometrical models with articulation constraints, and observed 3D sensor data. The proposed framework produces object-part pose beliefs iteratively. The problem is formulated as a pairwise Markov Random Field (MRF) where each hidden node (continuous pose variable) models an observed object-part's pose and each edge denotes an articulation constraint between a pair of parts. We propose articulated pose estimation by a Pull Message Passing algorithm for Nonparametric Belief Propagation (PMPNBP) and evaluate its convergence properties over scenes with articulated objects.

Keywords: Perception for Grasping and Manipulation, Simulation and Animation
Abstract: Accurate state estimation is a fundamental component of robotic control. In robotic manipulation tasks, as is our focus in this work, state estimation is essential for identifying the positions of objects in the scene, forming the basis of the manipulation plan. However, pose estimation typically requires expensive 3D cameras or additional instrumentation such as fiducial markers to perform accurately. Recently, Tobin et al. introduced an approach to pose estimation based on domain randomization, where a neural network is trained to predict pose directly from a 2D image of the scene. The network is trained on computer generated images with a high variation in textures and lighting, thereby generalizing to real world images. In this work, we investigate how to improve the accuracy of domain randomization based pose estimation. Our main idea is that active perception -- moving the robot to get a better estimate of pose -- can be trained in simulation and transferred to real using domain randomization. In our approach, the robot trains in a domain-randomized simulation how to estimate pose from a sequence of images.

We show that our approach can significantly improve the accuracy of standard pose estimation in several scenarios: when the robot holding an object moves, or when reference objects are moved in the scene.

Keywords: Perception for Grasping and Manipulation, Visual Learning, Grasping
Abstract: Recent progress of deep learning improved the capability of a robot to find a proper grasp of a novel object for different grasp modalities (e.g., pinch and suction). While these previous studies consider multiple modalities separately, several studies develop multi-modal grippers that can achieve simultaneous pinch and suction grasp (multi-modal grasp fusion) for more capable and stable object manipulation. However, the previous studies with these grippers restrict the situations: simple object geometry and uncluttered environments. To overcome these difficulties, we propose a system that consists of: 1) object-class-agnostic grasp modality detection; 2) object-class-agnostic instance segmentation; and 3) grasp template matching for different modalities. The key idea of our work is the introduction of instance segmentation to fuse multiple modalities regarding each instance eluding a grasp of multiple objects at once. In the experiments, we evaluated the proposed system on the real-world picking task in clutter. The experimental results show that the effectiveness of modality detection, instance segmentation, and the integrated system as a whole.

Keywords: Learning and Adaptive Systems
Abstract: Scarce data is a major challenge to scaling robot learning to truly complex tasks, as we need to generalize locally learned policies over different task contexts. Contextual policy search offers data-efficient learning and generalization by explicitly conditioning the policy on a parametric context space. In this paper, we further structure the contextual policy representation. We propose to factor contexts into two components: target contexts that describe the task objectives, e.g. target position for throwing a ball; and environment contexts that characterize the environment, e.g. initial position or mass of the ball. Our key observation is that experience can be directly generalized over target contexts. We show that this can be easily exploited in contextual policy search algorithms. In particular, we apply factorization to a Bayesian optimization approach to contextual policy search both in sampling-based and active learning settings. Our simulation results show faster learning and better generalization in various robotic domains. See our supplementary video: https://youtu.be/MNTbBAOufDY.

Keywords: Object Detection, Segmentation and Categorization, Computer Vision for Transportation, Deep Learning in Robotics and Automation
Abstract: We present structured domain randomization (SDR), a variant of domain randomization (DR) that takes into account the structure of the scene in order to add context to the generated data. In contrast to DR, which places objects and distractors randomly according to a uniform probability distribution, SDR places objects and distractors randomly according to probability distributions that arise from the specific problem at hand. In this manner, SDR-generated imagery enables the neural network to take the context around an object into consideration during detection. We demonstrate the power of SDR for the problem of 2D bounding box car detection, achieving competitive results on real data after training only on synthetic data. On the KITTI easy, moderate, and hard tasks, we show that SDR outperforms other approaches to generating synthetic data (VKITTI, Sim 200k, or DR), as well as real data collected in a different domain (BDD100K). Moreover, synthetic SDR data combined with real KITTI data outperforms real KITTI data alone.

Keywords: Probability and Statistical Methods, Learning and Adaptive Systems, Optimization and Optimal Control
Abstract: This paper proposes a probabilistic approach for object search in clutter. Due to heavy occlusions, it is vital for an agent to be able to gradually reduce uncertainty in observations of the objects in its workspace by systematically rearranging them. Probabilistic methodologies present a promising sample-efficient alternative to handle the massively complex state-action space that inherently comes with this problem, avoiding the need for both exhaustive training samples and the accompanying heuristics for traversing a large-scale model during runtime. We approach the object search problem by extending a Gaussian Process active filtering strategy with an additional model for capturing state dynamics as the objects are moved over the course of the activity. This allows viable models to be built upon relatively scarce training data, while the complexity of the action space is also reduced by shifting objects over relatively short distances. Validation in both simulation and with a real Baxter robot with a limited number of training samples demonstrates the efficacy of the proposed approach.

Keywords: Object Detection, Segmentation and Categorization, Visual Learning, Computer Vision for Other Robotic Applications
Abstract: Object classification is an important capability for robots as it provides vital semantic information that underpin most practical high-level tasks. Classic handcrafted features, such as point pair features, have demonstrated their robustness for this task. Combining these features with modern deep learning methods provide discriminative features that are rotation invariant and robust to various sources of noise. In this work, we aim to improve the descriptiveness of point pair features while retaining their robustness. We propose a method to achieve more structured sampling of pairs and combine this information through the use of graph convolutional networks. We introduce a novel attention model based on a repeatable local reference frame. Experiments show that our approach significantly improves the state of the art for object classification on large scale reconstruction such as the Stanford 3D indoor dataset and ScanNet and obtains competitive accuracy on the artificial dataset ModelNet.

Keywords: Object Detection, Segmentation and Categorization, AI-Based Methods, Computer Vision for Automation
Abstract: Despite the recent active research on processing point clouds with deep networks, few attention has been on the sensitivity of the networks to rotations. In this paper, we propose a deep learning architecture that achieves discrete SO(2)/SO(3) rotation equivariance for point cloud recognition. Specifically, the rotation of an input point cloud with elements of a rotation group is similar to shuffling the feature vectors generated by our approach. The equivariance is easily reduced to invariance by eliminating the permutation with operations such as maximum or average. Our method can be directly applied to any existing point cloud based networks, resulting in significant improvements in their performance for rotated inputs. We show state-of-the-art results in the classification tasks with various datasets under both SO(2) and SO(3) rotations. In addition, we further analyze the necessary conditions of applying our approach to PointNet based networks.

Keywords: Object Detection, Segmentation and Categorization, Computer Vision for Other Robotic Applications
Abstract: Many recent works on 3D object detection have focused on designing neural network architectures that can consume point cloud data. While these approaches demonstrate encouraging performance, they are typically based on a single modality and are unable to leverage information from other modalities, such as a camera. Although a few approaches fuse data from different modalities, these methods either use a complicated pipeline to process the modalities sequentially, or perform late-fusion and are unable to learn interaction between different modalities at early stages. In this work, we present PointFusion and VoxelFusion: two simple yet effective early-fusion approaches to combine the RGB and point cloud modalities, by leveraging the recently introduced VoxelNet architecture. Evaluation on the KITTI dataset demonstrates significant improvements in performance over approaches which only use point cloud data. Furthermore, the proposed method provides results competitive with the state-of-the-art multimodal algorithms, achieving top-2 ranking in five of the six bird’s eye view and 3D detection categories on the KITTI benchmark, by using a simple single stage network.

Keywords: Object Detection, Segmentation and Categorization, Computer Vision for Automation
Abstract: Recent computer vision research has demonstrated that Mask R-CNN can be trained to segment specific categories of objects in RGB images when massive hand-labeled datasets are available. As generating these datasets is time-consuming, we instead train with synthetic depth images. Many robots use depth sensors, and recent results suggest training on synthetic depth data can transfer successfully to the real world. We present a method for automated dataset generation and rapidly generate a synthetic training dataset of 50,000 depth images and 320,000 object masks using simulated heaps of 3D CAD models. We train a variant of Mask R-CNN with domain randomization on the generated dataset to perform category-agnostic instance segmentation without any hand-labeled data. We evaluate the trained network, which we refer to as Synthetic Depth (SD) Mask R-CNN, on a set of real, high-resolution depth images of challenging, densely-cluttered bins containing objects with highly-varied geometry. SD Mask R-CNN outperforms point cloud clustering baselines by an absolute 15% in Average Precision and 20% in Average Recall on COCO benchmarks, and achieves performance levels similar to a Mask R-CNN trained on a massive, hand-labeled RGB dataset and fine-tuned on real images from the experimental setup. We deploy the model in an instance-specific grasping pipeline to demonstrate its usefulness in a robotics application. See https://bit.ly/2letCuE for code, datasets, and supplementary material.

Keywords: Manipulation Planning, Task Planning, Motion and Path Planning
Abstract: Robots require novel reasoning systems to achieve complex objectives in new environments. Everyday activities in the physical world couple discrete and continuous reasoning. For example, to set the table in Fig. 1, the robot must make discrete decisions about which objects to pick and the order in which to do so, and execute these decisions by computing continuous motions to reach objects or desired locations. Robotics has traditionally treated these issues in isolation. Reasoning about discrete events is referred to as task planning while reasoning about and computing continuous motions is the realm of motion planning. However, several recent works have shown that separating task planning from motion planning---that is finding first a series of actions that will later be executed through continuous motion---is problematic; for example, the next discrete action may specify picking an object, but there may be no continuous motion for the robot to bring its hand to a configuration that can actually grasp the object to pick it up. Instead, Task--Motion Planning (TMP) tightly couples task planning and motion planning, producing a sequence of steps that can actually be executed by a real robot to bring the world from an initial to a final state. This article provides an introduction to TMP and discusses the implementation and use of an open-source TMP framework that is adaptable to new robots, scenarios, and algorithms.

Keywords: Marine Robotics, Grasping, Grippers and Other End-Effectors
Abstract: Current underwater end-effector technology has limits in terms of finesse and versatility. Because of this, the execution of several underwater operations, such as archeological recovery and biological sampling, often still requires direct intervention by human operators, exposing them to the risks of working in a difficult environment. This article proposes the design and implementation of an underactuated and compliant underwater end effector that embodies grasp capabilities comparable to those of a scuba's real hand as well as the large grasping envelope of grippers.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications, Service Robots
Abstract: This work reports on the development and evaluation of an aerial system for active tool handling on remote locations. In the proposed approach a multirotor UAV is responsible for moving an end-effector with a tool to the region of interest and providing sufficient contact force for the end-effector to accomplish the desired task. The end-effector is equipped with actuated wheels that rely on the contact force to both allow an operator to re-position while in contact with the environment and perform the tool operation.

Preliminary experiments validate the approach in a cleaning scenario and demonstrate the repeatability in an experiment with 18 consecutive repetitions of the approach.

Keywords: Aerial Systems: Applications, Aerial Systems: Mechanics and Control, Cooperative Manipulators
Abstract: This paper highlights the key components of Tele-MAGMaS and demonstrates practically the implementation of the described algorithm in software and hardware for aerial-ground co-manipulation. To the best of our knowledge, this is the first time a Flying Assistant has been implemented. In particular, for the first time a cooperative manipulation task between a ground industrial manipulator and an aerial manipulator has been robustly demonstrated. Tele-MAGMaS necessitated to develop software in a modular way in order to allow easier development, this required the careful design of the full software architecture used in this work, which implements a control framework allowing for the different modalities of i) full autonomy, ii) tele-operation and iii) shared control of the system, thus allowing the system to cope with the different complexity levels of various environments by also leveraging (when needed) the cognitive abilities of a human operator.

Keywords: Cooperative Manipulators, Compliant Assembly
Abstract: Almost all existing mobile manipulators employ a combination of serial manipulators with various mobile platforms. In order to handle heavy payload, they have to utilize a large-size manipulator and a large high-payload mobile base as well. This results in unwieldy and unoptimized robotic system, which tends to be unsuitable for many automotive assembly applications. To solve this, we develop an innovative mobile parallel manipulator to form a “Smart Companion Robot” which can cooperate with human workers in automotive assembly tasks. The robot employs an omnidirectional mobile base and a parallel manipulator to handle payload. To the best of our knowledge, this is the first robot with such a structure for heavy payload manipulation and transport in six degrees of freedom. Experimental results suggest that human workers can be assisted to effectively, flexibly and conveniently handing heavy parts by taking advantages of the Smart Companion Robot, which has greatly potential benefits in increasing the automotive assembly production efficiency and quality as well as improving ergonomics.

Keywords: Sensor Fusion, Object Detection, Segmentation and Categorization, Perception for Grasping and Manipulation
Abstract: This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object's geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object. Tactile information is acquired to augment the captured depth information. The network can then reason about the object's geometry by utilizing both the collected tactile and depth information. We demonstrate that even small amounts of additional tactile information can be incredibly helpful in reasoning about object geometry. This is particularly true when information from depth alone fails to produce an accurate geometric prediction. Our method is benchmarked against and outperforms other visual-tactile approaches to general geometric reasoning. We also provide experimental results comparing grasping success with our method.

Keywords: Automation Technologies for Smart Cities, Environment Monitoring and Management, Mechanism Design
Abstract: The pavement cleaning is essential to maintain urban hygiene and keep the long stretch of pavements spick and span. This paper reports on the development of novel reconfigurable pavement cleaning robot named Panthera. Reconfiguration in Panthera is gained by the expansion and contraction of the body frame using a single lead screw shaft and linkages mechanism. It gives the capability to reshape itself based on factors like pavement width and pedestrian density. The independent steering action is derived using two in-wheels motors for each steering axis. This imparts the flexibility in motion and make system omnidirectional and allows the convenient movement of the robot in any direction along the pavement. It is powered using onboard batteries that generate lesser noise compared to the existing solution powered with gasoline. The modeling and steering kinematics is presented along with experimental results of the path followed and discussion supporting the robot's capability.

Keywords: Cellular and Modular Robots, Biologically-Inspired Robots, Mechanism Design
Abstract: Automatic repair of mechanical structures would enable a robot to recover or improve functions after physical damage. Little work exists on real-world execution of automatic repair in robotic systems. State-of-the-art takes a modular approach where the robotic system is modular and a replacement module is available. However, the modular approach suffers from low granularity in repair even with tens of motors. In addition, there is a lack of quantitative evaluation of the effect of automatic repair on robot functionality. Here we propose a cooperative method for automatic repair in a robotic system. Our method is regeneration-based rather than module-based and does not assume availability of a replacement part. It integrates a fabrication process on the fly for robot structure regeneration. With a system that consists of a regenerating robot, a legged robot and a pre-engineered ribbon, we demonstrate end-to-end execution of automated repair of the legged robot's leg by the regenerating robot in 335 seconds. Experiments on repeatability show a 100% success rate for sub-processes such as positioning, leg fabrication, and legged robot disengagement and a 90% success rate for leg detachment. We quantify the effect of leg regeneration on mobility recovery and found a 90% recovery of forward speed, a 19.7% increase of peak power and a 9.3% reduction of cost of transport with a regenerated leg.

Keywords: Kinematics, Calibration and Identification
Abstract: While Product of Exponentials (POE) formula has been gaining increasing popularity in modeling the kinematics of a serial-link robot, the Denavit-Hartenberg (D-H) notation is still the most widely used due to its intuitive and concise geometric interpretation of the robot. This paper has developed an analytical solution to automatically convert a POE model into a D-H model for a robot with revolute, prismatic, and helical joints, which are the complete set of three basic one degree of freedom lower pair joints for constructing a serial-link robot. The conversion algorithm developed can be used in applications such as calibration where it is necessary to convert the D-H model to the POE model for identification and then back to the D-H model for compensation. The equivalence of the two models proved in this paper also benefits the analysis of the identifiability of the kinematic parameters. It is found that the maximum number of identifiable parameters in a general POE model is 5h+4r +2t +n+6 where h, r, t, and n stand for the number of helical, revolute, prismatic, and general joints, respectively. It is also suggested that the identifiability of the base frame and the tool frame in the D-H model is restricted rather than the arbitrary six parameters as assumed previously.

Keywords: Mechanism Design, Calibration and Identification
Abstract: A physically-motivated friction model with a parametric description of the nonlinear dependency of the temperature and velocity as well as the dependency on external load is presented. The fully parametric approach extends a static friction model in the gross sliding regime. We show how it can be seamlessly integrated in standard dynamic friction models such as Lund Grenoble (LuGre) and Generalized-Maxwell-Slip (GMS). Parameters of a Harmonic Drive CSD 25 gear are experimentally identified and the final model is evaluated on a dedicated test-bed. We show the integration and effectiveness in dynamic simulation, friction compensation, and external torque estimation.

Keywords: Tendon/Wire Mechanism, Mechanism Design
Abstract: This paper proposes a new wire-driven mechanism in order to relay many ropes very simply and compactly. Ropes pass through in a joint while bundled. Synthetic fiber rope can slide and twist, exploiting its low friction coefficient. In order to use this mechanism, it is necessary to investigate the influence of sliding on tension transmission efficiency and rope strength. The results of this study reveal that it is feasible for a robot arm using this mechanism to have more than 15 joints. Sliding has little influence on rope strength. The feasibility of this system was studied through hardware experiments and its mechanical performance was evaluated by constructing a horizontally extendable manipulator with three degrees of freedom.

Keywords: Medical Robots and Systems, Surgical Robotics: Laparoscopy, Mechanism Design
Abstract: Abstract— Single-incision laparoscopic surgery (SILS) has emerged as a procedure to further improve cosmetic profits and reduce the postoperative pain of multiport laparoscopic surgery. However, SILS is a difficult operation due to the limited workspace or accessibility. To improve surgical convenience, flexible surgical robots should be developed and applied to SILS for a large workspace. Flexible robots would penetrate a single incision point around the navel area and can provide a large workspace within the abdominal cavity. However, it is difficult to support the force required for surgical intervention during this process. In this study, a novel mechanism to lock the shape of a flexible joint to support the external forces during SILS is proposed. The developed shape-locking mechanism involves placing a latch between the joints; the shape locks by engagement of the latches via an electromagnetic force. The mechanism is implemented through mechanical coupling, so it can withstand large loads. Furthermore, the driving force of the mechanism is small because it only needs to engage the latch structure. This paper discusses the development of the mechanism, magnetic force simulation, and a payload experiment. Index Terms— Surgical Robotics: Laparoscopy, Flexible Robots, and Mechanism Design

Keywords: Soft Material Robotics, Biologically-Inspired Robots
Abstract: An echinoderm can actively modulate the structural stiffness of its body wall by as much as 10 times, using the material and structural features that make up its body, including calcite ossicles, connective tissue and inter-ossicular muscle. This capacity for variable stiffness makes it possible to adapt to the kinematics and dynamics required to perform a given task and the surrounding environment. This characteristic can improve the ability of soft material robots, which currently have limited application because of their low load-bearing capability. This paper presents a stiffness modulation method inspired by the connected ossicle structures of echinoderms. We introduce the mechanism, structure, and stiffness variation of the proposed design with respect to different ossicle shape, interval, and elastomer. Then we built a finger-shaped stiffening structure using the proposed design, measured its stiffness according to vacuum level, and showed its load-bearing capacity under control. The proposed design was then applied to a robotic gripper, a typical device that interacts with unpredictable environments and needs variable stiffening ability.

Keywords: Soft Material Robotics, Flexible Robots, Optimization and Optimal Control
Abstract: Soft robot is an emergent research field which has variant promising applications. However, the design of soft robots nowadays still follows the trial-and-error process, which is not at all efficient. This paper proposes to design soft robots by pre-checking controllability during the numerical design phase. Finite-element method is used to model the dynamics of silicon soft robots, based on which the differential geometric method is applied to analyze the controllability of the points of interest. Such a verification is also investigated via model order reduction technique and Galerkin projection. The proposed methodology is finally validated by numerically designing a controllable parallel soft robot.

Keywords: Soft Material Robotics, Flexible Robots, Grippers and Other End-Effectors
Abstract: Soft robotics has yielded numerous examples of soft grippers that utilize compliance to achieve impressive grasping performances with great simplicity, adaptability, and robustness.Designing soft grippers with substantial grasping strength while remaining compliant and gentle is one of the most important challenges in this field. In this paper, we present a light-weight, vacuum-driven soft robotic gripper made of an origami "magic-ball" and a flexible thin membrane. We also describe the design and fabrication method to rapidly manufacture the gripper with different combinations of low-cost materials for diverse applications. Grasping experiments demonstrate that our gripper can lift a large variety of objects, including delicate foods, heavy bottles, and other miscellaneous items. The grasp force on 3D printed objects is also characterized through mechanical load tests. The results reveal that our soft gripper can produce significant grasp force on various shapes using negative pneumatic pressure (vacuum). This new gripper holds the potential for many practical applications that require safe, strong, and simple grasping.

Keywords: Soft Material Robotics, Hydraulic/Pneumatic Actuators, Mechanism Design
Abstract: The Elastic Inflatable Actuators (EIAs) has several advantages such as the inherent compliance due to the body comprised of a soft materials such as silicone. Among them, the soft fiber reinforced actuator is based on the principle that the expansion of enclosure and constraint of fiber pattern lead to a desired operation. While lots of researches on the actuator has been attributed to linear and bending motions, however, there are only few researches on rotary, or torsional, motions. In this paper, we propose a new actuator that causes azimuthal deformation due to restriction of anisotropically distributed fiber element along the radial direction and expansion of the hyper elastic material. Structure design of the actuator and a fabrication process of the actuator are presented. Subsequently, FEM simulation and experiment are executed to measure rotation angles of the actuators corresponding to the applied pressure.

Keywords: Grippers and Other End-Effectors, Grasping, Soft Material Robotics
Abstract: Pneumatic actuators have gained huge popularity in the field of soft robotics. A class of this kind of devices exploits inflatable thin membranes which generate a desired displacement upon inflation, but often without providing sufficient force/torque to perform their task. In this paper, we propose a novel actuator combining a membrane and a rigid foldable structure. Experimental tests show that such INFlatable ORigami Actuator (INFORA) is characterized by relatively high stiffness compared to other actuators of the same class. We provide a mathematical model to be used for design purposes and we describe the fabrication process. In addition, we show how the INFORA can be used to build a tendril-like structure capable of performing grasping tasks.

Keywords: Soft Material Robotics, Biologically-Inspired Robots, Biomimetics
Abstract: Soft structures in nature endow organisms across scales with the ability to drastically deform their bodies and exhibit complex behaviours while overcoming challenges in their environments. Inspired by microstructures found in the cell membranes of the Euglena family of microorganisms, which exhibit giant changes in shape during their characteristic euglenoid movement, this paper presents the design, fabrication and characterisation of bio-inspired deforming surfaces. The result is a surface of interconnected strips, that deforms in 2D and 3D due to simple shear between adjacent members. We fabricate flexible polymeric strips and demonstrate three different shapes arising out of the same actuation by imposing various constraints. We characterise the strips in terms of the force required to separate them and show that the bio-inspired cross section of these strips enables them to hold up to 8N of force with a meagre 0.5mm of material thickness, while still being flexible to deform. Further, the design of a soft robot module, with an actively deformable surface has been presented which replicates the mechanism of shape change seen in the euglena. This work shows the potential for this new form of shape morphing surface in realising bio-mimetic soft robots exhibiting large changes in shape.

Keywords: Legged Robots, Biologically-Inspired Robots, Dynamics
Abstract: This research explored the generation of period-two dynamic running motion in a robot, based on the passive dynamic period-two motion of the reduced order, rolling spring-loaded inverted pendulum (R-SLIP) model. Each cycle of period-two motion consists of two stance phases separated by two flight phases. The distribution of the period-two fixed points of the model was analyzed using a return map. Models with the same or different landing angles per motion cycle were studied, and two sets of period-two motion trajectories were implemented in a robot for experimental evaluation. Without sensory feedback or control, this evaluation relied on the open loop trajectory of the model. Based on the experiments, the robot was capable of performing dynamic period-two motion.

Keywords: Legged Robots, Deep Learning in Robotics and Automation, Biologically-Inspired Robots
Abstract: Humans and animals are believed to use a very minimal set of trajectories to perform a wide variety of tasks including walking. Our main objective in this paper is two fold 1) Obtain an effective tool to realize these basic motion patterns for quadrupedal walking, called the kinematic motion primitives (kMPs), via trajectories learned from deep reinforcement learning (D-RL) and 2) Realize a set of behaviors, namely trot, walk, gallop and bound from these kinematic motion primitives in our custom four legged robot, called the “Stoch”. D-RL is a data driven approach, which has been shown to be very effective for realizing all kinds of robust locomotion behaviors, both in simulation and in experiment. On the other hand, kMPs are known to capture the underlying structure of walking and yield a set of derived behaviors. We first generate walking gaits from D-RL, which uses policy gradient based approaches. We then analyze the resulting walking by using principal component analysis. We observe that the kMPs extracted from PCA followed a similar pattern irrespective of the type of gaits generated. Leveraging on this underlying structure, we then realize walking in Stoch by a straightforward reconstruction of joint trajectories from kMPs. This type of methodology improves the transferability of these gaits to real hardware, lowers the computational overhead on-board, and also avoids multiple training iterations by generating a set of derived behaviors from a single learned gait.

Keywords: Legged Robots, Deep Learning in Robotics and Automation, Reactive and Sensor-Based Planning
Abstract: In this paper, we propose a method for selecting the optimal footholds for legged systems. The goal of the proposed method is to find the best foothold for the swing leg on a local elevation map. First, we evaluate the geometrical characteristics of each cell on the elevation map, checks kinematic constraints and collisions. Then, we apply the Convolutional Neural Network to learn the relationship between the local elevation map and the quality of potential footholds. During execution time, the controller obtains the qualitative measurement of each potential foothold from the neural model. This method evaluates hundreds of potential footholds and checks multiple constraints in a single step which takes 10 ms on a standard computer without GPU. The experiments were carried out on a quadruped robot walking over rough terrain in both simulation and real robotic platforms.

Keywords: Legged Robots, Optimization and Optimal Control, Motion and Path Planning
Abstract: This paper presents a novel methodology for implementing optimized jumping behavior on quadruped robots. Our method includes efficient trajectory optimization, precise high-frequency tracking controller and robust landing controller for stabilizing the robot body position and orientation after impact. Experimental validation was successfully conducted on the MIT Cheetah 3, enabling the robot to repeatably jump onto and jump down from a desk with the height of 30'' (0.76 m). The result demonstrates the advantages of the approach as well as the capability of the robot hardware itself.

Keywords: Legged Robots, Biologically-Inspired Robots
Abstract: In order for legged robotic platforms to become adept enough to operate in unstructured, outdoor environments it is critical that they have the ability to adapt to a variety of terrains. One class of terrains to consider are regions of shallow, dense fluids, such as a beach-head, stream banks, snow or mud. This work examines the behavior of a simulated SLIP runner operating in such a viscous medium. Simulation results show that intelligently retracting the leg during flight can have a profound effect on the maximum achievable velocity of the runner, the stability of the resulting gait, and the cost of transport of the runner. Results also show that trudging gaits, in which the leg is positioned behind the center of mass, can be favorable in certain situations in terms of energy consumption and forward velocity.

Keywords: Legged Robots, Field Robots, Compliant Joint/Mechanism
Abstract: Despite the development of a large number of mobile manipulation robots, very few platforms can demonstrate the required strength and mechanical sturdiness to accommodate the needs of real-world applications with high payload and moderate/harsh physical interaction demands, e.g. in disaster-response scenarios or heavy logistics/collaborative tasks. In this work we introduce the design of a wheeled-legged mobile manipulation platform capable of executing demanding manipulation tasks, and demonstrating significant physical resilience while possessing a body size (height/width) and weight compatible to that of human. The achieved performance is the result of combining a number of design and implementation principles related to the actuation system, the integration of body structure and actuation, and the wheeled-legged mobility concept. These design principles are discussed, and the solutions adopted for various robot components are detailed. Finally, the robot performance is demonstrated in a set of experiments validating its power and strength capability when manipulating heavy payload and executing tasks involving high impact physical interactions.

Keywords: Nonholonomic Motion Planning, Robot Safety, Reactive and Sensor-Based Planning
Abstract: We present a new framework for motion planning that wraps around existing kinodynamic planners and guarantees recursive feasibility when operating in a priori unknown, static environments. Our approach makes strong guarantees about overall safety and collision avoidance by utilizing a robust controller derived from reachability analysis. We ensure that motion plans never exit the safe backward reachable set of the initial state, while safely exploring the space. This preserves the safety of the initial state, and guarantees that that we will eventually find the goal if it is possible to do so while exploring safely. We implement our framework in the Robot Operating System (ROS) software environment and demonstrate it in a real-time simulation.

Keywords: Kinematics, Robot Safety, Task Planning
Abstract: Set-Based Multi-Task Priority is a recent framework to handle inverse kinematics for redundant structures. Both equality tasks, i.e., control objectives to be driven to a desired value, and set-bases tasks, i.e., control objectives to be satisfied with a set/range of values can be addressed in a rigorous manner within a priority framework. In addition, optimization tasks, driven by the gradient of a proper function, may be considered as well, usually as lower priority tasks. In this paper the proper design of the tasks, their priority and the use of a Set-Based Multi-Task Priority framework is proposed in order to handle several constraints simultaneously in real-time. It is shown that safety related tasks such as, e.g., joint limits or kinematic singularity, may be properly handled by consider them both at an higher priority as set-based task and at a lower within a proper optimization functional. Experimental results on a 7DOF Jaco^2 arm with and without the proposed approach show the effectiveness of the proposed method.

Keywords: Robot Safety, Force Control, Compliance and Impedance Control
Abstract: The current paper proposes a control methodology for ensuring safety during human–robot interaction based on a compliant sensor covering the robot links as a lightweight shell. The method can be used with existing robots without the need for mechanical redesign. To assess the behaviour of the proposed control law, the controller is analysed using a linear robot model. Stability analysis is performed and requirements on the controller parameters are derived. The effect of the controller parameters on the perceived impedance and the maximum safe operating velocity of the robot are determined via the linear model. The adverse impact of dry friction is analysed in simulation and methods are developed to mitigate the effects. The controller is implemented on a 1 DoF robotic joint and the results are compared to those of a traditional admittance control law, demonstrating comparable transient response while maintaining a simple control structure and decreased risk of instability.

Keywords: Robot Safety, Formal Methods in Robotics and Automation, Collision Avoidance
Abstract: Robots are entering an age of ubiquity, and to operate effectively, these systems must typically satisfy a series of constraints (e.g., collision avoidance, obeying speed limits, maintaining connectivity). In addition, modern applications hinge on the completion of particular tasks, like driving to a certain location or monitoring a crop patch. The dichotomy between satisfying constraints and completing objectives creates a need for constraint-satisfaction frameworks that are composable with a pre-existing primary objective. Barrier functions have recently emerged as a practical and composable method for constraint satisfaction, and prior results demonstrate a system of Boolean logic for nonsmooth barrier functions as well as a composable controller-synthesis framework; however, this prior work does not consider dynamically changing constraints (e.g., a robot sensing and avoiding an obstacle). Consequently, the main theoretical contribution of this paper extends nonsmooth barrier functions to time-varying barrier functions with jumps. In a practical instantiation of the theoretical main results, this work revisits a classic problem by formulating a collision-avoidance framework and composing it with a nominal controller. Experimental results show the efficacy of this framework on a LIDAR-equipped differential-drive robot in a real-time obstacle-avoidance scenario.

Keywords: Robot Safety, Computer Vision for Other Robotic Applications, Collision Avoidance
Abstract: We present a novel view synthesis method for mobile robots in dynamic environments and its application to the estimation of future traversability. Our method predicts future images for given virtual robot velocity commands using only the RGB images from previous and current time steps. The future images result from applying two types of image changes to the previous and current images: 1) changes caused by different camera pose, and 2) changes due to the motion of the dynamic obstacles. We learn to predict these two types of changes disjointly using two novel network architectures, SNet and DNet. We combine SNet and DNet to synthesize future images that we pass to our previously presented method GONet to estimate the traversable areas around the robot. Our quantitative and qualitative evaluation indicate that our approach for view synthesis predicts accurate future images in both static and dynamic environments. We also show that these virtual images can be used to estimate future traversability correctly. We apply our view synthesis-based traversability estimation method to two applications for assisted teleoperation.

Keywords: Soft Material Robotics, Robot Safety, Flexible Robots
Abstract: This paper proposes a sensor using conductive fabric that can detect proximity and contact by measuring the capacitance between the sensor and the surrounding environment. Due to the flexibility of the sensor used, it can be easily integrated with industrial robot arms to monitor proximity and contact between the robot and the surrounding environment including humans for safety reasons. However, the surrounding environment is constantly changing and significantly affects the capacitance measurements. To apply such proximity sensors in this scenario, the environmental variations have to be considered and the influences on the capacitance measurements have to be eliminated to ensure stable and robust proximity measurements. Therefore, in this paper, we propose an approach to adaptively update the reference capacitance to eliminate the influence of the environment. To experimentally validate the proposed sensor and approach, we developed a two-link robot arm and embedded the proposed sensing technology with each link. Experimental results demonstrate that proximity and contact can be successfully detected by the proposed sensor, independently of whether the robot arm is at rest or moving in a potentially dynamic environment.

Keywords: Wheeled Robots, Optimization and Optimal Control, Additive Manufacturing
Abstract: Applications of mobile ground robots demand high speed and agility while navigating in complex indoor environments. These present an ongoing challenge in mobile robotics. A system with these specifications would be of great use for a wide range of indoor inspection tasks. This paper introduces Ascento, a compact wheeled bipedal robot that is able to move quickly on flat terrain, and to overcome obstacles by jumping. The mechanical design and overall architecture of the system is presented, as well as the development of various controllers for different scenarios. A series of experiments with the final prototype system validate these behaviors in realistic scenarios.

Keywords: Wheeled Robots, Dynamics, Robust/Adaptive Control of Robotic Systems
Abstract: This paper presents a path following controller that is suitable for asymmetrical planar robots with significant mass and limited motor torques. The controller is resistant against environmental forces, and inaccurate estimates of robot’s inertia, by estimating their effects with Unscented Kalman Filter. The controller outputs wheel torque commands which take in account the motor torque limits and given relative priority of internal control elements. The method presented is thoroughly explained and the simulation results demonstrate the performance of the controller

Keywords: Wheeled Robots, Sensor Fusion, Dynamics
Abstract: Path tracking controllers for an autonomous vehicle are often designed by using either a dynamic model or a kinematic one and some models are related to wheel-ground contact, that makes the efficiency of the controller highly dependent on the ground parameters estimation, especially for off-road mobile robots intended to navigate in open environments. This paper proposes a new nonlinear observer designed to estimate the front and rear contact cornering stiffnesses in real time that are related both on tire and soil proprieties. The latter is estimated using steering angles as well as yaw rate and lateral velocity, which are provided by a preliminary Kalman-Bucy observer. The performance of the proposed nonlinear observer combined with the LQR controller is evaluated by both advanced simulations and experiments in real conditions at different speeds.

Keywords: Wheeled Robots, Mobile Manipulation, Humanoid Robots
Abstract: In this paper, we present a whole-body control framework for textit{Wheeled Inverted Pendulum (WIP) Humanoids}. WIP Humanoids are redundant manipulators dynamically balancing themselves on wheels. Characterized by several degrees of freedom, they have the ability to perform several tasks simultaneously, such as balancing, maintaining a body pose, controlling the gaze, lifting a load or maintaining end-effector configuration in operation space. The problem of whole-body control is to enable simultaneous performance of these tasks with optimal participation of all degrees of freedom at specified priorities for each objective. The control also has to obey constraint of angle and torque limits on each joint.

The proposed approach is hierarchical with a low level controller for body joints manipulation and a high-level controller that defines center of mass (CoM) targets for the low-level controller to control zero dynamics of the system driving the wheels. The low-level controller plans for shorter horizons while considering more complete dynamics of the system, while the high-level controller plans for longer horizon based on an approximate model of the robot for computational efficiency.

Keywords: Dynamics, Wheeled Robots, Contact Modeling
Abstract: Impulsive actuation enables robots to perform agile maneuvers and surpass difficult terrain, yet its capacity to induce continuous and stable locomotion have not been explored. We claim that strictly convex foot shapes can improve the impulse effectiveness (impulse used per travelled distance) and locomotion speed by facilitating periodicity and stability. To test this premise, we introduce a theoretical 2-D model based on rigid-body mechanics to prove stability. We then implement a more elaborate model in simulation to study transient behavior and impulse effectiveness. Finally, we test our findings on a robot platform to prove their physical validity. Our results prove that continuous and stable locomotion can be achieved in the strictly convex case of a disk with an off-centered mass. In keeping with our theory, stable limit cycles of the off-centered disk outperform the theoretical performance of a cube in simulation and experiment, using up to 10 times less impulse per distance to travel at the same locomotion speed.

Keywords: Hydraulic/Pneumatic Actuators, Compliant Joint/Mechanism, Soft Material Robotics
Abstract: Current robotics industry trends show an increased interest in the interaction between humans and robots in a variety of fields, ranging from collaborative robots in manufacturing to assisted medical devices in the medical field. One limiting factor in present applications is the ability to actively morph these robotic structures and control their stiffness using the same type of actuation system. This paper focuses on developing an actively controlled, variable stiffness structure that uses a pneumatic system for both morphing and locking the structure shape. The structure design integrates Pneumatic Artificial Muscles (PAMs) that are pressurized to control shape morphing. The pressurization of the PAM provides a radial force that allows bi-directional morphing based on the pressurization scheme. Layer Jamming, which utilizes varied friction between thin sheets based on pressure, is used to control the variable stiffness of the structure. In this paper, a control model is developed to predict the morphed curvature of the structure based on the input actuator pressure. This experimental control model is also validated using a theoretical pseudo-rigid-body model. The repeatability and accuracy of morphing is also discussed. Through experimental testing, a measure of the stiffness variation range of the structure is also developed. This novel research would positively impact the robotics field by creating lightweight morphing structures that are flexible and easily deformed.

Keywords: Hydraulic/Pneumatic Actuators, Grippers and Other End-Effectors, Mechanism Design
Abstract: Dexterous, serial-chain motor-driven robotic arms have high moving mass, since most of the actuators must be located in the arm itself. This necessitates high gear ratios, sacrificing passive compliance, backdrivability, and the capacity for delicate motion. We introduce the concept of a remote direct-drive (RDD) manipulator, in which every motor is located in the base, connected to remote joints via a low-friction hydrostatic transmission. We have designed a new hydrostatic linear actuator with a fully-floating piston; the piston floats within the cylinder using a pair of soft fiber-elastomer rolling-diaphragm seals. This eliminates static friction from seal rubbing and piston/rod misalignment. Actuators were developed with a 20mm bore, weighing 55 grams each with a 400:1 bidirectional strength-to-weight ratio (+/- 230N), which drive a 2-DOF manipulator (wrist pitch/finger pinch; 120-degree range-of-motion; 6.6 Nm max grip strength). The gripper is hydrostatically coupled to remotely-located direct-drive/backdrivable brushless electric motors. System hysteresis and friction are 1 percent of full-range force. This low-mass low-friction configuration is of great interest for powered prosthetic hand design, and passively-safe high dynamic range robot arms.

Keywords: Soft Material Robotics, Hydraulic/Pneumatic Actuators, Additive Manufacturing
Abstract: This work demonstrates 3D printed soft actuators with complex shapes and remote actuation using an external magnetic field. Instead of embedding magnetic particles in a polymeric matrix, we fabricated a novel ferrofluid-based actuator, in which the fluid can be moved to different locations in the actuator to affect actuator response. We studied the effect of both the ferrofluid and the 3D printed material on the motion of simple actuators using 3D printed tubes. In addition, we 3D printed more complex actuators mimicking a human hand and a worm to demonstrate more complex motion.

Keywords: Soft Material Robotics, Hydraulic/Pneumatic Actuators, Model Learning for Control
Abstract: Recent research on mobile robots has focused on increasing their adaptability to unpredictable and unstructured environments using soft materials and structures, and pneu- matic actuators have been widely used in soft mobile robots. In spite of their advantages, existing pneumatic actuators have limitations with regard to control and speed. We propose soft membrane vibration actuators for a simple tripod robot. The proposed actuators are composed of a rigid housing and a soft membrane that creates vibration based on the input air pressure. The mobile robot proposed here has three vibration actuators arranged in an equilateral triangle configuration. Based on the pressure on each actuator, the robot can easily re- alize both translational and rotational motions. We constructed an analytical model of the proposed actuator and characterized the behavior of the robot experimentally. We also implemented model-free control based on a Gaussian process. The robot demonstrated the ability to follow the trajectories of various polygons. It could also carry a payload five times heavier than its own weight.

Keywords: Soft Material Robotics, Haptics and Haptic Interfaces, Telerobotics and Teleoperation
Abstract: We present a new rolling diaphragm actuator for transmitting forces in a teleoperated system. The initial application is for MR-guided biopsy procedures, providing accurate transmission of motions and forces between the fingertips of a physician and a biopsy needle being inserted into tissue. Desirable actuator qualities include low hysteresis, high axial stiffness, and long travel. The actuator uses an anisotropic laser-patterned fabric embedded in a soft silicone sleeve for a combination of low stretch in the axial direction and sufficient stretch in the radial direction so that a taper is not required; hence the actuator can have almost any length. We present results for a prototype input/output system with 6 cm stroke, 1 cm diameter and a minimum force of 0.3N to initiate motion. We compare its performance to a system using commercial rolling diaphragm actuators and show that the new system provides an improved combination of long stroke, high stiffness, and accurate transmission of fingertip forces.

Keywords: Hydraulic/Pneumatic Actuators, Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles
Abstract: Magnetic resonance imaging (MRI)-guided intervention has drawn increasing attention over the last decade. It is accredited to the capability of monitoring any physiological change of soft tissue with the high-contrast MR images. This also gives rise to the demand for precise tele-manipulation of interventional instruments. However, there is still lack of choices of MR safe actuators that provide high-fidelity robot manipulation. In this paper, we present a three-cylinder hydraulic motor using rolling-diaphragm-sealed cylinders, which can provide continuous bidirectional rotation with unlimited range. Both kinematics and dynamics models of the presented motor were studied, which facilitate its overall design optimization and position/torque control. Motor performances, such as step response, frequency response, and accuracy, were experimentally evaluated. We also integrate the motor into our catheter robot prototype designed for intra-operative MRI-guided cardiac electrophysiology (EP), which is capable of providing full-degree-of-freedom and precise manipulation of a standard EP catheter.

Keywords: Autonomous Agents, Deep Learning in Robotics and Automation
Abstract: Achieving effective task performance on real mobile robots is a great challenge when hand-coding algorithms, both due to the amount of effort involved and manually tuned parameters required for each skill. Learning algorithms instead have the potential to lighten up this challenge by using one single set of training parameters for learning different skills, but the question of the feasibility of such learning in real robots remains a research pursuit. We focus on a kind of mobile robot system - the robot soccer ''small-size'' domain, in which tactical and high-level team strategies build upon individual robot ball-based skills. In this paper, we present our work using Deep Reinforcement Learning to learn three real robot primitive skills in continuous action space: go-to-ball, turn-and-shoot and shoot-goalie, for which there is a clear success metric to reach a destination or score a goal. We introduce the state and action representation, as well as the reward and network architecture. We describe our training and testing using a simulator of high physical and hardware fidelity. Then we test the policies trained from simulation on real robots. Our results show that the learned skills achieve an overall better success rate at the expense of taking 0.29 seconds slower on average of all three skills. In the end, we show that our policies trained in simulation have good performance on real robots by directly transferring the policy.

Keywords: Autonomous Agents, Learning and Adaptive Systems, Motion and Path Planning
Abstract: For safety reasons, robotic lawn mowers and similar devices are required to stay within a predefined working area. Keeping the robot within its workspace is typically achieved by special safeguards such as a wire installed in the ground. In the case of robotic lawn mowers, this causes a certain customer reluctance. It is more desirable to fulfill those safety-critical tasks by safe navigation and path planning. In this paper, we tackle the problem of planning a coverage path composed of parallel lanes that maximizes robot safety under the constraints of cheap, low range sensors and thus substantial uncertainty in the robot's belief and ability to execute actions. Our approach uses a map of the environment to estimate localizability at all locations, and it uses these estimates to search for an uncertainty-aware coverage path while avoiding collisions. We implemented our approach using C++ and ROS and thoroughly tested it on real garden data. The experiment shows that our approach leads to safer meander patterns for the lawn mower and takes expected localizability information into account.

Keywords: Deep Learning in Robotics and Automation, Agent-Based Systems, Learning from Demonstration
Abstract: Deep Reinforcement Learning (DRL) has become a powerful methodology to solve complex decision-making problems. However, DRL has several limitations when used in real-world problems (e.g., robotics applications). For instance, long training times are required and cannot be accelerated in contrast to simulated environments, and reward functions may be hard to specify/model and/or to compute. Moreover, the transfer of policies learned in a simulator to the real-world has limitations (reality gap). On the other hand, machine learning methods that rely on the transfer of human knowledge to an agent have shown to be time efficient for obtaining well performing policies and do not require a reward function. In this context, we analyze the use of human corrective feedback during task execution to learn policies with high-dimensional state spaces, by using the D-COACH framework, and we propose new variants of this framework. D-COACH is a Deep Learning based extension of COACH (COrrective Advice Communicated by Humans), where humans are able to shape policies through corrective advice. The enhanced version of D-COACH, which is proposed in this paper, largely reduces the time and effort of a human for training a policy. Experimental results validate the efficiency of the D-COACH framework in three different problems (simulated and with real robots), and show that its enhanced version reduces the human training effort considerably.

Keywords: Autonomous Agents, Learning from Demonstration, Cognitive Human-Robot Interaction
Abstract: Driving is a complex task that requires the perception of the surrounding environment, decision making and control of the vehicle. Human drivers predict how surrounding objects move and decide an appropriate driving behavior. As with human drivers, autonomous driving vehicles should consider the condition of the surrounding environment and behave naturally so as not to disturb the traffic flow. We propose a reward function for learning how natural the driving is based on the hypothesis that the movement of surrounding vehicles becomes unpredictable when the ego vehicle takes an unnatural driving behavior. The reward function is based on the prediction error of a deep predictive network that models the transition of the surrounding environment. Occupancy grid image is used to perceive the surrounding environment and the predictions up to two seconds are used to calculate the reward function. We evaluated the reward function using both simulated and the real world data. We trained the prediction network using real driving data and trained a reinforcement learning agent based on the reward function. Then we compared the speed planned by the agent and a human driver, which showed a correlation of 0.52. We also confirmed the benefit of taking prediction into account by observing the behavior of the agent in a specific traffic scenario.

Keywords: Autonomous Agents, Simulation and Animation, Visual Learning
Abstract: Autonomous driving has gained significant advancements in recent years. However, obtaining a robust control policy for driving remains challenging as it requires training data from a variety of scenarios, including rare situations (e.g., accidents), an effective policy architecture, and an efficient learning mechanism. We propose ADAPS for producing robust control policies for autonomous vehicles. ADAPS consists of two simulation platforms in generating and analyzing accidents to automatically produce labeled training data, and a memory-enabled hierarchical control policy. Additionally, ADAPS offers a more efficient online learning mechanism that reduces the number of iterations required in learning compared to existing methods such as DAGGER [1]. We present both theoretical and experimental results. The latter are produced in simulated environments, where qualitative and quantitative results are generated to demonstrate the benefits of ADAPS.

Keywords: Autonomous Agents, Agent-Based Systems, Motion and Path Planning
Abstract: This paper addresses the problem of using autonomous robots to record events that obey narrative structure. The work is motivated by a vision of robot teams that can, for example, produce individualized highlight videos for each runner in a large-scale road race such as a marathon. We introduce a method for specifying the desired structure as a function that describes how well the captured events can be used to produce an output that meets the specification. This function is specified in a compact, legible form similar to a weighted finite automaton. Then we describe a planner that uses simple predictions of future events to coordinate the robots' efforts to capture the most important events, as determined by the specification. We describe an implementation of this approach and demonstrate its effectiveness in a simulated race scenario both in simulation and in a hardware testbed.

Keywords: Contact Modeling, Dynamics, Simulation and Animation
Abstract: We present a differential-algebraic formulation with switching constraints to model the nonsmooth dynamics of robotic systems subject to changing constraints and multiple impacts. The formulation combines a single structurally simple governing equation, a set of switching kinematic constraints, and the plastic impact law, to represent the dynamics of robots that interact with their environment. The main contribution of this formulation is a novel algorithmic impact resolution method which provides an explicit solution to the classical plastic impact law in the case of multiple simultaneous impacts. This method serves as an alternative to prior linear-complementarity-based formulations which offer an implicit impact resolution through iterative calculation. We demonstrate the utility of the proposed method by simulating the locomotion of a planar anthropometric biped.

Keywords: Contact Modeling, Dynamics, Simulation and Animation
Abstract: We present a principled method for motion prediction via dynamic simulation for rigid bodies in intermittent contact with each other where the contact is assumed to be a planar non-convex contact patch. The planar non-convex contact patch can either be a topologically connected set or disconnected set. Such algorithms are useful in planning and control for robotic manipulation. Most work in rigid body dynamic simulation assumes that the contact between objects is a point contact, which may not be valid in many applications. In this paper, by using the convex hull of the contact patch, we build on our recent work on simulating rigid bodies with convex contact patches, for simulating the motion of objects with planar non-convex contact patches. We formulate a discrete-time mixed complementarity problem where we solve the contact detection and integration of the equations of motion simultaneously. Thus, our method is a geometrically-implicit method and we prove that in our formulation, there is no artificial penetration between the contacting rigid bodies. We solve for the equivalent contact point (ECP) and contact impulse of each contact patch simultaneously along with the state, i.e., configuration and velocity of the objects. We provide empirical evidence to show that our method can seamlessly capture transition between different contact modes like patch contact to multiple or single point contact during simulation.

Keywords: Contact Modeling, Simulation and Animation, Legged Robots
Abstract: In this paper, we propose a semi-empirical approach for simulating robot locomotion on granular media. We first develop a contact model based on the stick-slip behavior between rigid objects and granular grains, which is then learned through running extensive experiments. The contact model represents all possible contact wrenches that the granular substrate can provide as a convex volume, which our method formulates as constraints in an optimization-based contact force solver. During simulation, granular substrates are treated as rigid objects that allow penetration and the contact solver solves for wrenches that maximize frictional dissipation. We show that our method is able to simulate plausible interaction response with several granular media at interactive rates.

Keywords: Contact Modeling, Simulation and Animation, Dynamics
Abstract: Over the past several decades, as affordable computational power has increased, simulation has become increasingly important in robot analysis, planning, and control. Smooth robot dynamics can be simulated efficiently and accurately and therefore readily used in model-based control schemes. However, some of the most difficult and important problems in robotics, such as running, grasping, and parts assembly, involve intermittent frictional contacts, which introduce extreme nonlinearities into the dynamics.

Numerous approaches have been developed to simulate contact dynamics. Even though real bodies are not rigid, idealized rigid contact models have been used widely and productively for decades. However, the resulting non-smooth dynamics can be computationally difficult to solve or use in model-based planning and control, motivating researchers to propose various relaxations of the idealized contact models. The varied origins and formulations of these approaches can obscure their similarities and differences.

In this paper, we identify and explain differences between four contact models. We present the models in the context of one solver that is applicable to all of them, namely Projected Gauss-Seidel, in order to highlight their common structure and to avoid confounding their comparison with differences in the solution methods. Simulation results from sliding, wedging, grasping, and stacking experiments illustrate consequences of the differences.

Keywords: Contact Modeling, Grippers and Other End-Effectors, Grasping
Abstract: In this paper, we present an analytical method for study the role of the micro-patterned morphology of soft finger with wet adhesion in grasping of deformable thin shell object. This work was originated from a project on a robotic platform for autonomous attachment/removing soft contact lens to/from human's eyes in wet environment. In this scenario, a contact lens (hemispherical thin shell) was gripped by a soft-fingered hand in three conditions: inside/outside liquid environment, and in contact with a curvature substrate (as an eye). The finger's tip is attached by pads in two cases: flat surface, and micro-patterned surface inspired by adhesion mechanism of a tree-frog's toes. The pattern includes 3600 square cells, each cell has size of 85(mum)x85(mum). The proposed analytical model is utilized to evaluate the grasp forces in both cases of the pads, and verified by an actual application of grasping a contact lens. The experimental results showed a good agreement with the analysis, indicating that the micro-structured pads decreased the applied preload and deformation of the shell 1.5-2 times lower than that of the flat surface. This work could be extended to modeling grasping interface with deformable curvature objects in wet and high moisturized environment.

Keywords: Optimization and Optimal Control
Abstract: We consider the feedback design for stabilizing a rigid body system by making and breaking multiple contacts with the environment without prespecifying the timing or the number of occurrence of the contacts. We model such a system as a discrete-time hybrid polynomial system, where the state-input space is partitioned into several polytopic regions with each region associated with a different polynomial dynamics equation. Based on the notion of occupation measures, we present a novel controller synthesis approach that solves finite-dimensional semidefinite programs as approximations to an infinite-dimensional linear program to stabilize the system. The optimization formulation is simple and convex, and for any fixed degree of approximations the computational complexity is polynomial in the state and control input dimensions. We illustrate our approach on some robotics examples.

Keywords: Formal Methods in Robotics and Automation, Motion and Path Planning, Optimization and Optimal Control
Abstract: This paper presents an abstraction-refinement based framework for optimal controller synthesis of discrete-time systems with respect to w-regular objectives. It first abstracts the discrete-time concrete system into a finite weighted transition system using a finite partition of the state-space. Then, a two-player mean payoff parity game is solved on the product of the abstract system and the Buchi automaton corresponding to the w-regular objective, to obtain an optimal abstract controller that satisfies the w-regular objective. The abstract controller is guaranteed to be implementable in the concrete discrete-time system, with a sub-optimal cost. The abstraction is refined with finer partitions to reduce the sub-optimality. In contrast to existing formal controller synthesis algorithms based on abstractions, this technique provides an upper bound on the trajectory cost when implementing the suboptimal controller. A robot surveillance scenario is presented to illustrate the feasibility of the approach.

Keywords: Hybrid Logical/Dynamical Planning and Verification, Formal Methods in Robotics and Automation, Motion and Path Planning
Abstract: Piecewise affine (PWA) systems are widely used to model highly nonlinear behaviors such as contact dynamics in robot locomotion and manipulation. Existing control techniques for PWA systems have computational drawbacks, both in offline design and online implementation. In this paper, we introduce a method to obtain feedback control policies and a corresponding set of admissible initial conditions for discrete-time PWA systems such that all the closed-loop trajectories reach a goal polytope, while a cost function is optimized. The idea is conceptually similar to LQR-trees in Tedrake et al., 2010, which consists of 3 steps: (1) open-loop trajectory optimization, (2) feedback control for computation of "funnels" of states around trajectories, and (3) repeating (1) and (2) in a way that the funnels are grown backward from the goal in a tree fashion and fill the state-space as much as possible. We show PWA dynamics can be exploited to combine step (1) and (2) into a single step that is tackled using mixed-integer convex programming, which makes the method suitable for dealing with hard constraints. Illustrative examples on contact-based dynamics are presented.

Keywords: Hybrid Logical/Dynamical Planning and Verification, Optimization and Optimal Control, AI-Based Methods
Abstract: Hamilton-Jacobi (HJ) reachability analysis has been developed over the past decades into a widely-applicable tool for determining goal satisfaction and safety verification in nonlinear systems. While HJ reachability can be formulated very generally, computational complexity can be a serious impediment for many systems of practical interest. Much prior work has been devoted to computing approximate solutions to large reachability problems, yet many of these methods may only apply to very restrictive problem classes, do not generate controllers, and/or can be extremely conservative. In this paper, we present a new method for approximating the optimal controller of the HJ reachability problem for control-affine systems. While also a specific problem class, many dynamical systems of interest are, or can be well approximated, by control-affine models. We explicitly avoid storing a representation of the reachability value function, and instead emph{learn} a controller as a sequence of simple binary classifiers. We compare our approach to existing grid-based methodologies in HJ reachability and demonstrate its utility on several examples, including a physical quadrotor navigation task.

Keywords: Planning, Scheduling and Coordination, Hybrid Logical/Dynamical Planning and Verification, Task Planning
Abstract: In this paper we focus on finding practical resolution methods for Markov Decision Processes (MDPs) in robotics. Some of the main difficulties of applying MDPs to real-world robotics problems are: (1) having to deal with huge state spaces; and (2) designing a method that is robust enough to dead ends. These complications restrict or make more difficult the application of methods such as Value Iteration, Policy Iteration or Labeled Real Time Dynamic Programming (LRTDP). We see in determinization and heuristic search a way to successfully work around these problems. In addition, we believe that many practical use cases offer the opportunity to identify hierarchies of subtasks and solve smaller, simplified problems. We propose a decision-making unit that operates in a probabilistic planning setting through Stochastic Shortest Path Problems (SSPPs), which generalize the most common types of MDPs. Our decision-making unit combines: (1) automatic hierarchical organization of subtasks; and (2) on-line resolution via determinization. We argue that several applications of planning benefit from these two strategies. We exemplify our approach with a robotized disassembly application. The disassembly problem is modeled in Probabilistic Planning Definition Language (PPDDL), and serves to define our experiments. Our results show many advantages of our method over LRTDP, like a better capability to handle problems with large state spaces.

Keywords: Localization, Deep Learning in Robotics and Automation, Aerial Systems: Applications
Abstract: We present a novel method of integrating image-based measurements into a drone navigation system for the automated inspection of wind turbines. We take a model-based tracking approach, where a 3D skeleton representation of the turbine is matched to the image data. Matching is based on comparing the projection of the representation to that inferred from images using a convolutional neural network. This enables us to find image correspondences using a generic turbine model that can be applied to a wide range of turbine shapes and sizes. To estimate 3D pose of the drone, we fuse the network output with GPS and IMU measurements using a pose graph optimiser. Results illustrate that the use of the image measurements significantly improves the accuracy of the localisation over that obtained using GPS and IMU alone.

Keywords: Robotics in Hazardous Fields, Environment Monitoring and Management, Aerial Systems: Applications
Abstract: This paper presents experiments to assess the plume mapping performance of unmanned autonomous vehicles. The paper compares several mapping algorithms including Gaussian Process regression, Neural networks and polynomial and piecewise linear interpolation. The methods are compared in Monte Carlo simulations using a well known plume model and in indoor experiments using a ground robot. Unlike previous work on mapping using unmanned vehicles, the indoor experiments were performed in a controlled, repeatable, manner where a steady state ground truth could be obtained in order to properly assess the various regression methods using data from a real dispersive source and sensor. The effect of sampling time during data collection was assessed with regards to the mapping accuracy, and the data collected during the experiments has been made available. Overall, the Gaussian Process method was found to perform the best among the regression algorithms, showing more robustness to the noisy measurements obtained from short sampling periods, enabling an accurate map to be produced in significantly less time. Finally, plume mapping results are presented in uncontrolled outdoor conditions, using an unmanned aerial vehicle, to demonstrate the system in a realistic uncontrolled environment.

Keywords: Neural and Fuzzy Control, Learning and Adaptive Systems, Aerial Systems: Applications
Abstract: This work presents an online learning-based control method for improved trajectory tracking of unmanned aerial vehicles using both deep learning and expert knowledge. The proposed method does not require the exact model of the system to be controlled, and it is robust against variations in system dynamics as well as operational uncertainties. The learning is divided into two phases: offline (pre-)training and online (post-)training. In the former, a conventional controller performs a set of trajectories and, based on the input-output dataset, the deep neural network (DNN)-based controller is trained. In the latter, the trained DNN, which mimics the conventional controller, controls the system. Unlike the existing papers in the literature, the network is still being trained for different sets of trajectories which are not used in the training phase of DNN. Thanks to the rule-base, which contains the expert knowledge, the proposed framework learns the system dynamics and operational uncertainties in real-time. The experimental results show that the proposed online learning-based approach gives better trajectory tracking performance when compared to the only offline trained network.

Keywords: Swarms, Cooperating Robots, Aerial Systems: Applications
Abstract: Small unmanned aerial vehicles (UAVs) have generally little capacity to carry payloads. Through collaboration, the UAVs can increase their joint payload capacity and carry more significant loads. For maximum flexibility to dynamic and unstructured environments and task demands, we propose a fully decentralized control infrastructure based on a swarm-specific scripting language, Buzz. In this paper, we describe the control infrastructure and use it to compare two algorithms for collaborative transport: field potentials and spring-damper. We test the performance of our approach with a fleet of micro-UAVs, demonstrating the potential of decentralized control for collaborative transport.

Keywords: Biologically-Inspired Robots
Abstract: This paper describes recent developments of a robotic hummingbird project, aimed at achieving precision stationary hovering. To this end, the early version of our flapping mechanism is modified which, besides being more efficient, reduces significantly the asymmetry of the wing trajectory of the previous version. A cascade control strategy is used to compensate for the residual parasitic torques and the misalignment of the lift vector and the autopilot.

Keywords: Sensor Fusion, Aerial Systems: Perception and Autonomy, Localization
Abstract: This paper presents the robust Adaptive unscented Kalman filter (RAUKF) for attitude estimation. Since the proposed algorithm represents attitude as a unit quaternion, all basic tools used, including the standard UKF, are adapted to the unit quaternion algebra. Additionally, the algorithm adopts an outlier detector algorithm to identify abrupt changes in the UKF innovation and an adaptive strategy based on covariance matching to tune the measurement covariance matrix online. Adaptation and outlier detection make the proposed algorithm robust to fast and slow perturbations such as magnetic field interference and linear accelerations. Experimental results with a manipulator robot suggest that our method overcomes other algorithms found in the literature.

Keywords: Learning from Demonstration, Deep Learning in Robotics and Automation, Computer Vision for Automation
Abstract: Due to burdensome data requirements, learning from demonstration often falls short of its promise to allow users to quickly and naturally program robots. Demonstrations are inherently ambiguous and incomplete, making correct generalization to unseen situations difficult without a large number of demonstrations in varying conditions. By contrast, humans are often able to learn complex tasks from a single demonstration (typically observations without action labels) by leveraging context learned over a lifetime. Inspired by this capability, our goal is to enable robots to perform one-shot learning of multi-step tasks from observation by leveraging auxiliary video data as context. Our primary contribution is a novel system that achieves this goal by: (1) using a single user-segmented demonstration to define the primitive actions that comprise a task, (2) localizing additional examples of these actions in unsegmented auxiliary videos via a metalearning-based approach, (3) using these additional examples to learn a reward function for each action, and (4) performing reinforcement learning on top of the inferred reward functions to learn action policies that can be combined to accomplish the task. We empirically demonstrate that a robot can learn multi-step tasks more effectively when provided auxiliary video, and that performance greatly improves when localizing individual actions, compared to learning from unsegmented videos.

Keywords: Learning from Demonstration, Optimization and Optimal Control, Legged Robots
Abstract: Guided policy search is a popular approach for training controllers for high-dimensional systems, but it has a number of pitfalls. Non-convex trajectory optimization has local minima, and non-uniqueness in the optimal policy itself can mean that independently-optimized samples do not describe a coherent policy from which to train. We introduce LVIS, which circumvents the issue of local minima through global mixed-integer optimization and the issue of non-uniqueness through learning the optimal value function rather than the optimal policy. To avoid the expense of solving the mixed-integer programs to full global optimality, we instead solve them only partially, extracting intervals containing the true cost-to-go from early termination of the branch-and-bound algorithm. These interval samples are used to weakly supervise the training of a neural net which approximates the true cost-to-go. Online, we use that learned cost-to-go as the terminal cost of a one-step model-predictive controller, which we solve via a small mixed-integer optimization. We demonstrate LVIS on piecewise affine models of a cart-pole system with walls and a planar humanoid robot and show that it can be applied to a fundamentally hard problem in feedback control--control through contact.

Keywords: Learning from Demonstration, Domestic Robots
Abstract: Learning from demonstration is a powerful tool for teaching manipulation actions to a robot. It is, however, an unsolved problem how to consider knowledge about the world and action-induced reactions such as forces imposed onto the gripper or measured liquid levels during pouring without explicit and case dependent programming. In this paper, we present a novel approach to include such knowledge directly in form of measured features. To this end, we use action demonstrations together with external features to learn a motion encoded by a dynamic system in a Gaussian Mixture Model (GMM) representation. Accordingly, during action imitation, the system is able to couple the geometric trajectory of the motion to measured features in the scene. We demonstrate the feasibility of our approach with a broad range of external features in real-world robot experiments including a drinking, a handover and a pouring task.

Keywords: Learning from Demonstration
Abstract: We propose a learning framework, named Multi-Coordinate Cost Balancing (MCCB), to address the problem of acquiring point-to-point movement skills from demonstrations. MCCB encodes demonstrations simultaneously in multiple differential coordinates that specify local geometric properties. MCCB generates reproductions by solving a convex optimization problem with a multi-coordinate cost function and linear constraints on the reproductions, such as initial, target, and via points. Further, since the relative importance of each coordinate system in the cost function might be unknown for a given skill, MCCB learns optimal weighting factors that balance the cost function. We demonstrate the effectiveness of MCCB via detailed experiments conducted on one handwriting dataset and three complex skill datasets.

Keywords: Learning from Demonstration, Perception for Grasping and Manipulation, Learning and Adaptive Systems
Abstract: We address the challenge of how a robot can adapt its actions to successfully manipulate objects it has not previously encountered. We introduce Real-time Multisensory Affordance-based Control (RMAC), which uses multisensory inputs and Hidden Markov Models to enable a robot to determine how much to adapt an existing affordance model. We show that using the combination of haptic, audio, and visual information with RMAC allows the robot to learn afforance models and adaptively manipulate two very different objects (drawer, lamp), in multiple novel configurations. Offline evaluations and real-time online evaluations show that RMAC allows the robot to accurately open different drawer configurations and turn-on novel lamps with an average accuracy of 75%.

Keywords: Learning from Demonstration
Abstract: Robotic Learning from Demonstration (LfD) allows anyone, not just experts, to program a robot for an arbitrary task. Many LfD methods focus on low level primitive actions such as manipulator trajectories. Complex multi-step task with many primitive actions must be learned from demonstration if LfD is to encompass the full range of task a user may desire. Existing methods represent the high level task in various forms including, finite state machines, decision trees, formal logic, among others. Behavior trees are proposed as an alternative representation of high level task. Behavior trees are an execution model for the control of a robot designed for real time execution, modularity, and, consequently, transparency. Real time execution allows the robot to reactively perform the task. Modularity allows the reuse of learned primitive actions and high level task in new situations, speeding up the process of learning in new scenarios. Transparency allows users to understand and interactively modify the learned model. Behavior trees are used to represent high level tasks by building on the relationship it has with decision trees. We demonstrate a human teaching our Fetch robot a household cleaning task.

Keywords: Learning from Demonstration, Human-Centered Automation, Cognitive Human-Robot Interaction
Abstract: High-level human activities often have rich temporal structures that determine the order in which atomic actions are executed. We propose the Temporal Context Graph (TCG), a temporal reasoning model that integrates probabilistic inference with Allen's interval algebra, to capture these temporal structures. TCGs are capable of modeling tasks with cyclical atomic actions and consisting of sequential and parallel temporal relations. We present Learning from Demonstration as the application domain where the use of TCGs can improve policy selection and address the problem of perceptual aliasing. Experiments validating the model are presented for learning two tasks from demonstration that involve structured human-robot interactions.

Keywords: Learning from Demonstration, Planning, Scheduling and Coordination, Dual Arm Manipulation
Abstract: Multi-modal robot programming with natural language and demonstration is a promising technique for efficient teaching of manipulation tasks in industrial environments. In particular, with modern dual-arm robots designed to quickly take over tasks at typical industrial workbenches, the direct teaching of task sequences hardly utilizes the robots' capabilities. We therefore propose a two-staged approach that combines natural language instructions and demonstration with simultaneous task allocation and motion scheduling based on constraint programming. Instead of providing a task description and demonstrations that are replayed to a large extent, the user describes tasks to be scheduled with all relevant constraints and demonstrates relevant locations relative to workpieces and other objects. With explicitly stated constraints on the partial ordering of tasks, the solver allocates the tasks to the robot arms and schedules them in time while avoiding self-collisions and reducing the makespan in our experiment by 33%. The linguistic concepts of naming and grouping enable systematic reuse of sub-task ensembles. The proposed approach is evaluated with four variants of a gluing use-case from furniture assembly in user studies with ten participants. In these user studies, we observed a speed-up for the task definition of more than 6 times compared to a textual specification of the planning problems using the Python-based planner API.

Keywords: Learning from Demonstration, AI-Based Methods, Learning and Adaptive Systems
Abstract: Learning from demonstrations is an easy and intuitive way to show examples of successful behavior to a robot. However, the fact that humans optimize or take advantage of their body and not of the robot, usually called the embodiment problem in robotics, often prevents industrial robots from executing the task in a straightforward way. The shown movements often do not or cannot utilize the degrees of freedom of the robot efficiently, and moreover can suffer from excessive execution errors. In this letter, we explore a variety of solutions that address these shortcomings. In particular, we learn sparse movement primitive parameters from several demonstrations of a successful table tennis serve. The number of parameters learned using our procedure is independent of the degrees of freedom of the robot. Moreover, they can be ranked according to their importance in the regression task. Learning few parameters, which are ranked, is a desirable feature to combat the curse of dimensionality in reinforcement learning. Real robot experiments on the Barrett WAM for a table tennis serve using the learned movement primitives show that the representation can capture successfully the style of the movement with few parameters.

Keywords: Learning from Demonstration, Compliant Assembly
Abstract: We present a Learning from Demonstration method for teaching robots to perform search strategies imitated from humans in scenarios where alignment tasks fail due to position uncertainty. The method utilizes human demonstrations to learn both a state invariant dynamics model and an exploration distribution that captures the search area covered by the demonstrator. We present two alternative algorithms for computing a search trajectory from the exploration distribution, one based on sampling and another based on deterministic ergodic control. We augment the search trajectory with forces learnt through the dynamics model to enable searching both in force and position domains. An impedance controller with superposed forces is used for reproducing the learnt strategy. We experimentally evaluate the method on a KUKA LWR4+ performing a 2D peg-in-hole and a 3D electricity socket task. Results show that the proposed method can, with only few human demonstrations, learn to complete the search task.

Keywords: Learning from Demonstration, Sensor-based Control, Human-Centered Automation
Abstract: This paper combines an imitation learning approach with a model-based and constraint-based task specification and control methodology. Imitation learning provides an intuitive way for the end user to specify the context of a new robot application without the need for traditional programming skills. On the other hand, constraint-based robot programming allows us to define complex tasks involving different kinds of sensor input. Combination of both enables adaptation of complex tasks to new environments and new objects with a small number of demonstrations. The proposed method uses a statistical uni-modal model to describe the demonstrations in terms of a number of weighted basis functions. This is then combined with model-based descriptions of other aspects of the task at hand. This approach was tested in a use case inspired by an industrial application, in which the required transfer motions were learned from a small number of demonstrations, and gradually improved by adding new demonstrations. Information on a collision-free path was introduced through a small number of demonstrations. The method showed a high level of composability with force and vision controlled tasks. The use case showed that the deployment of a complex constraint-based task with sensor interactions can be expedited using imitation learning.

Keywords: Learning from Demonstration, Motion and Path Planning, Manipulation Planning
Abstract: Parametric dynamic movement primitives (PDMPs) are powerful motion representation algorithms which encode the multiple demonstrations and generalize them. As an online trajectory from PDMPs emulates the provided demonstrations, managing the qualified demonstrations for a given scenario is an important issue. This paper presents the process to manage the motion demonstrations in PDMPs when some demonstrations are poor. Our proposed process distinguishes safe motion primitives from unsafe ones. In order to establish a criterion for determining whether a motion is safe or not, we calculate the safe region of the PDMP parameters using an optimization technique. In the optimization formulation, we calculate the unsafe style parameters which produce the closest motion to the unsafe point. By eliminating unsafe demonstrations with the parameters in the unsafe criterion, and replacing them with safe ones, we incorporate the safety in the PDMPs framework. Simulation and experimental results validate that the proposed process can expand the motion primitives in the PDMPs framework to the new environmental settings by efficiently utilizing the previous demonstrations.

Keywords: Learning and Adaptive Systems, Force and Tactile Sensing, Motion and Path Planning
Abstract: In the present work, we propose an active tactile exploration framework to obtain a surface model of an unknown object utilizing multiple contacts simultaneously. To incorporate these multiple contacts, the exploration strategy is based on the differential entropy of the underlying Gaussian process implicit surface model, which formalizes the exploration with multiple contacts within an information theoretic context and additionally allows for nonmyopic multi-step planning. In contrast to many previous approaches, the robot continuously slides along the surface with its end-effectors to gather the tactile stimuli, instead of touching it at discrete locations. This is realized by closely integrating the surface model into the compliant controller framework. Furthermore, we extend our recently proposed sliding based tactile exploration approach to handle non-convex objects. In the experiments, it is shown that multiple contacts simultaneously leads to a more efficient exploration of complex, non-convex objects, not only in terms of time, but also with respect to the total moved distance of all end-effectors. Finally, we demonstrate our methodology with a real PR2 robot that explores an object with both of its arms.

Keywords: Learning and Adaptive Systems, Deep Learning in Robotics and Automation, Motion Control
Abstract: Guided Policy Search enables robots to learn control policies for complex manipulation tasks efficiently. Therein, the control policies are represented as high-dimensional neural networks which derive robot actions based on states. However, due to the small number of real-world trajectory samples in Guided Policy Search, the resulting neural networks are only robust in the neighbourhood of the trajectory distribution explored by real-world interactions. In this paper, we present a new policy representation called Generative Motor Reflexes, which is able to generate robust actions over a broader state space compared to previous methods. In contrast to prior state-action policies, Generative Motor Reflexes map states to parameters for a state-dependent motor reflex, which is then used to derive actions. Robustness is achieved by generating similar motor reflexes for many states. We evaluate the presented method in simulated and real-world manipulation tasks, including contact-rich peg-in-hole tasks. Using these evaluation tasks, we show that policies represented as Generative Motor Reflexes lead to robust manipulation skills also outside the explored trajectory distribution with less training needs compared to previous methods.

Keywords: Learning and Adaptive Systems, Industrial Robots, Intelligent and Flexible Manufacturing
Abstract: Collaborative robots are expected to work in cooperation with humans to improve productivity and maintain the quality of products. In the previous study, we have proposed an incremental learning system for adaptively scheduling a motion of the collaborative robot based on a worker’s behavior. Although this system could model the worker’s motion pattern precisely and robustly without collecting the worker’s data in advance, it required two different models for modeling the worker’s spatial and temporal features respectively and was not well considered for generalization. In this paper, we extend the previous incremental learning system by integrating the spatial and temporal models using a mixture model. In addition, we install a new incremental learning algorithm which improves a generalization capability of the mixture model and avoids overfitting in the situation where the prior information is limited. Implementing the proposed algorithm, we evaluate the effectiveness of the proposed system by experiments for several workers and for several assembly processes.

Keywords: Learning and Adaptive Systems, Task Planning, Deep Learning in Robotics and Automation
Abstract: Multi-object manipulation problems in continuous state and action spaces can be solved by planners that search over sampled values for the continuous parameters of operators. The efficiency of these planners depends critically on the effectiveness of the samplers used, but effective sampling in turn depends on details of the robot, environment, and task. Our strategy is to learn functions called "specializers" that generate values for continuous operator parameters, given a state description and values for the discrete parameters. Rather than trying to learn a single specializer for each operator from large amounts of data on a single task, we take a modular meta-learning approach. We train on multiple tasks and learn a variety of specializers that, on a new task, can be quickly adapted using relatively little data -- thus, our system "learns quickly to plan quickly" using these specializers. We validate our approach experimentally in simulated 3D pick-and-place tasks with continuous state and action spaces. Visit http://tinyurl.com/chitnis-icra-19 for a supplementary video.

Keywords: Learning and Adaptive Systems, Object Detection, Segmentation and Categorization, Deep Learning in Robotics and Automation
Abstract: When identifying an object and its properties, humans use features from multiple sensory modalities produced when manipulating the object. Motivated by this cognitive process, we propose a deep learning methodology for object category recognition which uses visual, auditory, and haptic sensory data coupled with exploratory behaviors (e.g., grasping, lifting, pushing, etc.). In our method, as the robot performs an action on an object, it uses a Tensor-Train Gated Recurrent Unit network to process its visual data, and Convolutional Neural Networks to process haptic and auditory data. We propose a novel strategy to train a single neural network that inputs video, audio and haptic data, and demonstrate that its performance is better than separate neural networks for each sensory modality. The proposed method was evaluated on a dataset in which the robot explored 100 different objects, each belonging to one of 20 categories. While the visual information was the dominant modality for most categories, adding the additional haptic and auditory networks further improves the robot's category recognition accuracy. For some of the behaviors, our approach outperforms the previous published baseline for the dataset which used handcrafted features for each modality. We also show that a robot does not need the sensory data from the entire interaction, but instead can make a good prediction early on during behavior execution.

Keywords: Learning and Adaptive Systems, Robust/Adaptive Control of Robotic Systems
Abstract: Humans can skilfully use tools and interact with the environment by adapting theirmovement trajectory, contact force, and impedance. Motivated by the human versatility, we develop here a robot controller that concurrently adapts feedforward force, impedance, and reference trajectory when interacting with an unknown environment. In particular, the robot’s reference trajectory is adapted to limit the interaction force and maintain it at a desired level, while feedforward force and impedance adaptation compensates for the interaction with the environment. An analysis of the interaction dynamics using Lyapunov theory yields the conditions for convergence of the closed-loop interaction mediated by this controller. Simulations exhibit adaptive properties similar to human motor adaptation. The implementation of this controller for typical interaction tasks including drilling, cutting, and haptic exploration shows that this controller can outperform conventional controllers in contact tooling.

Keywords: Deep Learning in Robotics and Automation, Optimization and Optimal Control
Abstract: This paper proposes a machine learning method to predict the solutions of related nonlinear optimal control problems given some parametric input, such as the initial state. The map between problem parameters to optimal solutions is called the problem-optimum map, and is often discontinuous due to nonconvexity, discrete homotopy classes, and control switching. This causes difficulties for traditional function approximators such as neural networks, which assume continuity of the underlying function. This paper proposes a mixture of experts (MoE) model composed of a classifier and several regressors, where each regressor is tuned to a particular continuous region. A novel training approach is proposed that trains classifier and regressors independently. MoE greatly outperforms standard neural networks, and achieves highly reliable trajectory prediction (over 99% accuracy) in several underactuated control problems.

Keywords: AI-Based Methods, Computer Vision for Automation
Abstract: Typically, to enlarge the operating domain of an object detector, more labeled training data is required. We describe a method called wormhole learning, which allows to extend the operating domain without additional data, but only with temporary access to an auxiliary sensor with certain invariance properties. We describe the instantiation of this principle with a regular visible-light RGB camera as the main sensor, and an infrared sensor as the temporary sensor. We start with a pre-trained RGB detector; then we train the infrared detector based on the RGB-inferred labels; then we re-train the RGB detector based on the infrared-inferred labels. After these two transfer-learning steps, the RGB detector has enlarged its operating domain by inheriting part of the invariance to illumination of the infrared sensor; in particular, the RGB detector is now able to see much better at night. We analyze the wormhole learning phenomen by bounding the possible gain in accuracy by mutual information properties of the two sensors and the labels.

Keywords: Physical Human-Robot Interaction, Human-Centered Robotics, Cooperating Robots
Abstract: Seamless communication of desired motions and goals is essential for enabling effective physical human-robot collaboration. In such cases, muscle activity measured via surface electromyography (EMG) can provide insight into a person's intentions while minimally distracting from the task. The presented system uses two muscle signals to create a control framework for team lifting tasks in which a human and robot lift an object together. A continuous setpoint algorithm uses biceps activity to estimate changes in the user's hand height, and also allows the user to explicitly adjust the robot by stiffening or relaxing their arm. In addition to this pipeline, a neural network trained only on previous users classifies biceps and triceps activity to detect up or down gestures on a rolling basis; this enables finer control over the robot and expands the feasible workspace. The resulting system is evaluated by 10 untrained subjects performing a variety of team lifting and assembly tasks with rigid and flexible objects.

Keywords: Medical Robots and Systems, Flexible Robots, Model Learning for Control
Abstract: Non-linearities in cable transmissions are important limitations for the accurate control of flexible instruments used in medical endoscopic systems. Hysteresis effects greatly impact the accuracy of conventional kinematic models. This is especially critical for implementing automatic motions in flexible medical robotic systems. In this paper, we propose a method for improving open-loop accuracy of flexible instruments by implementing a Position Inverse Kinematic Model which is able to take into account hysteresis effects. In order to avoid complex physical modeling, the method relies on the off-line learning of the behavior of the instruments. Basic knowledge of the kinematic is also incorporated in the learning process in order to make it fast. The validity of the approach is demonstrated by the execution of 2D and 3D trajectories with the instruments of the STRAS medical robot. The accuracy is shown to be significantly improved with respect to other learning-based methods.

Keywords: Prosthetics and Exoskeletons, Learning and Adaptive Systems, Collision Avoidance
Abstract: Avoiding obstacles poses a significant challenge for amputees using mechanically-passive transfemoral prosthetic limbs due to their lack of direct knee control. In contrast, powered prostheses can potentially improve obstacle avoidance via their ability to add energy to the system. In past work, researchers have proposed stumble recovery systems for powered prosthetic limbs that provide assistance in the event of a trip. However, these systems only aid recovery after an obstacle has disrupted the user's gait and do not proactively help the amputee avoid obstacles. To address this problem, we designed an adaptive system that learns online to use kinematic data from the prosthetic limb to detect the user's obstacle avoidance intent in early swing. When the system detects an obstacle, it alters the planned swing trajectory to help avoid trips. Additionally, the system uses a regression model to predict the required knee flexion angle for the trip response. We validated the system by comparing obstacle avoidance success rates with and without the obstacle avoidance system. For a non-amputee subject wearing the prosthesis through an adapter, the trip avoidance system improved the obstacle negotiation success rate from 37% to 89%, while an amputee subject improved his success rate from 35% to 71% when compared to utilizing minimum jerk trajectories for the knee and ankle joints.

Keywords: Mobile Manipulation, Manipulation Planning
Abstract: This paper presents a novel robotic manipulation technique for transporting objects on the ground in a passive dynamic, nonprehensile manner. The object is manipulated to rock from side to side repeatedly; in the meantime, the force of gravity enables the object to roll along a zigzag path that is eventually heading forward. We call it rock-and-walk object locomotion. First, we examine the kinematics and dynamics of the rocking motion to understand how the states of the object evolve. We then discuss how to control the robot to connect individual rocking motions into a stable gait of the object. Our rock-and-walk object transportation technique is implemented using a conventional manipulator arm and a simple end-effector, interacting with the object in a nonprehensile manner in favor of the passive dynamics of the object. A set of experiments demonstrates successful object locomotion.

Keywords: Marine Robotics, Multi-Robot Systems, Field Robots
Abstract: Autonomous robotic boats are devised to transport people and goods similar to self-driving cars. One of the attractive features specially applied in water environment is to dynamically link and join multiple boats into one unit in order to form floating infrastructure such as bridges, markets or concert stages, as well as autonomously self-detach to perform individual tasks.

In this paper we present a novel latching system that enables robotic boats to create dynamic united floating infrastructure while overcoming water disturbances. The proposed latching mechanism is based on the spherical joint (ball and socket) that allows rotation and free movements in two planes at the same time. In this configuration, the latching system is capable to securely and efficiently assemble/disassemble floating structures. The vision-based robot controller guides the self- driving robotic boats to latch with high accuracy in the millimeter range. Moreover, in case the robotic boat fails to latch due to harsh weather, the autonomous latching system is capable to recompute and reposition to latch successfully. We present experimental results from latching and docking in indoor environments. Also, we present results in outdoor environments from latching a couple of robotic boats in open water with calm and turbulent currents.

Keywords: Marine Robotics, Semantic Scene Understanding, Cognitive Human-Robot Interaction
Abstract: This paper proposes a bandwidth tunable technique for real-time probabilistic scene modeling and mapping to enable co-robotic exploration in communication constrained environments such as the deep sea. The parameters of the system enable the user to characterize the scene complexity represented by the map, which in turn determines the bandwidth requirements. The approach is demonstrated using an underwater robot that learns an unsupervised scene model of the environment and then uses this scene model to communicate the spatial distribution of various high-level semantic scene constructs to a human operator. Preliminary experiments in an artificially constructed tank environment as well as simulated missions over a 10m x 10m coral reef using real data show the tunability of the maps to different bandwidth constraints and science interests. To our knowledge this is the first paper to quantify how the free parameters of the unsupervised scene model impact both the scientific utility of and bandwidth required to communicate the resulting scene model.

Keywords: Marine Robotics, Deep Learning in Robotics and Automation, Computer Vision for Other Robotic Applications
Abstract: Stereo cameras are widely used for sensing and navigation of underwater robotic systems. They provide high resolution color views of a scene; the constrained camera geometry enables metrically accurate depth estimation; they are also relatively cost-effective. Traditional stereo vision algorithms rely on feature detection and matching to enable triangulation of points for estimating disparity. However, for underwater applications, the effects of underwater light propagation lead to image degradation, reducing image quality and contrast. This makes it challenging to detect and match features, especially from varying viewpoints. Recently, deep learning has shown success in end-to-end learning of dense disparity maps from stereo images. Many state-of-the-art methods are supervised and require ground truth depth, which is challenging to gather in subsea environments. Simultaneously, deep learning has also been applied to the problem of underwater image restoration. Again, it is difficult to gather real ground truth data for this problem. In this work, we present an unsupervised network that takes input raw underwater stereo imagery and outputs dense depth maps and color corrected imagery of underwater scenes. We leverage a model of the process of underwater image formation, image processing techniques, as well as geometric constraints inherent to the stereo vision problem to develop a modular network that outperforms existing methods.

Keywords: Mechanism Design, Marine Robotics, Field Robots
Abstract: We present a low-cost, three-degree-of-freedom (3- DOF) prismatic-spherical-revolute (PSR) parallel mechanism used as a testing platform for an unmanned aerial vehicle (UAV) tethered to an unmanned surface vehicle (USV). The mechanism has three actuated linear rails kinematically linked to a platform which replicates boat motion up to 2.5 m vertical heave (sea state 4, Douglas Sea Scale). A lookup table relating relative slider heights to platform roll and pitch was developed numerically leveraging geometric constraints. A design parameter study optimized the arm length, platform size, and ball joint mounting angle relative to the overall radius to maximize the workspace. For this design, a maximum roll and pitch range from -32◦ to 32◦ and -25◦ to 35◦, respectively, is achievable. A prototype was manufactured to carry the tethered UAV winch payload. Experimental testing confirmed the workspace and demonstrated boat motion replication, validated using an inertial measurement unit (IMU).

Keywords: Marine Robotics, Autonomous Vehicle Navigation, Visual-Based Navigation
Abstract: This letter studies the problem of autonomous navigation for unmanned underwater vehicles, using computer vision for localization. Parallel tracking and mapping is employed to localize the vehicle with respect to a visual map, using a single camera, whereas an extended Kalman filter (EKF) is used to fuse the visual information with data from an inertial measurement unit, in order to recover the scale of the map and improve the pose estimation. A proportional integral derivative controller with compensation of the restoring forces is proposed to accomplish trajectory tracking, where a pressure sensor and a magnetometer provide feedback for depth control and yaw, respectively, while the remaining states are provided by the EKF. Real-time experiments are presented to validate the navigation strategy, using a commercial remotely operated vehicle (ROV), the BlueROV2, which was adapted to perform as an autonomous underwater vehicle with the help of the robot operative system.

Keywords: Model Learning for Control, Marine Robotics, Learning and Adaptive Systems
Abstract: Learning the dynamics of robots from data can help achieve more accurate tracking controllers, or aid their navigation algorithms. However, when the actual dynamics of the robots change due to external conditions, on-line adaptation of their models is required to maintain high fidelity performance. In this work, a framework for on-line learning of robot dynamics is developed to adapt to such changes. The proposed framework employs an incremental support vector regression method to learn the model sequentially from data streams. In combination with the incremental learning, strategies for including and forgetting data are developed to obtain better generalization over the whole state space. The framework is tested in simulation and real experimental scenarios demonstrating its adaptation capabilities to changes in the robot's dynamics.

Keywords: Cognitive Human-Robot Interaction, Robot Audition
Abstract: Previous work on emotion recognition demonstrated a synergistic effect of combining several modalities such as auditory, visual, and transcribed text, to estimate the affective state of a speaker. Among these, the linguistic modality is crucial for the evaluation of an expressed emotion. However, manually transcribed spoken text cannot be given as input to a system practically. We argue that using ground truth transcriptions during training and evaluation phases leads to a significant discrepancy in performance compared to real-world conditions, as the spoken text has to be recognized on the fly and can contain speech recognition mistakes. In this paper, we propose a method of integrating an automatic speech recognition (ASR) output with a character-level recurrent neural network for sentiment recognition. In addition, we conduct several experiments investigating sentiment recognition in human-robot interaction in a noise realistic scenario which is challenging for the ASR systems. We quantify the improvement compared to using only the acoustic modality in sentiment recognition. We demonstrate the effectiveness of this approach on the Multimodal Corpus of Sentiment Intensity (MOSI) by achieving 73,6% accuracy in a binary sentiment classification task, exceeding previously reported results that use only acoustic input. In addition, we set a new state-of-the-art performance on the MOSI dataset (80.4% accuracy, 2% absolute improvement).

Keywords: Computer Vision for Automation, Gesture, Posture and Facial Expressions, Social Human-Robot Interaction
Abstract: Gesture interaction is a natural way of communicating with a robot as an alternative to speech. Gesture recognition methods leverage optical flow in order to under-stand human motion. However, while accurate optical flow estimation (i.e., traditional) methods are costly in terms of runtime, fast estimation (i.e., deep learning) methods’ accuracy can be improved. In this paper, we present a pipeline for gesture-based human-robot interaction that uses a novel optical flow estimation method in order to achieve an improved speed-accuracy trade-off. Our optical flow estimation method introduces four improvements to previous deep learning-based methods: strong feature extractors, attention to contours, mid-way features, and a combination of these three. This results in a better understanding of motion, and a finer representation of silhouettes. In order to evaluate our pipeline, we generated our own dataset, MIBURI, which contains gestures to command a house service robot. In our experiments, we show how our method improves not only optical flow estimation, but also gesture recognition, offering a speed-accuracy trade-off more realistic for practical robot applications.

Keywords: Multi-Robot Systems, Deep Learning in Robotics and Automation, Swarms
Abstract: Effective communication is required for teams of robots to solve sophisticated collaborative tasks. In practice it is typical for both the encoding and semantics of communication to be manually defined by an expert; this is true regardless of whether the behaviors themselves are bespoke, optimization based, or learned. We present an agent architecture and training methodology using neural networks to learn task-oriented communication semantics based on the example of a communication-unaware expert policy. A perimeter defense game illustrates the system's ability to handle dynamically changing numbers of agents and its graceful degradation in performance as communication constraints are tightened or the expert's observability assumptions are broken.

Keywords: AI-Based Methods, Learning from Demonstration, Social Human-Robot Interaction
Abstract: Past work on acquisition of word-object associations in robots has focused on either fast instruction-based methods which accept highly constrained input or gradual cross-situational learning methods, but not a mixture of both. In this paper, we present an integrated robotic system which allows for a combination of these methods to contribute to the task of learning the labels of objects in AI agents. We demonstrate the expanded word learning capabilities in the outcome system and how learning from both human-human and human-robot dialogues can be achieved in one integrated system.

Keywords: Human-Centered Robotics, Physical Human-Robot Interaction, Virtual Reality and Interfaces
Abstract: The ability to reference objects in the environment is a key communication skill that robots need for complex, task oriented human-robot collaborations. In this paper we explore the use of projections, which are a powerful communication channel for robot-to-human information transfer as they allow for situated, instantaneous, and parallelized visual referencing. We focus on the question of what makes a good projection for referencing a target object. To that end, we mathematically formulate legibility of projections intended to reference an object, and propose alternative arrow-object match functions for optimally computing the placement of an arrow to indicate a target object in a cluttered scene. We implement our approach on a PR2 robot with a head-mounted projector. Through an online (48 participants) and an in-person (12 participants) user study we validate the effectiveness of our approach, identify the types of scenes where projections may fail, and characterize the differences between alternative match functions.

Keywords: Human Factors and Human-in-the-Loop, Cognitive Human-Robot Interaction, Planning, Scheduling and Coordination
Abstract: In this work, we synthesize collaboration protocols for human-unmanned aerial vehicle (H-UAV) command and control systems, where the human operator aids in securing the UAV by intermittently performing geolocation tasks to confirm its reported location. We first present a stochastic game-based model for the system that accounts for both the operator and an adversary capable of launching stealthy false-data injection attacks, causing the UAV to deviate from its path. We also describe a synthesis challenge due to the UAV's hidden-information constraint. Next, we perform human experiments using a developed RESCHU-SA testbed to recognize the geolocation strategies that operators adopt. Furthermore, we deploy machine learning techniques on the collected experimental data to predict the correctness of a geolocation task at a given location based on its geographical features. By representing the model as a delayed-action game and formalizing the system objectives, we utilize off-the-shelf model checkers to synthesize protocols for the human-UAV coalition that satisfy these objectives. Finally, we demonstrate the usefulness of the H-UAV protocol synthesis through a case study where the protocols are experimentally analyzed and further evaluated by human operators.

Keywords: Cooperating Robots, Multi-Robot Systems
Abstract: In this paper we consider inter-robot communication in the context of joint activities. In particular, we focus on convoying and passive communication for radio-denied environments by using whole-body gestures to provide cues regarding future actions. We develop a communication protocol whereby information described by codewords is transmitted by a series of actions executed by a swimming robot. These action sequences are chosen to optimize robustness and transmission duration given the observability, natural activity of the robot and the frequency of different messages.Our approach uses a convolutional network to make core observations of the pose of the robot being tracked, which is sending messages. The observer robot then uses an adaptation of classical decoding methods to infer a message that is being transmitted. The system is trained and validated using simulated data, tested in the pool and is targeted for deployment in the open ocean. Our decoder achieves .94 precision and .66 recall on real footage of robot gesture execution recorded in a swimming pool.

Keywords: Cooperating Robots, Multi-Robot Systems, Networked Robots
Abstract: Spatial sensing is a fundamental requirement for applications in robotics and augmented reality. In urban spaces such as malls, airports, apartment buildings and others, it is quite challenging for a single robot to map the whole environment. So, we employ a swarm of robots to perform mapping. One challenge with this approach is the mechanism to merge maps. In this work, we use wireless access points, which are ubiquitous in most urban spaces, to provide us with coarse orientation between sub-maps, and use a custom ICP algorithm to refine this orientation to merge them. We demonstrate our approach with maps from a building on campus and evaluate it using two approaches. Our results show that, in the building we studied, we can achieve an Absolute Trajectory Error of 0.2m and Root Mean Square Error of 1.3m in known landmark positions.

Keywords: Distributed Robot Systems, Sensor Fusion, Learning and Adaptive Systems
Abstract: Bayesian nonparametric models, such as the Dirichlet Process Gaussian Process (DPGP), have been shown very effective at learning models of dynamic targets exclusively from data. Previous work on batch DPGP learning and inference, however, ceases to be efficient in multi-sensor applications that require decentralized measurements to be obtained sequentially over time. Batch processing, in this case, leads to redundant computations that may hinder online applicability. This paper develops a recursive approach for DPGP learning and inference in which a novel Dirichlet Process prior based on Wasserstein metric is used for measuring the similarity between multiple Gaussian Processes (GPs). Combined with the GP recursive fusion law, the proposed recursive DPGP fusion approach enables ecient online data fusion. The problem of active sensing for recursive DPGP learning and inference is also investigated by uncertainty reduction via expected mutual information. Simulation and experimental results show that the proposed approach successfully learns the models of moving targets and outperforms existing benchmark methods.

Keywords: Cooperating Robots, Climbing Robots, Autonomous Vehicle Navigation
Abstract: This paper proposes a novel cooperative system for an Unmanned Aerial Vehicle (UAV) and an Unmanned Ground Vehicle (UGV) which utilizes the UAV not only as a flying sensor but also as a tether attachment device. Two robots are connected with a tether, allowing the UAV to anchor the tether to a structure located at the top of a steep terrain, impossible to reach for UGVs. Thus, enhancing the poor traversability of the UGV by not only providing a wider range of scanning and mapping from the air, but also by allowing the UGV to climb steep terrains with the winding of the tether. In addition, we present an autonomous framework for the collaborative navigation and tether attachment in an unknown environment. The UAV employs visual inertial navigation with 3D voxel mapping and obstacle avoidance planning. The UGV makes use of the voxel map and generates an elevation map to execute path planning based on a traversability analysis. Furthermore, we compared the pros and cons of possible methods for the tether anchoring from multiple points of view. To increase the probability of successful anchoring, we evaluated the anchoring strategy with an experiment. Finally, the feasibility and capability of our proposed system were demonstrated by an autonomous mission experiment in the field with an obstacle and a cliff.

Keywords: Distributed Robot Systems, Multi-Robot Systems, Optimization and Optimal Control
Abstract: This paper considers a group of mobile sensing agents in a flow field and presents a distributed method for motion tomography (MT) that estimates the underlying flow field. MT formulates an underdetermined nonlinear system of equations as an inverse problem. Inspired by the Kaczmarz method which is an optimization approach for solving a linear system of equations, our previous work developed a nonlinear Kaczmarz method that solves the system of equations associated with MT. Considering distributed multi-agent systems for MT, this paper extends the nonlinear Kaczmarz method into a distributed framework. The distributed nonlinear Kaczmarz method is developed by formulating a constrained consensus problem that belongs to a class of projected consensus algorithms. To study the convergence and consensus for the method, its linear case is analyzed first and then its nonlinear case is discussed. The nonlinear case of the method is further validated through simulations by estimating a gyre flow field using mobile sensor networks with different numbers of neighboring agents. Resulting estimated flow fields are compared with a flow field estimated by its centralized counterpart.

Keywords: Distributed Robot Systems, Environment Monitoring and Management, Sensor Networks
Abstract: This paper presents a strategy to enable a team of mobile robots to adaptively sample and track a dynamic process. We propose a distributed strategy, where robots collect sparse sensor measurements, create a reduced-order model of a spatio-temporal process, and use this model to estimate field values for areas without sensor measurements of the dynamic process. The robots then use these estimates of the field, or inferences about the process, to adapt the model and reconfigure their sensing locations. The key contributions of this work are two-fold: 1) leveraging the dynamics of the process of interest to determine where to sample and how to estimate the process, and 2) maintaining fully distributed models, sensor measurements, and estimates of the time-varying process. We illustrate the application of the proposed solution in simulation and compare it to centralized and global approaches. We then test our approach with physical marine robots sampling a process in a water tank.

Keywords: Cognitive Human-Robot Interaction, Calibration and Identification, Human Detection and Tracking
Abstract: Advanced Internet of Things (IoT) techniques have made human-environment interaction much easier. Existing solutions usually enable such interactions without knowing the identities of action performers. However, identifying users who are interacting with environments is a key to enable personalized service. To provide such add-on service, we propose WTW (who takes what), a system that identifies which user takes what object. Unlike traditional vision-based approaches, which are typically vulnerable to blockage, our WTW combines the feature information of three types of data, i.e., images, skeletons and IMU data, to enable reliable user-object matching and identification. By correlating the moving trajectory of a user monitored by inertial sensors with the movement of an object recorded in the video, our WTW reliably identifies a user and matches him/her with the object on action. Our prototype evaluation shows that WTW achieves a recognition rate of over 90% even in a crowd. The system is reliable even when users locate close by and take objects roughly at the same time.

Keywords: Cognitive Human-Robot Interaction, Robot Audition
Abstract: Robots are usually equipped with microphones and cameras to perceive and understand the physical world. Though visual object detection technology has achieved great success, the detection in other modalities remains unsolved. In this paper, we establish a novel robotic sound-indicated visual object detection framework, and develop a two-stream weakly- supervised deep learning architecture to connect the visual and audio modalities for localizing the sounding object. A dataset is constructed from the AudioSet to validate the proposed method and some promising applications are demonstrated on robotic platforms.

Keywords: Human-Centered Automation, Learning from Demonstration, Deep Learning in Robotics and Automation
Abstract: Imitation learning has proven to be useful for many real-world problems, but approaches such as behavioral cloning suffer from data mismatch and compounding error issues. One attempt to address these limitations is the DAGGER algorithm, which uses the state distribution induced by the novice to sample corrective actions from the expert. Such sampling schemes, however, require the expert to provide action labels without being fully in control of the system. This can decrease safety and, when using humans as experts, is likely to degrade the quality of the collected labels due to perceived actuator lag. In this work, we propose HG-DAGGER, a variant of DAGGER that is more suitable for interactive imitation learning from human experts in real-world systems. In addition to training a novice policy, HG-DAGGER also learns a safety threshold for a model-uncertainty-based risk metric that can be used to predict the performance of the fully trained novice in different regions of the state space. We evaluate our method on both a simulated and real-world autonomous driving task, and demonstrate improved performance over both DAGGER and behavioral cloning.

Keywords: Cognitive Human-Robot Interaction, Localization, Gesture, Posture and Facial Expressions
Abstract: We present a system for interaction between co-located humans and mobile robots, which uses pointing gestures sensed by a wrist-mounted IMU. The operator begins by pointing, for a short time, at a moving robot. The system thus simultaneously determines: that the operator wants to interact; the robot they want to interact with; and the relative pose among the two. Then, the system can reconstruct pointed locations in the robot's own reference frame, and provide real-time feedback about them so that the user can adapt to misalignments. We discuss the challenges to be solved to implement such a system and propose practical solutions, including variants for fast flying robots and slow ground robots. We report different experiments with real robots and untrained users, validating the individual components and the system as a whole.

Keywords: Cognitive Human-Robot Interaction, Motion and Path Planning, Learning and Adaptive Systems
Abstract: Specifying complex task behaviours while ensuring good robot performance may be difficult for untrained users. We study a framework for users to specify rules for acceptable behaviour in a shared environment such as industrial facilities. As non-expert users might have little intuition about how their specification impacts the robot's performance, we design a learning system that interacts with the user to find an optimal solution. Using active preference learning, we iteratively show alternative paths that the robot could take on an interface. From the user feedback ranking the alternatives, we learn about the weights that users place on each part of their specification. We extend the user model from our previous work to a discrete Bayesian learning model and introduce a greedy algorithm for proposing alternative that operates on the notion of equivalence regions of user weights. We prove that with this algorithm the revision active learning process converges on the user-optimal path. In simulations on realistic industrial environments, we demonstrate the convergence and robustness of our approach.

Keywords: Calibration and Identification, Range Sensing, Mapping
Abstract: In a context of 3D mapping, it is very important to obtain accurate measurements from sensors. In particular, Light Detection And Ranging (LIDAR) measurements are typically treated as a zero-mean Gaussian distribution. We show that this assumption leads to predictable localisation drifts, especially when a bias related to measuring obstacles with high incidence angles is not taken into consideration. Moreover, we present a way to physically understand and model this bias, which generalizes to multiple sensors. Using an experimental setup, we measured the bias of the Sick LMS151, Velodyne HDL-32E, and Robosense RS-LiDAR-16 as a function of depth and incidence angle, and showed that the bias can reach 20 cm for high incidence angles. We then used our model to remove the bias from the measurements, leading to more accurate maps and a reduced localisation drift.

Keywords: Calibration and Identification, Sensor Fusion
Abstract: We present a novel open-source tool for extrinsic calibration of radar, camera and lidar. Unlike currently available offerings, our tool facilitates joint extrinsic calibration of all three sensing modalities on multiple measurements. Furthermore, our calibration target design extends existing work to obtain simultaneous measurements for all these modalities. We study how various factors of the calibration procedure affect the outcome on real multi-modal measurements of the target. Three different configurations of the optimization criterion are considered, namely using error terms for a minimal amount of sensor pairs, or using terms for all sensor pairs with additional loop closure constraints, or by adding terms for structure estimation in a probabilistic model. The experiments further evaluate how the number of calibration boards affect calibration performance, and robustness against different levels of zero mean Gaussian noise. Our results show that all configurations achieve good results for lidar to camera errors and that fully connected pose estimation shows the best performance for lidar to radar errors when more than five board locations are used.

Keywords: Calibration and Identification
Abstract: The ability to maintain and continuously update geometric calibration parameters of a mobile platform is a key functionality for every robotic system. These parameters include the intrinsic kinematic parameters of the platform, the extrinsic parameters of the sensors mounted on it and their time delays. In this paper, we present a unified pipeline for motion-based calibration of mobile platforms equipped with multiple heterogeneous sensors. We formulate a unified optimization problem to concurrently estimate the platform kinematic parameters, the sensors extrinsic parameters and their time delays. We analyze the influence of the trajectory followed by the robot on the accuracy of the estimate. Our framework automatically selects appropriate trajectories to maximize the information gathered and to obtain a more accurate parameters estimate. In combination with that, our pipeline observes the parameters evolution in long-term operation to detect possible values change in the parameters set. The experiments conducted on real data show a smooth convergence along with the ability to detect changes of parameters value. We release an open-source version of our framework to the community.

Keywords: Calibration and Identification, SLAM, Localization
Abstract: In this paper we perform in-depth observability analysis for both spatial and temporal calibration parameters of an aided inertial navigation system (INS) with global and/or local sensing modalities. In particular, we analytically show that both spatial and temporal calibration parameters are observable if the sensor platform undergoes random motion. More importantly, we identify four degenerate motion primitives that harm the calibration accuracy and thus should be avoided in reality whenever possible. Interestingly, we also prove that these degenerate motions would still hold even in the case where global pose measurements are available. Leveraging a particular multi-state constrained Kalman filter (MSCKF)-based vision-aided inertial navigation system (VINS) with online spatial and temporal calibration, we perform extensively both Monte-Carlo simulations and real-world experiments with the identified degenerate motions to validate our analysis.

Keywords: Calibration and Identification, Formal Methods in Robotics and Automation, Sensor Fusion
Abstract: A geometry-based analytic attitude estimation using a rate measurement and measurement of a single reference vector has been recently proposed. Because rigid body attitude estimation is a fundamentally nonlinear problem, the geometry-based method yields a better attitude estimate, when compared to other methods. A critical source of residual error in the geometric solution is on account of the noise and bias in the vector and rate measurements. A methodical perturbation analysis of the attitude estimate is performed in this letter that reveals the effects of measurement noise and bias, and provides means to compensate for, or filter out, such errors. Application of the filter and compensation provides better attitude estimation than a standard Extended Kalman filter using an optimal Kalman gain. The geometric method is first verified in experiments and then simulation results are provided that validate the better performance of the geometric attitude and bias estimator.

Keywords: Calibration and Identification, Kinematics, Medical Robots and Systems
Abstract: Currently, surgical continuum robots (CRs) are predominantly used as telemanipulators where modeling errors are overcome by the user. Such errors preclude their use for autonomous tasks. In this paper, we investigate the calibration of CRs with specific focus on capturing joint space errors due to homing offsets, assembly errors causing twist about the robot’s backbone, and uncertainty in the equilibrium bending shapes of segments of these robots. A kinematic framework focusing on multibackbone CR is presented with emphasis on deriving calibration identification Jacobians. This framework captures the coupling between twist and the equilibrium shapes of a continuum segment as a function of its bending angle. To capture equilibrium shape variations as a function of bending, a homotopy of curves is defined and represented by respective modal coefficients. The estimation of the calibration parameters is cast as a nonlinear least-squares problem. The framework is validated by simulations and experimentally using a single-port access surgery robot. We believe this calibration framework will facilitate semiautomation of surgical tasks carried out by CRs.

Keywords: Semantic Scene Understanding, AI-Based Methods, RGB-D Perception
Abstract: This paper proposes a hierarchical depthwise graph convolutional neural network (HDGCN) for point cloud semantic segmentation. The main chanllenge for learning on point clouds is to capture local structures or relationships. Graph convolution has the strong ability to extract local shape information from neighbors. Inspired by depthwise convolution, we propose a depthwise graph convolution which requires less memory consumption compared with the previous graph convolution. While depthwise graph convolution aggregates features channel-wisely, pointwise convolution is used to learn features across different channels. A customized block called DGConv is specially designed for local feature extraction based on depthwise graph convolution and pointwise convolution. The DGConv block can extract features from points and transfer features to neighbors while being invariant to different point orders. HDGCN is constructed by a series of DGConv blocks using a hierarchical structure which can extract both local and global features of point clouds. Experiments show that HDGCN achieves the state-of-the-art performance in the indoor dataset S3DIS and the outdoor dataset Paris-Lille-3D.

Keywords: Semantic Scene Understanding, Object Detection, Segmentation and Categorization, Computer Vision for Transportation
Abstract: In this work, we research and evaluate end-to-end learning of monocular semantic-metric occupancy grid mapping from weak binocular ground truth. The network learns to predict four classes, as well as a camera to bird's eye view mapping. At the core, it utilizes a variational encoder-decoder network that encodes the front-view visual information of the driving scene and subsequently decodes it into a 2-D top-view Cartesian coordinate system. The evaluations on Cityscapes show that the end-to-end learning of semantic-metric occupancy grids outperforms the deterministic mapping approach with flat-plane assumption by more than 12% mean IoU. Furthermore, we show that the variational sampling with a relatively small embedding vector brings robustness against vehicle dynamic perturbations, and generalizability for unseen KITTI data. Our network achieves real-time inference rates of approx. 35 Hz for an input image with a resolution of 256x512 pixels and an output map with 64x64 occupancy grid cells using a Titan V GPU.

Keywords: Computer Vision for Other Robotic Applications, Visual Tracking, Computer Vision for Automation
Abstract: Spatial convolution is arguably the most fundamental of 2D image processing operations. Conventional spatial image convolution can only be applied to a conventional image, that is, an array of pixel values (or similar image representation) that are associated with a single instant in time. Event cameras have serial, asynchronous output with no natural notion of an image frame, and each event arrives with a different timestamp. In this paper, we propose a method to compute the convolution of a linear spatial kernel with the output of an event camera. The approach operates on the event stream output of the camera directly without synthesising pseudo-image frames as is common in the literature. The key idea is the introduction of an internal state that directly encodes the convolved image information, which is updated asynchronously as each event arrives from the camera. The state can be read-off as-often-as and whenever required for use in higher level vision algorithms for real-time robotic systems. We demonstrate the application of our method to corner detection, providing an implementation of a Harris corner-response "state" that can be used in real-time for feature detection and tracking on robotic systems.

Keywords: Semantic Scene Understanding, Visual-Based Navigation, Visual Learning
Abstract: Legged robots have the potential to traverse diverse and rugged terrain. To find a safe and efficient navigation path and to carefully select individual footholds, it is useful to be able to predict properties of the terrain ahead of the robot. In this work, we propose a method to collect data from robot-terrain interaction and associate it to images. Using sparse data acquired in teleoperation experiments with a quadrupedal robot, we train a neural network to generate a dense prediction of the terrain properties in front of the robot. To generate training data, we project the foothold positions from the robot trajectory into on-board camera images. We then attach labels to these footholds by identifying the dominant features of the force-torque signal measured with sensorized feet. We show that data collected in this fashion can be used to train a convolutional network for terrain property prediction as well as weakly supervised semantic segmentation. Finally, we show that the predicted terrain properties can be used for autonomous navigation of the ANYmal quadruped robot.

Keywords: Computer Vision for Transportation, Intelligent Transportation Systems
Abstract: Semantic segmentation using deep neural networks has been widely explored to generate high-level contextual information for autonomous vehicles. To acquire a complete 180◦ semantic understanding of the forward surroundings, we propose to stitch semantic images from multiple cameras with varying orientations. However, previously trained semantic segmentation models showed unacceptable performance after significant changes to the camera orientations and the lighting conditions. To avoid timeconsuming hand labeling, we explore and evaluate the use of data augmentation techniques, specifically skew and gamma correction, from a practical real-world standpoint to extend the existing model and provide more robust performance. The experimental results presented have shown significant improvements with varying illumination and camera perspective changes. A comparison of the results from a high-performance network (PSPNet), and a realtime capable network (ENet) is provided.

Keywords: SLAM, Range Sensing, Learning and Adaptive Systems
Abstract: The fusion of Iterative Closest Point (ICP) registrations in existing state estimation frameworks relies on an accurate estimation of their uncertainty. In this paper, we study the estimation of this uncertainty in the form of a covariance. First, we scrutinize the limitations of existing closed-form covariance estimation algorithms over 3D datasets. Then, we set out to estimate the covariance of ICP registrations through a data-driven approach, with over 5 100 000 registrations on 1020 pairs from real 3D point clouds. We assess our solution upon a wide spectrum of environments, ranging from structured to unstructured and indoor to outdoor. The capacity of our algorithm to predict covariances is accurately assessed, as well as the usefulness of these estimations for uncertainty estimation over trajectories. The proposed method estimates covariances better than existing closed-form solutions, and makes predictions that are consistent with observed trajectories.

Keywords: SLAM, Visual-Based Navigation, Recognition
Abstract: A key feature in robotics applications is to recognize whether the current environment observation corresponds to a previously visited location. Should the place be recognized by the robot, a Loop Closure Detection (LCD) has occurred. The letter in hand deploys a novel low complexity LCD method based on the representation of the route by unique Visual Features (VFs). Each of these VFs, referred to as “Tracked Word” (TW), is generated on-line through a tracking technique coupled with a guided-feature-detection mechanism and belongs to a group of successive images. During the robot’s navigation, new TWs are added to the database forming a bag of tracked words. When querying the database seeking for loop closures, the new local-feature-descriptors are associated with the nearest neighboring TWs in the map casting votes to the corresponding instances. The system relies on a probabilistic method to select the most suitable loop closing pair, based on the number of votes each location polls. The proposed system depends solely on the appearance information of the scenes on the trajectory, without requiring any pre-training phase. The evaluation of the method is administered via a variety of tests with several community datasets, thus proving its capability of achieving high recall rates for perfect precision.

Keywords: SLAM
Abstract: Simultaneous trajectory estimation and mapping (STEAM) offers an efficient approach to continuous-time trajectory estimation, by representing the trajectory as a Gaussian process (GP). Previous formulations of the STEAM framework use a GP prior that assumes white-noise-on-acceleration, with the prior mean encouraging constant body-centric velocity. We show that such a prior cannot sufficiently represent trajectory sections with non-zero acceleration, resulting in a bias to the posterior estimates.

This paper derives a novel motion prior that assumes white-noise-on-jerk, where the prior mean encourages constant body-centric acceleration. With the new prior, we formulate a variation of STEAM that estimates the pose, body-centric velocity, and body-centric acceleration. By evaluating across several datasets, we show that the new prior greatly outperforms the white-noise-on-acceleration prior in terms of solution accuracy.

Keywords: SLAM
Abstract: A local map module is often implemented in modern VO/VSLAM systems to improve data association and pose estimation. Conventionally, the local map contents are determined by co-visibility. While co-visibility is cheap to establish, it utilizes the relatively-weak temporal prior (i.e. seen before, likely to be seen now), therefore admitting more features into the local map than necessary. This paper describes an enhancement to co-visibility local map building by incorporating a strong appearance prior, which leads to a more compact local map and latency reduction in downstream data association. The appearance prior collected from the current image influences the local map contents: only the map features visually similar to the current measurements are potentially useful for data association. To that end, mapped features are indexed and queried with Multi-index Hashing (MIH). An online hash table selection algorithm is developed to further reduce the query overhead of MIH and the local map size. The proposed appearance-based local map building method is integrated into a state-of-the-art VO/VSLAM system. When evaluated on two public benchmarks, the size of the local map, as well as the latency of real-time pose tracking in VO/VSLAM are significantly reduced. Meanwhile, the VO/VSLAM mean performance is preserved or improves.

Keywords: SLAM, Visual-Based Navigation, Sensor Fusion
Abstract: Visual-Inertial Odometry (VIO) algorithms typically rely on a point cloud representation of the scene that does not model the topology of the environment. A 3D mesh instead offers a richer, yet lightweight, model. Nevertheless, building a 3D mesh out of the sparse and noisy 3D landmarks triangulated by a VIO algorithm often results in a mesh that does not fit the real scene. In order to regularize the mesh, previous approaches decouple state estimation from the 3D mesh regularization step, and either limit the 3D mesh to the current frame [1], [2] or let the mesh grow indefinitely [3], [4]. We propose instead to tightly couple mesh regularization and state estimation by detecting and enforcing structural regularities in a novel factor-graph formulation. We also propose to incrementally build the mesh by restricting its extent to the time-horizon of the VIO optimization; the resulting 3D mesh covers a larger portion of the scene than a per-frame approach while its memory usage and computational complexity remain bounded. We show that our approach successfully regularizes the mesh, while improving localization accuracy, when structural regularities are present, and remains operational in scenes without regularities.

Keywords: Gesture, Posture and Facial Expressions, Surveillance Systems
Abstract: The paper presents an unsupervised out-of-context action (O2CA) paradigm that is based on facilitating understanding by separately presenting both human action and context within a video sequence. As a means of generating an unsupervised label, we comprehensively evaluate responses from action-based (ActionNet) and context-based (ContextNet) convolutional neural networks (CNNs). Additionally, we have created three synthetic databases based on the human action (UCF101, HMDB51) and motion capture (mocap) (SURREAL) datasets. We then conducted experimental comparisons between our approach and conventional approaches. We also compared our unsupervised learning method with supervised learning using an O2CA ground truth given by synthetic data. From the results obtained, we achieved a 96.8 score on Synth-UCF, a 96.8 score on Synth-HMDB, and 89.0 on SURREAL-O2CA with F-score.