Keywords: Deep Learning for Visual Perception, Recognition, Semantic Scene Understanding
Abstract: Recognising in what type of environment one is located is an important perception task. For instance, for a robot operating indoors it is helpful to be aware whether it is in a kitchen, a hallway or a bedroom. Existing approaches attempt to classify the scene based on 2D images or 2.5D range images. Here, we study scene recognition from 3D point cloud (or voxel) data, and show that it greatly outperforms methods based on 2D birds-eye views. Moreover, we advocate multi-task learning as a way to improve scene recognition, building on the fact that the scene type is highly correlated with the objects in the scene, and therefore with its semantic segmentation into different object classes. In a series of ablation studies, we show that successful scene recognition is not just the recognition of individual objects unique to some scene type (such as a bathtub), but depends on several different cues, including coarse 3D geometry, colour, and the (implicit) distribution of object categories. Moreover, we demonstrate that surprisingly sparse 3D data is sufficient to classify indoor scenes with good accuracy.

Keywords: Deep Learning for Visual Perception, Representation Learning
Abstract: Estimating 6D poses of rigid objects from RGB images is an important but challenging task. This is especially true for textureless objects with strong symmetry, since they have only sparse visual features to be leveraged for the task and their symmetry leads to pose ambiguity. The implicit encoding of orientations learned by autoencoders [31], [32] has demonstrated its effectiveness in handling such objects without requiring explicit pose labeling. In this paper, we further improve this methodology with two key technical contributions. First, we use edge cues to complement the color images with more discriminative features and reduce the domain gap between the real images for testing and the synthetic ones for training. Second, we enhance the regularity of the implicitly learned pose representations by a self-supervision scheme to enforce the geometric prior that the latent representations of two images presenting nearby rotations should be close too. Our approach achieves the state-of-the-art performance on the T-LESS benchmark in the RGB domain; its evaluation on the LINEMOD dataset also outperforms other synthetically trained approaches. Extensive ablation tests demonstrate the improvements enabled by our technical designs. Our code is publicly available for research use.

Keywords: AI-Based Methods, Cognitive Human-Robot Interaction, Computer Vision for Other Robotic Applications
Abstract: Humans perceive and describe their surroundings with qualitative statements (e.g., "Alice's hand is in contact with a bottle."), rather than quantitative values (e.g., 6-D poses of Alice's hand and a bottle). Qualitative spatial representation (QSR) is a framework that represents the spatial information of objects in a qualitative manner. Region connection calculus (RCC), qualitative trajectory calculus (QTC), and qualitative distance calculus (QDC) are some popular QSR calculi. With the recent development of computer vision, it is important to compute QSR calculi from the visual inputs (e.g., RGB-D images). In fact, many QSR application domains (e.g., human activity recognition (HAR) in robotics) involve visual inputs. We propose a qualitative spatial representation network (QSRNet) that computes the three QSR calculi (i.e., RCC, QTC, and QDC) from the RGB-D images. QSRNet has the following novel contributions. First, QSRNet models the dependencies among the three QSR calculi. We introduce the dependencies as kinematics for QSR because they are analogous to the kinematics in classical mechanics. Second, QSRNet applies the 3-D point cloud instance segmentation to compute the QSR calculi. The experimental results show that QSRNet improves the accuracy in comparison to the other state-of-the-art techniques.

Keywords: Deep Learning for Visual Perception, Representation Learning, Deep Learning in Grasping and Manipulation
Abstract: We consider the problem of acquiring mechanical knowledge through visual cues to help robots use objects in new situations. In this work, we propose a novel deep learning approach that allows a robot to acquire mechanical knowledge from 3D point clouds. This presents two main challenges. The first challenge is that a robot needs to infer novel objects’ functions from its experience. Secondly, the robot should also need to know how to manipulate these novel objects. To solve these problems, we present a two-branch deep neural network. The first branch detects function parts from the point clouds while the second branch predicts offset poses. Fusing the results from these two branches, our approach can not only detect what functions the novel objects may have but also generate key object states which can be used to guide a robot to manipulate these objects. We show that even though most of the training samples are synthetic data, our model still learns useful features and outputs proper results. Finally, we evaluate our approach on a real robot to run a series of tasks. The experimental results show that our approach has the capability to transfer mechanical knowledge in new situations.

Keywords: Deep Learning in Grasping and Manipulation, Visual Learning, Assembly
Abstract: Manipulation and assembly tasks require non-trivial planning of actions depending on the environment and the final goal. Previous work in this domain often assembles particular instances of objects from known sets of primitives. In contrast, we aim to handle varying sets of primitives and to construct different objects of a shape category. Given a single object instance of a category, e.g. an arch, and a binary shape classifier, we learn a visual policy to assemble other instances of the same category. In particular, we propose a disassembly procedure and learn a state policy that discovers new object instances and their assembly plans in state space. We then render simulated states in the observation space and learn a heatmap representation to predict alternative actions from a given input image. To validate our approach, we first demonstrate its efficiency for building object categories in state space. We then show the success of our visual policies for building arches from different primitives. Moreover, we demonstrate (i) the reactive ability of our method to re-assemble objects using additional primitives and (ii) the robust performance of our policy for unseen primitives resembling building blocks used during training. Our visual assembly policies are trained with no real images and reach up to 95% success rate when evaluated on a real robot.

Keywords: Cognitive Control Architectures, AI-Based Methods, Learning from Demonstration
Abstract: To act effectively in its environment, a cognitive robot needs to understand the causal dependencies of all intermediate actions leading up to its goal. For example, the system has to infer that it is instrumental to open a cupboard door before trying to grasp an object inside the cupboard. In this paper, we introduce a novel learning method for extracting instrumental dependencies by following the scientific cycle of observations, generation of causal hypotheses and testing through experiments. Our method uses a virtual reality dataset containing observations from human activities to generate hypotheses about causal dependencies between actions. It detects pairs of actions with a high temporal co-occurrence and verifies if one action is instrumental in executing the other action through mental simulation in a virtual reality environment which represents the system's mental model. Our approach is able to extract all present instrumental action dependencies while significantly reducing the search space for mental simulation, resulting in a 6-fold reduction in computational time.

Keywords: Cognitive Control Architectures, Representation Learning, Reinforecment Learning
Abstract: This paper introduces an algorithm for discovering implicit and delayed causal relations between events observed by a robot at arbitrary times, with the objective of improving data-efficiency and interpretability of model-based reinforcement learning (RL) techniques. The proposed algorithm initially predicts observations with the Markov assumption, and incrementally introduces new hidden variables to explain and reduce the stochasticity of the observations. The hidden variables are memory units that keep track of pertinent past events. Such events are systematically identified by their information gains. The learned transition and reward models are then used for planning. Experiments on simulated and real robotic tasks show that this method significantly improves over current RL techniques.

Keywords: AI-Based Methods, Cognitive Control Architectures, Manipulation Planning
Abstract: Current approaches combining task and motion planning require intensive geometric and symbolic reasoning to find feasible motions for task execution. The poor expressiveness of task planning domains for characterizing geometric changes with actions and the difficulties faced by current approaches to efficiently identify motion dependencies for plan execution produce expensive callings to motion planning on unfeasible actions and intensive reasoning to find realizable plans. In this work we combine two recent approaches to address these problems. Task planning is carried out using an object-centered description of geometric relations that consistently characterizes changes in the object configuration space. Plan execution is implemented using a symbol to motion hierarchical decomposition that depends on consecutive actions in the plan, rather than on single actions, which permits considering motion dependencies across plan actions for a successful execution.

Keywords: Behavior-Based Systems, Hybrid Logical/Dynamical Planning and Verification, Cognitive Control Architectures
Abstract: A robot control system is often composed of a set of low level continuous controllers and a switching policy that decides which of those continuous controllers to apply at each time instant. The switching policy can be either a Finite State Machine (FSM), or a Behavior Tree (BT). In previous work we have shown how to create BTs using a backward chained approach that results in a reactive goal directed policy. This policy can be thought of as providing disturbance rejection at the task level in the sense that if a disturbance changes the state in such a way that the currently running continuous controller cannot handle it, the policy will switch to the appropriate continuous controller. In this paper we show how to provide convergence guarantees for such policies.

Keywords: Cognitive Control Architectures, Control Architectures and Programming, Distributed Robot Systems
Abstract: It has been claimed that a main advantage of cognitive architectures (compared to other types of specialized robotic architectures) is that they are task-general and can thus learn to perform any task as long as they have the right perceptual and action primitives. In this paper, we provide empirical evidence for this claim by directly comparing a high-performing custom robotic architecture developed for the standardized robotic “FetchIt!” challenge task to a hybrid cognitive robotic architecture that allows for online one-shot task learning and task modifications through natural language instructions. The results show that there is no disadvantage of running the hybrid architecture (i.e., no significant difference in overall performance or computational overhead compared to the custom architecture) while adding the flexibility of online one-shot task instruction and modification not available in the custom architecture.

Keywords: Biomimetics, Mapping, Cognitive Control Architectures
Abstract: This paper presents a robotic episodic cognitive learning framework based on the biological cognitive mechanism of hippocampal spatial cells. By emphasizing the cognition process and episodic memory in brain, the framework adopts the velocity modulated grid cells and place cells to afford the robot position cognition, abstracts the state neurons to represent the robotic state, and uses state neurons’ activity and connection to simulate the episodic memory construction process. The episodic memory is formed by a sequence of particular events consisting of visual features, state neuron, phase, and pose information. Besides an episodic-cognitive map building approach based on this framework is proposed, which performs closed-loop correction by resetting the spatial cells phase to keep the map accurate. The episodic-cognitive map built in this paper is a topological metric map to describe the topological relations of the particular events coordinates in the unknown environment. The framework is applied on a mobile robot platform, the robotic episodic cognitive learning and episodic-cognitive map building approach are investigated. The robotic experiments demonstrate that the framework can effectively achieve the robotic incremental accumulative learning, update the spatial cognition to the environment and construct the episodic-cognitive map.

Keywords: Visual Learning, Visual Tracking
Abstract: 3D object trackers usually require training on large amounts of annotated data that is expensive and time-consuming to collect. Instead, we propose leveraging vast unlabeled datasets by self-supervised metric learning of 3D object trackers, with a focus on data association. Large scale annotations for unlabeled data are cheaply obtained by automatic object detection and association across frames. We show how these self-supervised annotations can be used in a principled manner to learn point-cloud embeddings that are effective for 3D tracking. We estimate and incorporate uncertainty in self-supervised tracking to learn more robust embeddings, without needing any labeled data. We design embeddings to differentiate objects across frames, and learn them using uncertainty-aware self-supervised training. Finally, we demonstrate their ability to perform accurate data association across frames, towards effective and accurate 3D tracking.

Keywords: Visual Tracking, Deep Learning for Visual Perception, Sensor Fusion
Abstract: This paper presents F-Siamese Tracker, a novel approach for single object tracking prominently characterized by more robustly integrating 2D and 3D information to reduce redundant search space. A main challenge in 3D single object tracking is how to reduce search space for generating appropriate 3D candidates. Instead of solely relying on 3D proposals, firstly, our method leverages the Siamese network applied on RGB images to produce 2D region proposals which are then extruded into 3D viewing frustums. Besides, we perform an online accuracy validation on the 3D frustum to generate refined point cloud searching space, which can be embedded directly into the existing 3D tracking backbone. For efficiency, our approach gains better performance with fewer candidates by reducing search space. In addition, benefited from introducing the online accuracy validation, for occasional cases with strong occlusions or very sparse points, our approach can still achieve high precision, even when the 2D Siamese tracker loses the target. This approach allows us to set a new state-of-the-art in 3D single object tracking by a significant margin on a sparse outdoor dataset (KITTI tracking). Moreover, experiments on 2D single object tracking show that our framework boosts 2D tracking performance as well.

Keywords: Visual Learning, Deep Learning for Visual Perception, Computer Vision for Transportation
Abstract: Predicting the future is an important aspect for decision-making in robotics or autonomous driving systems, which heavily rely upon visual scene understanding. While prior work attempts to predict future video pixels, anticipate activities or forecast future scene semantic segments from segmentation of the preceding frames, methods that predict future semantic segmentation solely from the previous frame RGB data in a single end-to-end trainable model do not exist. In this paper, we propose a temporal encoder-decoder network architecture that encodes RGB frames from the past and decodes the future semantic segmentation. The network is coupled with a new knowledge distillation training framework specific for the forecasting task. Our method, only seeing preceding video frames, implicitly models the scene segments while simultaneously accounting for the object dynamics to infer the future scene semantic segments. Our results on Cityscapes and Apolloscape outperform the baseline and current state-of-the-art methods. Code will be available soon.

Keywords: Visual Learning, Representation Learning, Deep Learning for Visual Perception
Abstract: As warehouses, storage facilities and factories become more expanded and equipped with smart devices, there is a substantial need for rapid, intelligent and autonomous detection of unusual and potentially hazardous situations, also called anomalies. In particular for Autonomous Guided Vehicles (AGVs) that drive around these premises independently, unforeseen obstructions along their path---e.g.~a cardboard box in the middle of a corridor or bumps in the floor---and sudden or unexpected actions executed by personnel---e.g.~someone walking in a restricted area---make it hard for AGVs to navigate safely. We therefore propose a novel approach to detect such anomalies in an unsupervised manner by measuring Bayesian surprise: whenever an event is observed that does not align with the agent's prior knowledge of the world, this event is deemed surprising and could indicate an anomaly. This paper lays out the details on how to learn both the prior and posterior models of an AGV that drives around a warehouse and observes the environment through an RGBD camera. In the experiments we show that our Bayesian surprise approach outperforms a baseline that is traditionally used to detect anomalies in sequences of images.

Keywords: Visual Tracking, Deep Learning for Visual Perception, Deep Learning in Grasping and Manipulation
Abstract: This paper deals with predicting future 3D motions of 3D object scans from the previous two consecutive frames. Previous methods mostly focus on sparse motion prediction in the form of skeletons. While in this paper, we focus on predicting dense 3D motions in the form of 3D point clouds. To approach this problem, we propose a self-supervised approach that leverages the power of the deep neural network to learn a continuous flow function of 3D point clouds that can predict temporally consistent future motions and naturally bring out the correspondences among consecutive point clouds at the same time. More specifically, in our approach, to eliminate the unsolved and challenging process of defining a discrete point convolution on 3D point cloud sequences to encode spatial and temporal information, we introduce a learnable latent code to represent the temporal-aware shape descriptor, which is optimized during the model training. Moreover, a temporally consistent motion Morpher is proposed to learn a continuous flow field which deforms a 3D scan from the current frame to the next frame. We perform extensive experiments on D-FAUST, SCAPE, and TOSCA benchmark data sets. The results demonstrate that our approach is capable of handling temporally inconsistent input and produces consistent future 3D motion while requiring no ground truth supervision.

Keywords: Visual Learning, Robot Safety, Deep Learning for Visual Perception
Abstract: Robots operating in human environments must be careful, when executing their manipulation skills, not to disturb nearby objects. This requires robots to reason about the effect of their manipulation choices by accounting for the support relationships among objects in the scene. Humans do this in part by visually assessing their surroundings and using physics intuition for how likely it is that a particular object can be safely manipulated (i.e., cause no disruption in the rest of the scene). Existing work has shown that deep convolutional neural networks can learn intuitive physics over images generated in simulation and determine the stability of a scene in the real world. In this paper, we extend these physics intuition models to the task of assessing safe object extraction by conditioning the visual images on specific objects in the scene. Our results, in both simulation and real-world settings, show that with our proposed method, physics intuition models can be used to inform a robot of which objects can be safely extracted and from which direction to extract them.

Keywords: Computer Vision for Other Robotic Applications, Novel Deep Learning Methods, Computer Vision for Automation
Abstract: Point-clouds are a popular choice for robotics and computer vision tasks due to their accurate shape description and direct acquisition from range-scanners. This demands the ability to synthesize and reconstruct high-quality point-clouds. Current deep generative models for 3D data generally work on simplified representations (e.g., voxelized objects) and cannot deal with the inherent redundancy and irregularity in point-clouds. A few recent efforts on 3D point-cloud generation offer limited resolution and their complexity grows with the increase in output resolution. In this paper, we develop a principled approach to synthesize 3D point-clouds using a spectral-domain Generative Adversarial Network (GAN). Our spectral representation is highly structured and allows us to disentangle various frequency bands such that the learning task is simplified for a GAN model. As compared to spatial-domain generative approaches, our formulation allows us to generate high-resolution point-clouds with minimal computational overhead. Furthermore, we propose a fully differentiable block to transform from {the} spectral to the spatial domain and back, thereby allowing us to integrate knowledge from well-established spatial models. We demonstrate that Spectral-GAN performs well for point-cloud generation task. Additionally, it can learn {a} highly discriminative representation in an unsupervised fashion and can be used to accurately reconstruct 3D objects. Our codes are available at https://github.com/samgregoost/Spectral-GAN/

Keywords: Omnidirectional Vision, Computer Vision for Automation, Deep Learning for Visual Perception
Abstract: In classical computer vision, rectification is an integral part of multi-view depth estimation. It typically includes epipolar rectification and lens distortion correction. This process simplifies the depth estimation significantly, and thus it has been adopted in CNN approaches. However, rectification has several side effects, including a reduced field-of-view (FOV), resampling distortion, and sensitivity to calibration errors. The effects are particularly pronounced in case of significant distortion (e.g., wide-angle fisheye cameras). In this paper, we propose a generic scale-aware self-supervised pipeline for estimating depth, euclidean distance, and visual odometry from unrectified monocular videos. We demonstrate a similar level of precision on the unrectified KITTI dataset with barrel distortion comparable to the rectified KITTI dataset. The intuition being that the rectification step can be implicitly absorbed within the CNN model, which learns the distortion model without increasing complexity. Our approach does not suffer from a reduced field of view and avoids computational costs for rectification at inference time. To further illustrate the general applicability of the proposed framework, we apply it to wide-angle fisheye cameras with 190 degrees horizontal field-of-view (FOV). The training framework UnRectDepthNet takes in the camera distortion model as an argument and adapts projection and unprojection functions accordingly. The proposed algorithm is evaluated further on the KITTI dataset, and we achieve state-of-the-art results that improve upon our previous work FisheyeDistanceNet~cite{kumar2019fisheyedistancenet}. Qualitative results on a distorted test scene video sequence indicate excellent performance https://youtu.be/K6pbx3bU4Ss.

Keywords: Computer Vision for Other Robotic Applications, Deep Learning for Visual Perception, Autonomous Vehicle Navigation
Abstract: Given an image or a video captured from a monocular camera, amodal layout estimation is the task of predicting semantics and occupancy in bird’s eye view. The term amodal implies we also reason about entities in the scene that are occluded or truncated in image space. While several recent efforts have tackled this problem, there is a lack of standardization in task specification, datasets, and evaluation protocols. We address these gaps with AutoLay, a dataset and benchmark for amodal layout estimation from monocular images. AutoLay encompasses driving imagery from two popular datasets: KITTI Tracking [1] and Argoverse [2]. In addition to fine-grained attributes such as lanes, sidewalks, and vehicles, we also provide semantically annotated 3D pointclouds. We implement several baselines and bleeding edge approaches, and release our data and code

Keywords: Sensor Networks, Behavior-Based Systems
Abstract: Nowadays, the prevalence of sensor networks has enabled tracking of the states of dynamic objects for a wide spectrum of applications from autonomous driving to environmental monitoring and urban planning. However, tracking real-world objects often faces two key challenges: First, due to the limitation of individual sensors, state estimation needs to be solved in a collaborative and distributed manner. Second, the objects’ movement behavior model is unknown, and needs to be learned using sensor observations. In this work, for the first time, we formally formulate the problem of simultaneous state estimation and behavior learning in a sensor network. We then propose a simple yet effective solution to this new problem by extending the Gaussian process-based Bayes filters (GP-BayesFilters) to an online, distributed setting. The effectiveness of the proposed method is evaluated on tracking objects with unknown movement behaviors using both synthetic data and data collected from a multi-robot platform.

Keywords: Computer Vision for Other Robotic Applications, Gesture, Posture and Facial Expressions, Social Human-Robot Interaction
Abstract: We present ProxEmo, a novel end-to-end emotion prediction algorithm for socially aware robot navigation among pedestrians. Our approach predicts the perceived emotions of a pedestrian from walking gaits, which is then used for emotion-guided navigation taking into account social and proxemic constraints. To classify emotions, we propose a multi-view skeleton graph convolution-based model that works on a commodity camera mounted onto a moving robot. Our emotion recognition is integrated into a mapless navigation scheme and makes no assumptions about the environment of pedestrian motion. It achieves a mean average emotion prediction precision of 82.47 % on the Emotion-Gait benchmark dataset. We outperform current state-of-art algorithms for emotion recognition from 3D gaits. We highlight its benefits in terms of navigation in indoor scenes using a Clearpath Jackal robot.

Keywords: Sensor-based Control, Probability and Statistical Methods, Optimization and Optimal Control
Abstract: Gaussian processes (GPs) have been exploited for various applications even including online learning. To learn time-varying hyperparameters from an information-limited sparse data stream, we consider the infinite-horizon Gaussian process (IHGP) with a low computational complexity for online learning. For example, the IHGP framework could provide efficient GP online learning with a sparse data stream in mobile devices. However, we show that the originally proposed IHGP has difficulty in learning time-varying hyperparameters online from the sparse data stream due to the numerically approximated gradient of the marginal likelihood function. In this paper, we show how to extend the IHGP in order to learn time-varying hyperparameters using a sparse and non-stationary data stream. In particular, our solution approach offers the exact gradient as the solution of a Lyapunov equation. Therefore, our approach achieves better performance with a sparse data stream while still keeping the computational complexity low. Finally, we present the comparison results to demonstrate that our approach outperforms the originally proposed IHGP on practical applications with sparse data streams. To demonstrate the effectiveness of our approach, we consider a multi-rate sensor fusion or an interpolation problem where slow vision systems need to be combined with other fast sensory units for feedback control in the field of autonomous driving. In particular, we apply our approach to vehicle lateral position error estimation together with a deep learning model for autonomous driving using non-stationary lateral position error signals in a model-free and data-driven fashion.

Keywords: Semantic Scene Understanding, Deep Learning for Visual Perception, SLAM
Abstract: Due to the difficulty in generating the effective descriptors which are robust to occlusion and viewpoint changes, place recognition for 3D point cloud remains an open issue. Unlike most of the existing methods that focus on extracting local, global, and statistical features of raw point clouds, our method aims at the semantic level that can be superior in terms of robustness to environmental changes. Inspired by the perspective of humans, who recognize scenes through identifying semantic objects and capturing their relations, this paper presents a novel semantic graph based approach for place recognition. First, we propose a novel semantic graph representation for the point cloud scenes by reserving the semantic and topological information of the raw point cloud. Thus, place recognition is modeled as a graph matching problem. Then we design a fast and effective graph similarity network to compute the similarity. Exhaustive evaluations on the KITTI dataset show that our approach is robust to the occlusion as well as viewpoint changes and outperforms the state-of-the-art methods with a large margin. Our code is available at: https://github.com/kxhit/SG_PR.

Keywords: Semantic Scene Understanding, Cognitive Human-Robot Interaction
Abstract: For high-level human-robot interaction tasks, 3D scene understanding is important and non-trivial for autonomous robots. However, parsing and utilizing effective environment information of the 3D scene is not trivial due to the complexity of the 3D environment and the limited ability for reasoning about our visual world. Although there have been great efforts on semantic detection and scene analysis, the existing solutions for parsing and representation of the 3D scene still fail to preserve accurate semantic information and equip sufficient applicability. This study proposes a bottom-up construction framework for structured 3D scene graph generation, which efficiently describes the objects, relations and attributes of the 3D indoor environment with structured representation. In the proposed method, we adopt visual perception to capture the semantic information and inference from scene priors to calculate the optimal parse graph. Afterwards, an improved probabilistic grammar model is used to represent the scene priors. Experiment results demonstrate that the proposed framework significantly outperforms existing methods in terms of accuracy, and a demonstration is provided to verify the applicability in applying to high-level human-robot interaction tasks.The supplementary video can be accessed at the following link: https://youtu.be/vEWNxnSwmKI.

Keywords: Surveillance Systems, Perception-Action Coupling, Reactive and Sensor-Based Planning
Abstract: Current approaches to physical security suffer from high false alarm rates and frequent human operator involvement, despite the relative rarity of real-world threats. We present a novel architecture for autonomous adaptive physical security called autonomous detection and assessment with moving sensors (ADAMS). ADAMS is a framework for reducing nuisance and false alarms by placing mobile robotic platforms equipped with sensors outside the normal asset perimeter. These robotic agents integrate sensor data from multiple perspectives over time, and autonomously move to obtain the best new data to reduce uncertainty in the threat scene. Inferences drawn from data fused over time provide ultimate decisions regarding whether to alert human operators. This paper describes the framework and algorithms used in a prototype ADAMS implementation. We describe the results of simulations comparing this framework to alternate paradigms. These simulations show ADAMS has a 4x increase in the range at which threats are identified versus traditional static sensors, and a 5x reduction in false alarms triggered versus frameworks where all sensor detections become alarms, leading to reduced operator load. Further, these simulations show this framework for reacting to new potential threats significantly outperforms methods which merely patrol the site. We also present the results of preliminary hardware trials of an exemplar prototype system, providing limited validation of the simulations in a real-time physical demonstration.

Keywords: Semantic Scene Understanding, Distributed Robot Systems, SLAM
Abstract: We present an approach for multi-robot consistent distributed localization and semantic mapping in an unknown environment, considering scenarios with classification ambiguity, where objects' visual appearance generally varies with viewpoint. Our approach addresses such a setting by maintaining a distributed posterior hybrid belief over continuous localization and discrete classification variables. In particular, we utilize a viewpoint-dependent classifier model to leverage the coupling between semantics and geometry. Moreover, our approach yields a consistent estimation of both continuous and discrete variables, with the latter being addressed for the first time, to the best of our knowledge. We evaluate the performance of our approach in a multi-robot semantic SLAM simulation and in a real-world experiment, demonstrating an increase in both classification and localization accuracy compared to maintaining a hybrid belief using local information only.

Keywords: Semantic Scene Understanding, Deep Learning for Visual Perception, Object Detection, Segmentation and Categorization
Abstract: Neural networks have recently achieved impressive success in semantic and instance segmentation on 2D images. However, their capabilities have not been fully explored to address semantic instance segmentation on unstructured 3D point cloud data. Digging into the regional feature representation to boost point cloud comprehension, we propose a region-feature-enhanced structure consisting of adaptive regional feature complementary (ARFC) module and affinity-based regional relational reasoning (AR^3) module. The ARFC module aims to complement low-level features of sparse regions adaptively. The AR^3 module emphasizes on mining the potential reasoning relationships between high-level features based on affinity. Both the ARFC and AR^3 modules are plug-and-play. Besides, a novel dual spatial-aware discriminative loss is proposed to improve the discrimination of instance embedding by incorporating location information. Our proposal-free point cloud instance segmentation network (RegionNet) equipped with the region-feature-enhanced structure and dual spatial-aware discriminative loss achieves state-of-the-art performance on S3DIS dataset and ScanNet-v2 dataset.

Keywords: Semantic Scene Understanding, Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception
Abstract: LIDAR semantic segmentation is an essential task that provides 3D semantic information about the environment to robots. Fast and efficient semantic segmentation methods are needed to match the strong computational and temporal restrictions of many real-world robotic applications. This work presents 3D-MiniNet, a novel approach for LIDAR semantic segmentation that combines 3D and 2D learning layers. It first learns a 2D representation from the raw points through a novel projection which extracts local and global information from the 3D data. This representation is fed to an efficient 2D Fully Convolutional Neural Network (FCNN) that produces a 2D semantic segmentation. These 2D semantic labels are re-projected back to the 3D space and enhanced through a post-processing module. The main novelty in our strategy relies on the projection learning module. Our detailed ablation study shows how each component contributes to the final performance of 3D-MiniNet. We validate our approach on well known public benchmarks (SemanticKITTI and KITTI), where 3D-MiniNet gets state-of-the-art results while being faster and more parameter-efficient than previous methods.

Keywords: Computer Vision for Transportation, Object Detection, Segmentation and Categorization, Transfer Learning
Abstract: Inferring semantic information towards an understanding of the surrounding environment is crucial for autonomous vehicles to drive safely. Deep learning-based segmentation methods can infer semantic information directly from laser range data, even in the absence of other sensor modalities such as cameras. In this paper, we address improving the generalization capabilities of such deep learning models to range data that was captured using a different sensor and in situations where no labeled data is available for the new sensor setup. Our approach assists the domain transfer of a LiDAR-only semantic segmentation model to a different sensor and environment exploiting existing geometric mapping systems. To this end, we fuse sequential scans in the source dataset into a dense mesh and render semi-synthetic scans that match those of the target sensor setup. Unlike simulation, this approach provides a real-to-real transfer of geometric information and delivers additionally more accurate remission information. We implemented and thoroughly tested our approach by transferring semantic scans between two different real-world datasets with different sensor setups. Our experiments show that we can improve the segmentation performance substantially with zero manual re-labeling. This approach solves the number one feature request since we released our semantic segmentation library LiDAR-bonnetal

Keywords: Planning, Scheduling and Coordination, Object Detection, Segmentation and Categorization
Abstract: In this paper, we propose a camera control system towards occasionally videoing preassigned objects. Based on the technique of real-time visual detection and tracking, using the Kalman filter and re-identification (ReID), we propose continuous composition of lens, based on the atomic rules of shots, and give the trajectory planning of the camera, to generate the PID controller to the pan-tilt. By both simulation and emulation by frame-wise cropping of video clips, we illustrate the efficiency of this method. Based on this model, we design and produce an AI automatic camera for lively photography and clip videoing.

Keywords: Computer Vision for Other Robotic Applications, Semantic Scene Understanding
Abstract: We present a novel Multi-Relational Graph Convolutional Network (MRGCN) based framework to model on-road vehicle behaviors from a sequence of temporally ordered frames as grabbed by a moving monocular camera. The input to MRGCN is a multi-relational graph where the graph's nodes represent the active and passive agents/objects in the scene, and the bidirectional edges that connect every pair of nodes are encodings of their Spatio-temporal relations. We show that this proposed explicit encoding and usage of an intermediate spatio-temporal interaction graph to be well suited for our tasks over learning end-end directly on a set of temporally ordered spatial relations. We also propose an attention mechanism for MRGCNs that conditioned on the scene dynamically scores the importance of information from different interaction types. The proposed framework achieves significant performance gain over prior methods on vehicle-behavior classification tasks on four datasets. We also show a seamless transfer of learning to multiple datasets without resorting to fine-tuning. Such behavior prediction methods find immediate relevance in a variety of navigation tasks such as behavior planning, state estimation, and applications relating to the detection of traffic violations over videos.

Keywords: Surveillance Systems, Aerial Systems: Mechanics and Control, Computer Vision for Other Robotic Applications
Abstract: Large special-events parking involves various parking scenarios, e.g., temporary parking and on-street parking. Their occupancy detection is challenging as it is unrealistic to construct gates/stations for temporary parking areas or build a sensor-based detection system to cover every single street. To address this issue, this study develops a quadrotor-enabled autonomous parking occupancy detection system. A camera-equipped quadrotor is flying over the parking lot first; then the images are captured by the on-board camera of the quadrotor and transferred to the ground station; finally, the ground station will process and release the occupancy information to the driver’s mobile devices. The decision tree learning algorithm is adopted to determine the optimal flying speed for the quadrotor to balance the trade-off between the detection efficiency and accuracy. In order to tackle the complex environment in real-life parking, a convolutional neural network (CNN)-based vehicle detection model has been trained and implemented, where the realistic factors, e.g., passing pedestrians and tree blocking, are considered. Experiments are conducted to illustrate the effectiveness of the proposed system.

Keywords: Surveillance Systems, Visual-Based Navigation, Aerial Systems: Perception and Autonomy
Abstract: Correlation filter (CF)-based methods have demonstrated exceptional performance in visual object tracking for unmanned aerial vehicle (UAV) applications, but suffer from the undesirable boundary effect. To solve this issue, spatially regularized correlation filters (SRDCF) proposes the spatial regularization to penalize filter coefficients, thereby significantly improving the tracking performance. However, the temporal information hidden in the response maps is not considered in SRDCF, which limits the discriminative power and the robustness for accurate tracking. This work proposes a novel approach with dynamic consistency pursued correlation filters, i.e., the CPCF tracker. Specifically, through a correlation operation between adjacent response maps, a practical consistency map is generated to represent the consistency level across frames. By minimizing the difference between the practical and the scheduled ideal consistency map, the consistency level is constrained to maintain temporal smoothness, and rich temporal information contained in response maps is introduced. Besides, a dynamic constraint strategy is proposed to further improve the adaptability of the proposed tracker in complex situations. Comprehensive experiments are conducted on three challenging UAV benchmarks, i.e., UAV123@10FPS, UAVDT, and DTB70. Based on the experimental results, the proposed tracker favorably surpasses the other 25 state-of-the-art trackers with real-time running speed (~43FPS) on a single CPU.

Keywords: Semantic Scene Understanding, Autonomous Vehicle Navigation, Autonomous Agents
Abstract: In this paper, we propose a novel approach for agent motion prediction in cluttered environments. One of the main challenges in predicting agent motion is accounting for location and context-specific information. Our main contribution is the concept of learning context maps to improve the prediction task. Context maps are a set of location-specific latent maps that are trained alongside the predictor. Thus, the proposed maps are capable of capturing location context beyond visual context cues (e.g. usual average speeds and typical trajectories) or predefined map primitives (such as lanes and stop lines). We pose context map learning as a multi-task training problem and describe our map model and its incorporation into a state-of-the-art trajectory predictor. In extensive experiments, it is shown that use of learned maps can significantly improve predictor accuracy. Furthermore, the performance can be additionally boosted by providing partial knowledge of map semantics.

Keywords: Soft Robot Applications, Modeling, Control, and Learning for Soft Robots, Learning from Demonstration
Abstract: Physically soft robots are promising for robotic assembly tasks as they allow stable contacts with the environment. In this study, we propose a novel learning system for soft robotic assembly strategies. We formulate this problem as a reinforcement learning task and design the reward function from human demonstrations. Our key insight is that the failed demonstrations can be used as constraints to avoid failed behaviors. To this end, we developed a teaching device with which humans can intuitively provide various demonstrations. Moreover, we leverage Physically-Consistent Gaussian Mixture Models to clearly assign Gaussian components to the successful and failed trials.We then create the reference trajectories via Gaussian Mixture Regressions, which fit the successful demonstrations while considering the failed ones. Finally, we apply a sampleefficient deep model-based reinforcement learning method to obtain robust strategies with a few interactions. To validate our method, we developed a real-robot experimental system composed of a rigid collaborative robot arm with a compliant wrist and the teaching device. Our results demonstrate that our method learned the assembly strategies with a higher success rate than when using only successful demonstrations.

Keywords: Learning from Demonstration, Motion and Path Planning, Optimization and Optimal Control
Abstract: A robot in the future may initially has a good learning capability but an empty library of movements. It gradually enriches its library of movements through human demonstrations. Dynamic Movement Primitives (DMPs) has been proved to be an effective way to represent trajectories. Trajectories are classified into discrete and rhythmic ones, and parameters are set for each demonstrated trajectory. However, what kind of trajectory will be provided by robot users is sometimes unknown to robot developers, so trajectory pattern and the parameters can not be determined in advance. It's also impossible for non-technical robot users to set these parameters and determine the pattern of movements they are going to demonstrate. To make it easier for non-expert robot users to programme their robots by demonstration, this work presents an efficient way to deal with these two problems. The effectiveness of the proposed methodology is proved by teaching a robot to clean the whiteboard in different ways and stack a set of cubic boxes in specific order.

Keywords: Learning from Demonstration, Force and Tactile Sensing, Imitation Learning
Abstract: Deploying robot learning frameworks in unconstrained environments requires robustness and tractability. We must not only equip the robot with a sufficient range of sensing capabilities, but also provide training data in a sample-efficient manner. To this end, we identify and address a need specifically in robot learning from demonstration (LfD) literature to account for not only end-effector pose and wrench signals, but also tactile signals for contact. While traditional pose and wrench signals have proven to be sufficient for robots to learn basic position and force-control behaviors, they are inherently too constraining for the learning of general manipulation tasks. In particular, useful manipulation tasks often rely on the geometry of the contact interaction. To explore the value of geometry-based tactile signals, we utilize a LfD framework built upon hidden Markov models and Gaussian mixture regression, adapt it to our robotic system equipped with a soft tactile sensor, and validate its performance with an edge-following task and a manipulation task involving different object geometries.

Keywords: Learning from Demonstration, Optimization and Optimal Control, Imitation Learning
Abstract: This paper proposes a dynamic system based learning from demonstration approach to teach a robot activities of daily living. The approach takes inspiration from human movement literature to formulate trajectory learning as an optimal control problem. We assume a weighted combination of basis objective functions is the true objective function for a demonstrated motion. We derive basis objective functions analogous to those in human movement literature to optimize the robot's motion. This method aims to naturally adapt the learned motion in different situations. To validate our approach, we learn motions from two categories: 1) commonly prescribed therapeutic exercises and 2) tea making. We show the reproduction accuracy of our method and compare torque requirements to the dynamic motion primitive for each motion, with and without an added load.

Keywords: Learning from Demonstration, Reactive and Sensor-Based Planning, Force Control
Abstract: In this paper, we aim to expedite the deployment of challenging manipulation tasks involving both motion and contact wrenches (forces and moments). To this end, we acquire motion and wrench signals from a small set of demonstrations using passive observation. To learn these tasks, we introduce Trajectory parameterized Probabilistic Principal Component Analysis (traPPCA) which compactly re-parameterizes the acquired signals using trajectory information and encodes the signal correlations using Probabilistic Principal Component Analysis (PPCA). Finally, the task is transferred to a robot setup by specifying the robot behavior using a constraint-based task specification and control approach. This framework results in increased robustness of the system against different sources of uncertainty: imprecise sensors, adaptation of the tool, and changes in the execution speed.

Keywords: Learning Categories and Concepts, Recognition, Visual Learning
Abstract: For many applications, robots will need to be incrementally trained to recognize the specific objects needed for an application. This paper presents a practical system for incrementally training a robot to recognize different object categories using only a small set of visual examples provided by a human. The paper evaluates and verifies a recently developed state-of-the-art method for few-shot incremental learning. After learning the object classes incrementally, the robot performs a table cleaning task organizing objects into categories specified by the human. We also demonstrate the system's ability to learn arrangements of objects and predict missing or incorrectly placed objects. Experimental evaluations demonstrate that our approach achieves nearly the same performance as a system trained with all examples at one time (batch training), which constitutes a theoretical upper bound.

Keywords: Representation Learning, SLAM
Abstract: Place recognition is a critical component towards addressing the key problem of Simultaneous Localization and Mapping (SLAM). Most existing methods use visual images; whereas, place recognition using 3D point clouds, especially based on the voxel representations, has not been well addressed yet. In this paper, we introduce the novel approach of voxel-based representation learning (VBRL) that uses 3D point clouds to recognize places with long-term environment variations. VBRL splits a 3D point cloud input into voxels and uses multi-modal features extracted from these voxels to perform place recognition. Additionally, VBRL uses structured sparsity-inducing norms to learn representative voxels and feature modalities that are important to match places under long-term changes. Both place recognition, and voxel and feature learning are integrated into a unified regularized optimization formulation. As the sparsity-inducing norms are non-smooth, it is hard to solve the formulated optimization problem. Thus, we design a new iterative optimization algorithm, which has a theoretical convergence guarantee. Experimental results have shown that VBRL performs place recognition well using 3D point cloud data and is capable of learning the importance of voxels and feature modalities.

Keywords: Cognitive Human-Robot Interaction, Semantic Scene Understanding
Abstract: Abstract--- Understanding spatial relations of objects is critical in many robotic applications such as grasping, manipulation, and obstacle avoidance. Humans can simply reason object's spatial relations from a glimpse of a scene based on prior knowledge of spatial constraints. The proposed method enables a robot to comprehend spatial relationships among objects from RGB-D data. This paper proposed a neural-logic learning framework to learn and reason spatial relations from raw data by following logic rules on spatial constraints. The neural-logic network consists of three blocks: grounding block, spatial logic block, and inference block. The grounding block extracts high-level features from the raw sensory data. The spatial logic blocks can predicate fundamental spatial relations by training a neural network with spatial constraints. The inference block can infer complex spatial relations based on the predicated fundamental spatial relations. Simulations and robotic experiments evaluated the performance of the proposed method.

Keywords: Learning Categories and Concepts, Visual Learning, Deep Learning for Visual Perception
Abstract: Manipulation tasks in daily life, such as pouring water, unfold through human intentions. Being able to process contextual knowledge from these Activities of Daily Living (ADLs) over time can help us understand manipulation intentions, which are essential for an intelligent robot to transition smoothly between various manipulation actions. In this paper, to model the intended concepts of manipulation, we present a vision dataset under a strictly constrained knowledge domain for both robot and human manipulations, where manipulation concepts and relations are stored by an ontology system in a taxonomic manner. Furthermore, we propose a scheme to generate a combination of visual attentions and an evolving knowledge graph filled with commonsense knowledge. Our scheme works with real-world camera streams and fuses an attention-based Vision-Language model with the ontology system. The experimental results demonstrate that the proposed scheme can successfully represent the evolution of an intended object manipulation procedure for both robots and humans. The proposed scheme allows the robot to mimic human-like intentional behaviors by watching real-time videos. We aim to develop this scheme further for real-world robot intelligence in Human-Robot Interaction.

Keywords: Learning Categories and Concepts, Cognitive Human-Robot Interaction, Representation Learning
Abstract: Understanding spatial relations is a key element for natural human-robot interaction. Especially, a robot must be able to manipulate a given scene according to a human verbal command specifying desired spatial relations between objects. To endow robots with this ability, a suitable representation of spatial relations is necessary, which should be derivable from human demonstrations. We claim that polar coordinates can capture the underlying structure of spatial relations better than Cartesian coordinates and propose a parametric probability distribution defined in polar coordinates to represent spatial relations. We consider static spatial relations such as left of, behind, and near, as well as dynamic ones such as closer to and other side of, and take into account verbal modifiers such as roughly and a lot. We show that adequate distributions can be derived for various combinations of spatial relations and modifiers in a sample-efficient way using Maximum Likelihood Estimation, evaluate the effects of modifiers on the distribution parameters, and demonstrate our representation's usefulness in a pick-and-place task on a real robot.

Keywords: Gesture, Posture and Facial Expressions, Multi-Modal Perception, Computer Vision for Other Robotic Applications
Abstract: The ability to synchronize expectations among human-robot teams and understand discrepancies between expectations and reality is essential for human-robot collaboration scenarios. To ensure this, human activities and intentions must be interpreted quickly and reliably by the robot using various modalities. In this paper we propose a multimodal recognition system designed to detect physical interactions as well as nonverbal gestures. Existing approaches feature high post-transfer recognition rates which, however, can only be achieved based on well-prepared and large datasets. Unfortunately, the acquisition and preparation of domain-specific samples especially in industrial context is time consuming and expensive. To reduce this effort we introduce a weakly-supervised classification approach. Therefore, we learn a latent representation of the human activities with a variational autoencoder network. Additional modalities and unlabeled samples are incorporated by a scalable product-of-expert sampling approach. The applicability in industrial context is evaluated by two domain-specific collaborative robot datasets. Our results demonstrate, that we can keep the number of labeled samples constant while increasing the network performance by providing additional unprocessed information.

Keywords: Model Learning for Control, AI-Based Methods, Deep Learning in Grasping and Manipulation
Abstract: When executing a certain task, human beings can choose or make an appropriate tool to achieve the task. This research especially addresses the optimization of tool shape for robotic tool-use. We propose a method in which a robot obtains an optimized tool shape, tool trajectory, or both, depending on a given task. The feature of our method is that a transition of the task state when the robot moves a certain tool along a certain trajectory is represented by a deep neural network. We applied this method to object manipulation tasks on a 2D plane, and verified that appropriate tool shapes are generated by using this novel method.

Keywords: AI-Based Methods, Learning Categories and Concepts
Abstract: While abstract knowledge like cause-and-effect relations enables robots to problem-solve in new environments, acquiring such knowledge remains out of reach for many traditional machine learning techniques. In this work, we introduce a method for a robot to learn an explicit model of cause-and-effect by constructing a structural causal model through a mix of observation and self-supervised experimentation, allowing a robot to reason from causes to effects and from effects to causes. We demonstrate our method on tool affordance learning tasks, where a humanoid robot must leverage its prior learning to utilize novel tools effectively. Our results suggest that after minimal training examples, our system can preferentially choose new tools based on the context, and can use these tools for goal-directed object manipulation.

Keywords: Cognitive Human-Robot Interaction, Multi-Modal Perception, Human-Centered Robotics
Abstract: While grounded language learning, or learning the meaning of language with respect to the physical world in which a robot operates, is a major area in human-robot interaction studies, most research occurs in closed worlds or domain-constrained settings. We present a system in which language is grounded in visual percepts without using categorical constraints by combining CNN-based visual featurization with natural language labels. We demonstrate results comparable to those achieved using handcrafted features for specific traits, a step towards moving language grounding into the space of fully open world recognition.

Keywords: Deep Learning in Grasping and Manipulation, Reinforecment Learning, RGB-D Perception
Abstract: When searching for objects in cluttered environments, it is often necessary to perform complex interactions in order to move occluding objects out of the way and fully reveal the object of interest and make it graspable. Due to the complexity of the physics involved and the lack of accurate models of the clutter, planning and controlling precise predefined interactions with accurate outcome is extremely hard, when not impossible. In problems where accurate (forward) models are lacking, Deep Reinforcement Learning (RL) has shown to be a viable solution to map observations (e.g. images) to good interactions in the form of close-loop visuomotor policies. However, Deep RL is sample inefficient and fails when applied directly to the problem of unoccluding objects based on images. In this work we present a novel Deep RL procedure that combines i) teacher-aided exploration, ii) a critic with privileged information, and iii) mid-level representations, resulting in sample efficient and effective learning for the problem of uncovering a target object occluded by a heap of unknown objects. Our experiments show that our approach trains faster and converges to more efficient uncovering solutions than baselines and ablations, and that our uncovering policies lead to an average improvement in the graspability of the target object, facilitating downstream retrieval applications.

Keywords: Deep Learning in Grasping and Manipulation
Abstract: Learning-based approaches to grasp planning are preferred over analytical methods due to their ability to better generalize to new, partially observed objects. However, data collection remains one of the biggest bottlenecks for grasp learning methods, particularly for multi-fingered hands. The relatively high dimensional configuration space of the hands coupled with the diversity of objects common in daily life requires a significant number of samples to produce robust and confident grasp success classifiers. In this paper, we present the first active deep learning approach to grasping that searches over the grasp configuration space and classifier confidence in a unified manner. We base our approach on recent success in planning multi-fingered grasps as probabilistic inference with a learned neural network likelihood function. We embed this within a multi-armed bandit formulation of sample selection. We show that our active grasp learning approach uses fewer training samples to produce grasp success rates comparable with the passive supervised learning method trained with grasping data generated by an analytical planner. We additionally show that grasps generated by the active learner have greater qualitative and quantitative diversity in shape.

Keywords: Deep Learning in Grasping and Manipulation, Multi-Modal Perception, Grasping
Abstract: Robotic grasping systems often rely on visual observations to drive the grasping process, where the robot must be able to detect and localize an object, extract features relevant to the task, and then combine this information to plan a manipulation strategy. But what happens when some of the most impactful features are not observed by the robot? Without context on an objects center-of-mass, for example, a robot may make assumptions such as uniform density that do not hold, and which may in turn guide the robot into perceiving a sub-optimal set of grasping configurations. In this work, we examine how having prior knowledge of an object's intrinsic properties influences the task of dense grasp accordance prediction. We investigate a simple, constrained grasping task where object properties heavily regulate the space of successful grasps, and further evaluate how learning is affected when generalizing across unseen weight configurations and unseen object shapes.

Keywords: Perception for Grasping and Manipulation, Big Data in Robotics and Automation, Object Detection, Segmentation and Categorization
Abstract: We propose a method to annotate segmentation masks accurately and automatically using invisible marker for object manipulation. Invisible marker is invisible under visible (regular) light conditions, but becomes visible under invisible light, such as ultraviolet (UV) light. By painting objects with the invisible marker, and by capturing images while alternately switching between regular and UV light at high speed, massive annotated datasets are created quickly and inexpensively. We show a comparison between our proposed method and manual annotations. We demonstrate semantic segmentation for deformable objects including clothes, liquids, and powders under controlled environmental light conditions. In addition, we show demonstrations of liquid pouring tasks under uncontrolled environmental light conditions in complex environments such as inside the office, house, and outdoors. Furthermore, it is possible to capture data while the camera is in motion so it becomes easier to capture large datasets, as shown in our demonstration.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception, Recognition
Abstract: Video object segmentation plays a vital role to many robotic tasks, beyond the satisfied accuracy, quickly adapt to the new scenario with very limited annotations and conduct a quick inference are also important. In this paper, we are specifically concerned with the task of fast segmenting all pixels of a target object in all frames, given the annotation mask in the first frame. Even when such annotation is available, this remains a challenging problem because of the changing appearance and shape of the object over time. In this paper, we tackle this task by formulating it as a meta-learning problem, where the base learner grasping the semantic scene understanding for a general type of objects, and the meta learner quickly adapting the appearance of the target object with a few examples. Our proposed meta-learning method uses a closed form optimizer, the so-called ``ridge regression", which has been shown to be conducive for fast and better training convergence. Moreover, we propose a mechanism, named ``block splitting", to further speed up the training process as well as to reduce the number of learning parameters. In comparison with the state-of-the art methods, our proposed framework achieves significant boost up in processing speed, while having highly comparable performance compared to the best performing methods on the widely used datasets.

Keywords: Object Detection, Segmentation and Categorization, Semantic Scene Understanding, Visual Learning
Abstract: In this paper, we propose a cascaded non-local neural network for point cloud segmentation. The proposed network aims to build the long-range dependencies of point clouds for the accurate segmentation. Specifically, we develop a novel cascaded non-local module, which consists of the neighborhood-level, superpoint-level and global-level non-local blocks. First, in the neighborhood-level block, we extract the local features of the centroid points of point clouds by assigning different weights to the neighboring points. The extracted local features of the centroid points are then used to encode the superpoint-level block with the non-local operation. Finally, the global-level block aggregates the non-local features of the superpoints for semantic segmentation in an encoder-decoder framework. Benefiting from the cascaded structure, geometric structure information of different neighborhoods with the same label can be propagated. In addition, the cascaded structure can largely reduce the computational cost of the original non-local operation on point clouds. Experiments on different indoor and outdoor datasets show that our method achieves state-of-the-art performance and effectively reduces the time consumption and memory occupation.

Keywords: Object Detection, Segmentation and Categorization, Semantic Scene Understanding, RGB-D Perception
Abstract: Identifying new, moved or missing objects is an important capability for robot tasks such as surveillance or maintaining order in homes, offices and industrial settings. However, current approaches do not distinguish between novel objects or simple scene rearrangements nor do they sufficiently deal with localization errors and sensor noise. To overcome these limitations, we combine the strengths of global and local methods for efficient detection of novel objects in 3D reconstructions of indoor environments. Global structure, determined from 3D semantic information, is exploited to establish object candidates. These are then locally verified by comparing isolated geometry to a reference reconstruction provided by the task. We evaluate our approach on a novel dataset containing different types of rooms with 31 scenes and 260 annotated objects. Experiments show that our proposed approach significantly outperforms baseline methods.

Keywords: Computer Vision for Other Robotic Applications, Semantic Scene Understanding, Intelligent Transportation Systems
Abstract: The majority of learning-based semantic segmentation methods are optimized for daytime scenarios and favorable lighting conditions. Real-world driving scenarios, however, entail adverse environmental conditions such as nighttime illumination or glare which remain a challenge for existing approaches. In this work, we propose a multimodal semantic segmentation model that can be applied during daytime and nighttime. To this end, besides RGB images, we leverage thermal images, making our network significantly more robust. We avoid the expensive annotation of nighttime images by leveraging an existing daytime RGB-dataset and propose a teacher-student training approach that transfers the dataset's knowledge to the nighttime domain. We further adopt a domain adaptation method to align the learned feature spaces across the domains and propose a novel two-stage training scheme. Furthermore, due to a lack of thermal data for autonomous driving, we present a new dataset comprising over 20,000 time-synchronized and aligned RGB-thermal image pairs. In this context, we also present a novel target-less calibration method that allows for automatic robust extrinsic and intrinsic thermal camera calibration. Among others, we use our new dataset to show state-of-the-art results for nighttime semantic segmentation.

Keywords: Object Detection, Segmentation and Categorization, Semantic Scene Understanding
Abstract: Following considerable development in 3D scanning technologies, many studies have recently been proposed with various approaches for 3D vision tasks, including some methods that utilize 2D convolutional neural networks (CNNs). However, even though 2D CNNs have achieved high performance in many 2D vision tasks, existing works have not effectively applied them onto 3D vision tasks. In particular, segmentation has not been well studied because of the difficulty of dense prediction for each point, which requires rich feature representation. In this paper, we propose a simple and efficient architecture named point projection and back-projection network (PBP-Net), which leverages 2D CNNs for the 3D point cloud segmentation. 3 modules are introduced, each of which projects 3D point cloud onto 2D planes, extracts features using a 2D CNN backbone, and back-projects features onto the original 3D point cloud. To demonstrate effective 3D feature extraction using 2D CNN, we perform various experiments including comparison to recent methods. We analyze the proposed modules through ablation studies and perform experiments on object part segmentation (ShapeNet-Part dataset) and indoor scene semantic segmentation (S3DIS dataset). The experimental results show that proposed PBP-Net achieves comparable performance to existing state-of-the-art methods.

Keywords: Object Detection, Segmentation and Categorization, Semantic Scene Understanding, Deep Learning for Visual Perception
Abstract: We present a novel end-to-end single-shot method that segments countable object instances (things) as well as background regions (stuff) into a non-overlapping panoptic segmentation at almost video frame rate. Current state-of-the-art methods are far from reaching video frame rate and mostly rely on merging instance segmentation with semantic background segmentation, making them impractical to use in many applications such as robotics. Our approach relaxes this requirement by using an object detector but is still able to re-solve inter- and intra-class overlaps to achieve a non-overlapping segmentation. On top of a shared encoder-decoder backbone, we utilize multiple branches for semantic segmentation, object detection, and instance center prediction. Finally, our panoptic head combines all outputs into a panoptic segmentation and can even handle conflicting predictions between branches as well as certain false predictions. Our network achieves 32.6% PQ on MS-COCO at 23.5 FPS, opening up panoptic segmentation to a broader field of applications.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception, Recognition
Abstract: Accurate video object segmentation methods finetune a model using the first annotated frame, and/or use additional inputs such as optical flow and complex post-processing. In contrast, we develop a fast algorithm that requires no finetuning, auxiliary inputs or post-processing, and segments a variable number of objects in a single forward-pass. We represent an object with clusters, or ''visual words'', in the embedding space, which correspond to object parts in the image space. This allows us to robustly match to the reference objects throughout the video, because although the global appearance of an object changes as it undergoes occlusions and deformations, the appearance of more local parts may stay consistent. We learn these visual words in an unsupervised manner, using meta-learning to ensure that our training objective matches our inference procedure. We achieve comparable accuracy to finetuning based methods (whilst being 1 to 2 orders of magnitude faster), and state-of-the-art in terms of speed/accuracy trade-offs on four video segmentation datasets.

Keywords: RGB-D Perception, Object Detection, Segmentation and Categorization, Semantic Scene Understanding
Abstract: This paper focuses on pose registration of different object instances from the same category. This is required in online object mapping because object instances detected at test time usually differ from the training instances. Our approach transforms instances of the same category to a normalized canonical coordinate frame and uses metric learning to train fully convolutional geometric features. The resulting model is able to generate pairs of matching points between the instances, allowing category-level registration. Evaluation on both synthetic and real-world data shows that our method provides robust features, leading to accurate alignment of instances with different shapes.

Keywords: Object Detection, Segmentation and Categorization, Surgical Robotics: Laparoscopy
Abstract: Surgical tool segmentation in endoscopic images is an important problem: it is a crucial step towards full instrument pose estimation and it is used for integration of pre- and intra-operative images into the endoscopic view. While many recent approaches based on convolutional neural networks have shown great results, a key barrier to progress lies in the acquisition of a large number of manually-annotated images which is necessary for an algorithm to generalize and work well in diverse surgical scenarios. Unlike the surgical image data itself, annotations are difficult to acquire and may be of variable quality. On the other hand, synthetic annotations can be automatically generated by using forward kinematic model of the robot and CAD models of tools by projecting them onto an image plane. Unfortunately, this model is very inaccurate and cannot be used for supervised learning of image segmentation models. Since generated annotations will not directly correspond to endoscopic images due to errors, we formulate the problem as an unpaired image-to-image translation where the goal is to learn the mapping between an input endoscopic image and a corresponding annotation using an adversarial model. Our approach allows to train image segmentation models without the need to acquire expensive annotations and can potentially exploit large unlabeled endoscopic image collection outside the annotated distributions of image/annotation data. We test our proposed method on Endovis 2017 challenge dataset and show that it is competetive with supervised segmentation methods.

Keywords: Object Detection, Segmentation and Categorization, Semantic Scene Understanding
Abstract: Truly autonomous driving without the need for human intervention can only be attained when self-driving cars fully understand their surroundings. Most of these vehicles rely on a suite of active and passive sensors. LiDAR sensors are a cornerstone in most of these hardware stacks, and leveraging them as a complement to other passive sensors such as RGB cameras is an enticing goal. Understanding the semantic class of each point in a LiDAR sweep is important, as well as knowing to which instance of that class it belongs to. To this end, we present a novel, single-stage, and real-time capable panoptic segmentation approach using a shared encoder with a semantic and instance decoder. We leverage the geometric information of the LiDAR scan to perform a distance-aware trilinear upsampling, which allows our approach to use larger output strides than using transpose convolutions leading to substantial savings in computation time. Our experimental evaluation and ablation studies for each module show that combining our geometric and semantic embeddings with our learned, variable instance thresholds, a category-specific loss, and the novel trilinear upsampling module leads to higher panoptic quality. We will release the code of our approach in our LiDAR processing library LiDAR-Bonnetal.

Keywords: Visual-Based Navigation, Computer Vision for Automation, Computer Vision for Other Robotic Applications
Abstract: Detecting small obstacles on the road is critical for autonomous driving. In this paper, we present a method to reliably detect such obstacles through a multi-modal framework of sparse LiDAR(VLP-16) and Monocular vision. LiDAR is employed to provide additional context in the form of confidence maps to monocular segmentation networks. We show significant performance gains when the context is fed as an additional input to monocular semantic segmentation frameworks. We further present a new semantic segmentation dataset to the community, comprising of over 3000 image frames with corresponding LiDAR observations. The images come with pixel-wise annotations of three classes off-road, road, and small obstacle. We stress that precise calibration between LiDAR and camera is crucial for this task and thus propose a novel Hausdorff distance based calibration refinement method over extrinsic parameters. As a first benchmark over this dataset, we report our results with 73 % instance detection up to a distance of 50 meters on challenging scenarios. Qualitatively by showcasing accurate segmentation of obstacles less than 15 cms at 50m depth and quantitatively through favourable comparisons vis a vis prior art, we vindicate the method’s efficacy. Our project and dataset is hosted at https://small-obstacle-dataset.github.io/.

Keywords: Localization, Deep Learning for Visual Perception
Abstract: We propose a method to facilitate robot navigation relative to sketched maps of human environments. Our main contribution centers around using thin plate splines for registering the robot's LIDAR observation with the hand-drawn maps. Thin plate splines are particularly effective for this task because they are able to handle many of the non-rigid deformations commonly seen in sketches of maps, which render traditional rigid transformations inappropriate. Our proposed approach uses a convolutional neural network to efficiently predict the control points which define the spline transform, from which we then compute the pose of the robot on the hand drawn map for navigation purposes. Our systematic evaluations in simulation using a synthetic dataset and real, hand-drawn sketches show that the proposed spline-based registration approach outperforms baseline methods.

Keywords: Localization, Visual Learning, Service Robots
Abstract: In this paper, we propose a dependable visual kidnap recovery (KR) framework that pinpoints a unique pose in a given 3D map when a device is turned on. For this framework, we first develop indoor-GeM (i-GeM), which is an extension of GeM but considerably more robust than other global descriptors, including GeM itself. Then, we propose a convolutional neural network (CNN)-based system called KR-Net, which is based on a coarse-to-fine paradigm as in Inloc and HFNet. To our knowledge, KR-Net is the first network that can pinpoint a wake-up pose with a confidence level near 100% within a 1.0m translational error boundary. This dependable success rate is enabled not only by i-GeM, but also by a combinatorial pooling approach that uses multiple images around the wake-up spot, whereas previous implementations were constrained to a single image. Experiments were conducted in two challenging datasets: a large-scale (12,557m^2) area with frequent featureless or repetitive places and a place with significant view changes due to a one-year gap between prior modeling and query acquisition. Given 59 test query sets (eight images per pose), KR-Net successfully found all wakeup poses, with average and maximum errors of 0.246m and 0.983m, respectively.

Keywords: Localization, Range Sensing, Sensor Fusion
Abstract: In this paper, we study the localization problem under non-Gaussian noise. In particular, we consider systems that can be represented by a state transition and a measurement component. The state transition indicates how the system evolves given a control variable. The measurement component compares, for a given state, the received and predicted measurements. Here we consider a radio based range sensor which is the primary source of non-Gaussian noise in the system. We solve the problem using a MHE (Maximum Horizon Estimator) with a correntropy similarity metric. The MHE seeks the best set of states given a window of time, that explains the system and the received measurements. Moreover, the main advantage of a MHE is that it allows the re-estimation of past states. Additionally, the correntropy is a similarity metric that, given the amount of error in the estimation, behaves as L2, L1 or L0 norms and has been successfully used in many applications under non-Gaussian noise. We evaluate our proposed method using both simulated and real data. The results show that correntropy is able to work well in comparison with other methods in presence of impulsive noise.

Keywords: Localization, Semantic Scene Understanding, Range Sensing
Abstract: Semantics can be leveraged in ego-vehicle localization to improve robustness and accuracy because objects with the same labels can be correctly matched with each other. Object recognition has significantly improved owing to advances in machine learning algorithms. However, perfect object recognition is still challenging in real environments. Hence, the uncertainty of object recognition must be considered in localization. This paper proposes a novel localization method that integrates a supervised object recognition method, which predicts probabilistic distributions over object classes for individual sensor measurements, into the Bayesian network for localization. The proposed method uses the estimated probabilities and Dirichlet distribution to calculate the likelihood for estimating an ego-vehicle pose. Consequently, the uncertainty can be handled in localization. We present an implementation example of the proposed method using a particle filter and deep-neural-network-based point cloud semantic segmentation and evaluate it by simulation and the SemanticKITTI dataset. Experimental results show that the proposed method can accurately generate likelihood distribution even when object recognition accuracy is degraded, and its estimation accuracy is the highest compared to that of two conventional methods.

Keywords: Localization, Visual-Based Navigation, Recognition
Abstract: Recent years have seen the emergence of very effective ConvNet-based object detectors that have reconfigured the computer vision landscape. As a consequence, new approaches that propose object-based reasoning to solve traditional problems, such as camera pose estimation, have appeared. In particular, these methods have shown that modelling 3D objects by ellipsoids and 2D detections by ellipses offers a convenient manner to link 2D and 3D data. Following that promising direction, we propose here a novel object-based pose estimation algorithm that does not require any sensor but a RGB camera. Our method operates from at least two bject detections, and is based on a new paradigm that enables to decrease the Degrees of Freedom (DoF) of the pose stimation problem from six to three, while two simplifying yet realistic assumptions reduce the remaining DoF to only one. Exhaustive search is performed over the unique unknown parameter to recover the full camera pose. Robust algorithms designed to deal with any number of objects as well as a refinement step are introduced. Effectiveness of the method has been assessed on the challenging T-LESS and Freiburg datasets.

Keywords: Localization
Abstract: Visual place recognition is challenging because there are so many factors that can cause the appearance of a place to change, from day-night cycles to seasonal change to atmospheric conditions. In recent years a large range of approaches have been developed to address this challenge including deep-learnt image descriptors, domain translation, and sequential filtering, all with shortcomings including generality and velocity-sensitivity. In this paper we propose a novel descriptor derived from tracking changes in any learned global descriptor over time, dubbed Delta Descriptors. Delta Descriptors mitigate the offsets induced in the original descriptor matching space in an unsupervised manner by considering temporal differences across places observed along a route. Like all other approaches, Delta Descriptors have a shortcoming - volatility on a frame to frame basis - which can be overcome by combining them with sequential filtering methods. Using two benchmark datasets, we first demonstrate the high performance of Delta Descriptors in isolation, before showing new state-of-the-art performance when combined with sequence-based matching. We also present results demonstrating the approach working with four different underlying descriptor types, and two other beneficial properties of Delta Descriptors in comparison to existing techniques: their increased inherent robustness to variations in camera motion and a reduced rate of performance degradation as dimensional reduction is applied. Source code is made available at https://github.com/oravus/DeltaDescriptors.

Keywords: Localization, SLAM, Deep Learning for Visual Perception
Abstract: With its essential role in achieving full autonomy of robot navigation, place recognition has been widely studied with various approaches. Recently, numerous point cloud-based methods with deep learning implementation have been proposed with promising results for their application in outdoor environments. However, their performances are not as promising in indoor spaces because of the high level of occlusion caused by structures and moving objects. In this paper, we propose a point cloud-based place recognition method for crowded indoor spaces. The method consists of voxelizing point clouds in spherical coordinates and defining the occupancy of each voxel in ternary values. We also present SpoxelNet, a neural network architecture that encodes input voxels into global descriptor vectors by extracting the structural features in both fine and coarse scales. It also reinforces its performance in occluded places by concatenating feature vectors from multiple directions. Our method is evaluated in various indoor datasets and outperforms existing methods with a large margin.

Keywords: Localization, Autonomous Vehicle Navigation, SLAM
Abstract: Many indoor spaces have constantly changing layouts and may not be mapped by an autonomous vehicle, yet maps such as floor plans or evacuation maps of these places are common. We propose a method for an autonomous robot to localize itself on such maps with inconsistent scale using Stochastic Gradient Descent (SGD) with scan matching using a 2D LiDAR. We also introduce a new scale state in 2D localization to manage the possible inconsistent scale of the input map. Experiments are conducted in an indoor corridor using three different input maps - a point cloud, a floor plan, and a hand-drawn map. The SGD localization algorithm is bench-marked to Adaptive Monte Carlo Localization (AMCL). In a point cloud mapped environment, our algorithm achieves 0.264m and 5.26 degrees average position and heading error respectively. On the hand-drawn map, our SGD localization algorithm remains robust while AMCL fails. The role of the scale state in our SGD localization algorithm is demonstrated in poorly scaled maps.

Keywords: Localization, Mapping, Visual-Based Navigation
Abstract: Focal-plane Sensor-processor (FPSP) is a next-generation camera technology which enables every pixel on the sensor chip to perform computation in parallel, on the focal plane where the light intensity is captured. SCAMP-5 is a general-purpose FPSP used in this work and it carries out computations in the analog domain before analog to digital conversion. By extracting features from the image on the focal plane, data which is digitised and transferred is reduced. As a consequence, SCAMP-5 offers a high frame rate while maintaining low energy consumption. Here, we present BIT-VO, which is the first 6-Degrees of Freedom visual odometry algorithm which utilises the FPSP. Our entire system operates at 300 FPS in a natural environment, using binary edges and corner features detected by the SCAMP-5.

Keywords: Localization
Abstract: In many applications of mobile robot, the environment is constantly changing. How to use historical information to analysis environmental changes and generate a map corresponding with current environment is important to achieve high-precision localization. Inspired by predictive mechanism of brain, this paper presents a long-term localization approach named ArmMPU (ARMA-based Map Prediction and Update) based on time series modeling and prediction. Autoregressive moving average model (ARMA), a kind of time series modeling method, is employed for environmental map modeling and prediction, then predicted map and filtered observation are fused to fix the prediction error. The simulation and experiment results show that the proposed method improves long-term localization performance in dynamic environments.

Keywords: Localization, Intelligent Transportation Systems
Abstract: Maps provide an important source of information for autonomous vehicles. They can be used along with cameras and lidars to localize the vehicle. This requires the ability to correctly associate observations to features referenced in the map. The problem is all the more difficult than all observations are not necessarily referenced, and all map features might not be detectable with the embedded sensors.

This paper presents an adjustment technique than enables to increase the number of associations that can be made while limiting the chance of obtaining wrong associations. This is achieved by matching observations in batches in a buffer and matching them regularly. Periodically, the observation buffer is used to adjust the trajectory used to match observations. This is done without making any assumption on the association between observations and map features through a likelihood maximization process. The adjusted trajectory then provides the best associations that are used for real-time localization.

The method was tested with data recorded on public roads using an experimental vehicle. The results show that thanks to the trajectory adjustment step and the use of an observation buffer, the number of associations that can be made is increased. This also results in greater localization accuracy and consistency with an average error of 0.7 meters at 50 Hz using road markings and traffic signs.

Keywords: Soft Robot Materials and Design, Soft Sensors and Actuators, Modeling, Control, and Learning for Soft Robots
Abstract: The field of soft robotics has evolved as a domain for developing light, compliant and safe actuators. However, one of the challenges in the field is the lack of repeatable fabrication techniques as well as customizability that restricts the application of soft robots. We present a fabrication technique using sacrificial molding to fabricate pneumatic channels that are repeatable and less prone to variability. This technique enables the monolithic fabrication of actuators which eliminates conventional failure modes such as delamination. We then use embedded endostructures manufactured using Fused Deposition Modelling (FDM) 3D printers to customize the behavior of bending actuator by altering local mechanical characteristics. Finite element analysis (FEA) was used as a tool to tune the choice of materials and the geometry of the 3D printed layers based on the required application. We analyze the effect of the mechanical properties of the endostructures on actuator behavior and its utility in improving customizability. We analyzed the behavior of actuators with a variety of endostructures using visual markers. As predicted by the FEA and Euler-Bernoulli beam theory, the behavior of the actuators was seen to be influenced by the mechanical properties of the endostucture. Thus, we present a new methodology for tuning the mechanical properties of Soft Pneumatic Actuators (SPAs), which is simple and efficient to predict as well as easy to execute.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications, Soft Sensors and Actuators
Abstract: Super-coiled polymer (SCP) artificial muscles have many interesting properties that show potentials for making high performance bionic devices. To realize human-like robotic devices from this type of actuator, it is important for the SCP driven mechanisms to achieve human-like performance, such as compliant behaviors through antagonistic mechanisms. This paper presents the simultaneous position-stiffness control of an antagonistic joint driven by hybrid twisted-coiled polymer actuation bundles made from Spandex and nylon fibers, which is a common human compliant behavior. Based on a linear model of the system, which is identified and verified experimentally, a controller based on model predictive control (MPC) is designed. The MPC performance is enhanced by the incorporation of time delay estimation to estimate model variations and external disturbances. The controlled system is verified through simulations and experiments. The results show the controller's ability to control the joint angle with the highest position error of 0.6 degrees while changing joint stiffness, verified with both step command and sinusoidal reference with composite frequencies of 0.01Hz to 0.1Hz.

Keywords: Modeling, Control, and Learning for Soft Robots, Telerobotics and Teleoperation, Contact Modeling
Abstract: Gastric cancer is the third leading cause of cancer deaths worldwide, with most new cases occurring in low and middle income countries, where access to screening programs is hindered by the high cost of conventional endoscopy. The waterjet-actuated HydroJet endoscopic platform was developed as a low-cost, disposable alternative for inspection of the gastric cavity in low-resource settings. In this work, we present a teleoperation scheme and contact detection algorithm that work together to enable intuitive teleoperation of the HydroJet within the confined space of the stomach. Using a geometrically accurate stomach model and realistic anatomical inspection targets, we demonstrate that, using these methods, a novice user can complete a gastroscopy in approximately the same amount of time with the HydroJet as with a conventional endoscope.

Keywords: Modeling, Control, and Learning for Soft Robots, Dynamics, Soft Robot Applications
Abstract: Unlike traditional robots, soft robots can intrinsically interact with their environment in a continuous, robust, and safe manner. These abilities - and the new opportunities they open - motivate the development of algorithms that provide reliable information on the nature of environmental interactions and, thereby, enable soft robots to reason on and properly react to external contact events. However, directly extracting such information with integrated sensors remains an arduous task that is further complicated by also needing to sense the soft robot's configuration. As an alternative to direct sensing, this paper addresses the challenge of estimating contact forces directly from the robot's posture. We propose a new technique that merges a nominal disturbance observer, a model-based component, with corrections learned from data. The result is an algorithm that is accurate yet sample efficient, and one that can reliably estimate external contact events with the environment. We prove the convergence of our proposed method analytically, and we demonstrate its performance with simulations and physical experiments.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications, Biologically-Inspired Robots
Abstract: Soft materials with embedded magnetic properties can be actuated in a contactless manner for dexterous motion in restricted and unstructured environments. Magnetic soft robots have been demonstrated to be capable of versatile and programmable untethered motion. However, magnetic soft robots reported in literature are typically actuated by utilizing magnetic fields to generate torques that produce deformation. By contrast, this work investigates the utilization of field gradients to produce tethering forces for anchoring soft robots to the working surface, in conjunction with the use of magnetic fields to generate torques for deformation. The methodology applied here uses a six-coil electromagnetic system for field generation. The approach to achieve the magnetic field and gradients desired for soft robot motion is described, along with the restrictions imposed by Maxwell's equations. The design and fabrication of the soft robots is explained together with calculations to assess the capabilities of the actuation system. Proof-of-concept demonstrations of soft robot motion show Hexapede robots with the ability to `walk' untethered on the ceiling of the workspace, working against gravity; and lightweight Worm robots made of thin strips of material are demonstrated to locomote while staying in contact with the ground.

Keywords: Modeling, Control, and Learning for Soft Robots, Calibration and Identification, Flexible Robots
Abstract: Soft links and actuators are nowadays emerging technologies aiming to overcome some problems in robotics such as weight, cost or human interaction. However, the nonlinear nature of their elements can make their characterization challenging and hinder the use of standard control engineering tools. In this paper, we explore different state-of-the-art identification methods for the soft neck, in order to find a reliable plant model. Even though the neck has three Degrees of freedom, in this work we only consider the planar deflection of the link as a starting point for future analysis. Given the nonlinear nature of the soft neck, we consider two identification strategies, i.e., set membership, which is a data driven, nonlinear and nonparametric identification strategy, and Recursive Least Squares at selected linearization points. A neural network identification is also given for comparison purposes. Results show that the explored methods offer a suitable alternative to identify the dynamics of the neck that allows their implementation for simulation and future control.

Keywords: Modeling, Control, and Learning for Soft Robots, Motion Control, Soft Robot Applications
Abstract: In the last decade, soft robots have been at the forefront of a robotic revolution. Due to the flexibility of the soft materials employed, soft robots are equipped with a capability to execute new tasks in new application areas - beyond what can be achieved using classical rigid-link robots. Despite these promising properties, many soft robots nowadays lack the capability to exert sufficient force to perform various real-life tasks. This has led to the development of stiffness-controllable inflatable robots instilled with the ability to modify their stiffness during motion. This new capability, however, poses an even greater challenge for robot control. In this paper, we propose a model-based kinematic control strategy to guide the tip of an inflatable robot arm in its environment. The bending of the robot is modelled using an Euler-Bernoulli beam theory which takes into account the variation of the robot's structural stiffness. The parameters of the model are estimated online using an observer based on the Extended Kalman Filter (EKF). The parameters' estimates are used to approximate the Jacobian matrix online and used to control the robot's tip considering also variations in the robot's stiffness. Simulation results and experiments using a fabric-based planar 3-degree-of-freedom (DOF) inflatable manipulators demonstrate the promising performance of the proposed control algorithm.

Keywords: Flexible Robots, Kinematics, Modeling, Control, and Learning for Soft Robots
Abstract: Choosing a kinematic model for a continuum robot typically involves making a tradeoff between accuracy and computational complexity. One common modeling approach is to use the Cosserat rod equations, which have been shown to be accurate for many types of continuum robots. This approach, however, still presents significant computational cost, particularly when many Cosserat rods are coupled via kinematic constraints. In this work, we propose a numerical method that combines orthogonal collocation on the local rod curvature and forward integration of the Cosserat rod kinematic equations via the Magnus expansion, allowing the equilibrium shape to be written as a product of matrix exponentials. We provide a bound on the maximum step size to guarantee convergence of the Magnus expansion for the case of Cosserat rods, compare in simulation against other approaches, and demonstrate the tradeoffs between speed and accuracy for the fourth and sixth order Magnus expansions as well as for different numbers of collocation points. Our results show that the proposed method can find accurate solutions to the Cosserat rod equations and can potentially be competitive in computation speed.

Keywords: Soft Sensors and Actuators, Soft Robot Materials and Design, Surgical Robotics: Steerable Catheters/Needles
Abstract: Soft actuators have been widely studied in recent years because of their ability to adapt to diverse environments and safely interact with humans. Their softness broadens their potential range of medical applications since they can provide inherent safety. Among the various motions a soft robot can perform, “torsion” can maximize the efficiency of motion in confined spaces like the human abdominal cavity. This paper presents a fully soft actuator with a double-helix tendon routing path for large-angle torsional motions. The double-helix tendon routing enables the actuator to generate large twisting deformations, while also avoiding buckling generally associated with the torque imbalance in small diameter soft cylinder structures. A sequential casting method was developed for cylindrical structures with internal double-helix pathing. A parametric study of the actuator’s twisting angle and the axial contraction with respect to different design parameters was conducted, including the wire tension and path pitch. From the results, when the tendon was pulled with 40 N after the pitch was decreased, the axial contraction of the soft actuator was reduced by half and the torsional angle was doubled up to 600 degrees without buckling.

Keywords: Soft Robot Materials and Design, Biologically-Inspired Robots, Flexible Robots
Abstract: This paper presents the design of a new soft pneumatic actuator whose direction and magnitude of bending may be precisely controlled via activation of different shape memory alloy (SMA) springs within the actuator, in conjunction with pneumatic actuation. This design is inspired by examples seen in nature such as the human tongue, where the combination of hydrostatic pressure and contraction of intrinsic muscle groups enables precise maneuverability and morphing capabilities. Here, SMA springs are embedded in the walls of the actuator, serving as intrinsic muscles that may be selectively activated to constrain the device. The pneumatic SMA (PneuSMA) actuator demonstrates remarkable spatial controllability evidenced by testing under different pressures and SMA activation combinations. A baseline finite element model is also developed to predict the actuator deformation under different pressure and activation conditions.

Keywords: Soft Sensors and Actuators, Soft Robot Materials and Design, Soft Robot Applications
Abstract: Soft pneumatic actuators are commonly used in robotics for creating single-axis compression, extension, or bending motions. If these actuators are composed of compliant materials, they can also have low off-axis stiffnesses, making it difficult to restrict off-axis motions. In this work, we exploit the low off-axis stiffnesses of pneumatic actuators to design a modular actuator system that is capable of multi-modal extension, compression, two-axis bending, and twisting motions. By combining physical constraint mechanisms and motion planning, we demonstrate closed loop control with up to 24 mm of compression, 70 mm of extension, 115 degrees of bending, and 240 degrees of twisting. This actuator system is then used to illustrate several unique applications including twisting for unscrewing bottle caps and peristaltic crawling for locomotion.

Keywords: Soft Robot Applications, Force Control
Abstract: We propose a wireless soft actuator controlled thermally by millimeter-wave irradiation. The actuator is composed of low boiling point liquid sealed in a soft bellows. By irradiating high-power millimeter-waves, the liquid can be evaporated to generate a strong mechanical force. We characterize the force and work extracted from the bellows as a function of the liquid volume and temperature. We then demonstrate the wireless actuation by irradiating a millimeter-wave on the bellows. We also evaluate its dynamic response by modulating the millimeter-wave. Our approach provides novel usage and design space of soft actuators.

Keywords: Soft Sensors and Actuators, Medical Robots and Systems, Micro/Nano Robots
Abstract: Microfluidic devices and micro-pumps are increasingly necessitated in many fields ranging from untethered soft robots, to pharmaceutical and biomedical technology. While realization of such devices is limited by miniaturization constraints of conventional actuators, these restrictions can be resolved by using smart material transducers instead. This paper proposes and investigates the first ionic polymer metal composite (IPMC) actuator-driven linear peristaltic pump. With the aim of designing a monolithic device, our concept is based on a single IPMC actuator that is etched on both sides and cut with kirigami-inspired slits by laser ablation. Our pump has a planar configuration, operates with low activation voltages (< 5 V) and is simple to manufacture and thus miniaturize. We build proof-of-principle prototypes of an open and closed design of our proposed pump concept, model the closed design, and evaluate both configurations experimentally. Results show the feasibility of the proposed IPMC-driven pump. Without any optimization, the open pump achieved pumping rates of 669 pL · s−1, while the closed pump configuration attained a 4.57 Pa pressure buildup and 9.18 nL · s−1 pumping rate. These results indicate feasibility of the concept and future work will focus on design optimization.

Keywords: Soft Robot Materials and Design, Soft Sensors and Actuators, Soft Robot Applications
Abstract: This paper presents the design of the Multi-material Actuator for Variable Stiffness (MAVS), which consists of an inflatable soft fabric actuator fixed between two layers of rigid retainer pieces. The MAVS is designed to be integrated with a soft robotic ankle-foot orthosis (SR-AFO) exosuit to aid in supporting the human ankle in the inversion/eversion directions. This design aims to assist individuals affected with chronic ankle instability (CAI) or other impairments to the ankle joint. The MAVS design is made from compliant fabric materials, layered and constrained by thin rigid retainers to prevent volume increase during actuation. The design was optimized to provide the greatest stiffness and least deflection for a beam positioned as a cantilever with a point load. Geometric programming of materials was used to maximize stiffness when inflated and minimize stiffness when passive. An analytic model of the MAVS was created to evaluate the effects in stiffness observed by varying the ratio in length between the rigid pieces and the soft actuator. A finite element analysis (FEA) was generated to analyze and predict the behavior of the MAVS prior to fabrication. The results from the analytic model and FEA study were compared to experimentally obtained results of the MAVS. The MAVS with the greatest stiffness was observed when the gap between the rigid retainers was smallest and the rigid retainer length was smallest. The MAVS design with the highest stiffness at 100 kPa was determined, which required 26.71 +/- 0.06 N to deflect the actuator 20 mm, and a resulting stiffness of 1,335.5 N/m and 9.1% margin of error from the model predictions.

Keywords: Soft Robot Applications, Soft Robot Materials and Design, Soft Sensors and Actuators
Abstract: ctuation means for soft robotic structures are manifold: despite actuation mechanisms such as tendon-driven manipulators or shape memory alloys, the majority of soft robotic actuators are fluidically actuated - either purely by positive or negative air pressure or by hydraulic actuation only. This paper presents the novel idea of employing hybrid fluidic - hydraulic and pneumatic - actuation for soft robotic systems. The concept and design of the hybrid actuation system as well as the fabrication of the soft actuator are presented: Polyvinyl Alcohol (PVA) foam is embedded inside a casted, reinforced silicone chamber. A hydraulic and pneumatic robotic syringe pump are connected to the base and top of the soft actuator. We found that a higher percentage of hydraulics resulted in a higher output force. Hydraulic actuation further is able to change displacements at a higher rate compared to pneumatic actuation. Changing between Hydraulic:Pneumatic (HP) ratios shows how stiffness properties of a soft actuator can be varied.

Keywords: Soft Robot Materials and Design
Abstract: Although soft robots are a good alternative to rigid, traditional robots due to their intrinsic compliance and environmental adaptability, there are several drawbacks that limit their impact, such as low force exertion capability and low resistance to deformation. For this reason, soft structures of variable stiffness have become a popular solution in the field to combine the benefits of both soft and rigid designs. In this paper, we develop laminar jamming flexure joints that facilitate the development of adaptive robot grippers with variable stiffness. Initially, we propose a mathematical model of the laminar jamming structures. Then, the model is experimentally validated through bending tests using different materials, pressures, and number of layers. Finally, the soft, laminar jamming structured are employed to develop variable stiffness flexure joints for two different adaptive robot grippers. Bending profile analysis and grasping tests have demonstrated the benefits of the proposed jamming structures and the capabilities of the designed grippers.

Keywords: Soft Robot Materials and Design, Grippers and Other End-Effectors, Underactuated Robots
Abstract: This paper presents a novel approach for developing robotic grippers with variable stiffness hinges for dexterous grasps. This approach for the first time uses pneumatically actuated pouch actuators to fold and unfold morphable flaps of flexure hinges thus change stiffness of the hinge. By varying the air pressure in pouch actuators, the flexure hinge morphs into a beam with various open sections while the flaps bend, enabling stiffness variation of the flexure hinge. This design allows 3D printing of the flexure hinge using printable soft filaments. Utilizing the variable stiffness flexure hinges as the joints of robotic fingers, a light-weight and low-cost two-fingered tendon driven robotic gripper is developed. The stiffness variation caused due to the shape morphing of flexure hinges is studied by conducting static tests on fabricated hinges with different flap angles and on a flexure hinge with flaps that are erected by pouch actuators subjected to various pressures. Multiple grasp modes of the two-fingered gripper are demonstrated by grasping objects with various geometric shapes. The gripper is then integrated with a robot manipulator in a teleoperation setup for conducting a pick-and-place operation in a confined environment.

Keywords: Multifingered Hands, Soft Robot Materials and Design, Humanoid Robot Systems
Abstract: We present a novel underactued humanoid five finger soft hand, the KIT softhand, which is equipped with cameras inside the fingertips and integrates a high performance embedded system for visual processing and control. We describe the actuation mechanism of the hand and the tendon-driven soft finger design with internally routed high-bandwidth flat-flex cables. For efficient on-board parallel processing of visual data from multiple fingertip cameras, we present a hybrid embedded architecture consisting of a field programmable logic array (FPGA) and a microcontroller that allows the realization of visual object segmentation based on convolutional neural networks.

We evaluate the hand design by conducting durability experiments with one finger and quantify the grasp performance in terms of grasping force, speed and grasp success. The results show that the hand exhibits a grasp force of 31.8 N and a mechanical durability of the finger of more than 15.000 closing cycles. Finally, we evaluate the accuracy of visual object segmentation during the different phases of the grasping process using five different objects. Hereby, an accuracy above 90 % can be achieved.

Keywords: Multifingered Hands, Mechanism Design, Natural Machine Motion
Abstract: Robotic hand engineers usually focus on finger capabilities, often disregarding the palm contribution. Inspired by human anatomy, this paper explores the advantages of including a flexible concave palm into the design of a robotic hand actuated by soft synergies. We analyse how the inclusion of an articulated palm improves finger workspace and manipulability. We propose a mechanical design of a modular palm with two elastic rolling-contact palmar joints, that can be integrated on the Pisa/IIT SoftHand, without introducing additional motors. With this prototype, we evaluate experimentally the grasping capabilities of a robotic palm. We compare its performance to that of the same robotic hand with the palm fixed, and to that of a human hand. To assess the effective grasp quality achieved by the three systems, we measure the contact area using painttransfer patterns in different grasping actions. Preliminary grasping experiments show a closer resemblance of the softpalm robotic hand to the human hand. Results evidence a higher adaptive capability and a larger involvement of all fingers in grasping.

Keywords: Grippers and Other End-Effectors, Biologically-Inspired Robots, Dexterous Manipulation
Abstract: We propose a soft robotic gripper that can handle various types of fabrics with delicacy for applications in the field of garment manufacturing. The design was inspired by the adhesion mechanism of a parasitic fish called 'lamprey.' The proposed gripper not only is able to pick up and hold a single sheet of fabric from a stack but also does not make any damages on it. In this work, we first modeled the holding force of the gripper and then experimentally evaluated its performance with different types of fabrics, in terms of the holding force and the response time. The experimental data showed a reasonable agreement with the predicted values by the model. The actuation time and the maximum holding force measured in the experiments were 0.32 seconds and 1.12 N, respectively. The gripper showed high success rates in picking up a single sheet of air permeable fabric, which was not possible by a commercial vacuum pad. It also showed durability in repeated motions of gripping test over 20,000 cycles. We believe the proposed gripper has a high potential in realizing smart manufacturing in garment industry.

Keywords: Grippers and Other End-Effectors, Soft Robot Materials and Design, Biologically-Inspired Robots
Abstract: Compared to traditional grippers, soft grippers can typically grasp a wider range of objects, including ones that are soft, fragile, or irregularly shaped, but at the cost of a relatively low gripping force. To increase gripping force for soft grippers, this research presents a gripper with an integrated electrostatic and gecko-inspired adhesive. Synthetic gecko-inspired, microstructured adhesives are controllable (i.e. they can be turned on and off) and work on a wide range of substrates and materials; however, they are not typically effective on rough surfaces. In contrast, electrostatic adhesives, also controllable, have a higher tolerance to rough surfaces. By combining the two, it is possible to create an adhesive that is effective on a wider range of materials and roughness, including fabric. To increase the gripping force, parameters that affect electrostatic adhesion, including the electrode gap, electrode width, relative permittivity of gecko-inspired layer, and air gap between the adhesive and substrate were studied with Comsol Multiphysics software and experimentally validated. Results show that adding the two adhesives improves the gripping capabilities across acrylic, Tyvek fabric, and Kapton hemispheres of different diameters on an average of 100, 39, and 168%, respectively.

Keywords: Factory Automation, Soft Robot Applications, Soft Robot Materials and Design
Abstract: Physical softness has been proposed to absorb impacts when establishing contact with a robot or its workpiece, to relax control requirements and improve performance in assembly and insertion tasks. Previous work has focused on special end effector solutions for isolated tasks, such as the peg-in-hole task. However, as many robot tasks require the precision of rigid robots, and their performance would degrade when simply adding compliance, it has been difficult to take advantage of physical softness in real applications. A wrist that could switch between soft and rigid modes could solve this problem, but actuators with sufficient strength for this state transition would increase the size and weight of the module and decrease the payload of the robot. To solve this problem, we propose a novel design of a soft module consisting of a cable-driven mechanism, which allows the robot end effector to change between soft and rigid mode while being very compact and light. The module effectively combines the advantages of soft and rigid robots, and can be retrofitted to existing robots and grippers while preserving the characteristics of the robotic system. We evaluate the effectiveness of our proposed design through experiments modeling assembly tasks, and investigate design parameters quantitatively.

Keywords: Soft Robot Applications, Biologically-Inspired Robots, Motion and Path Planning
Abstract: Soft robots are capable of inherently safer interactions with their environment than rigid robots since they can mechanically deform in response to unanticipated stimuli. However, their complex mechanics can make planning and control difficult, particularly with tasks such as locomotion. In this work, we present a mobile and untethered underwater crawling soft robot, PATRICK, paired with a testbed that demonstrates closed-loop locomotion planning. PATRICK is inspired by the brittle star, with five flexible legs actuated by a total of 20 shape-memory alloy (SMA) wires, providing a rich variety of possible motions via its large input space. We propose a motion planning infrastructure based on a simple set of PATRICK's motion primitives, and provide experiments showing that the planner can command the robot to locomote to a goal state. These experiments contribute the first examples of closed-loop, state-space goal seeking of an underwater, untethered, soft crawling robot, and make progress towards full autonomy of soft mobile robotic systems.

Keywords: Soft Robot Materials and Design, Biologically-Inspired Robots, Marine Robotics
Abstract: The exploration of spatially limited terrestrial or aquatic environments requires miniature and lightweight robots. Soft-bodied robot research is paving ways for a new class of small-scale robots that can navigate a variety of environments with minimum influence on the environment itself. However, it is generally challenging to design miniature soft-bodied robots that efficiently adapt to the change between viscous environments. A small-scale soft-bodied robot, which could slowly move on dry land, will need rapid motions to be able to swim in a wet environment. Although using snap-through buckling of a deformable body could help to create swift motions of the robot, merely applying the snap-through buckling does not improve the swimming speed of the robot so much. Here we propose a design of a stringy soft-bodied robot that can crawl on dry surfaces and swim in liquid environments. Besides taking advantage of the snap-through buckling using coil shape memory alloys (SMAs), we design the body of the robot with a geometrical overlapping of the active body segments and control the frequency of the undulation movement, which is crucial for the swimming locomotion. We evaluate the performance of the robot in different density and viscosity liquids such as cooking oil and Glycerin solution. We found that the robot needs to drastically change its undulation from low to high frequency when it moves from high to low viscosity environments. Our robot can swim at a speed of 3.37 body-lengths per minute (BL/min}) and crawl at a speed of 1.74 BL/min. We anticipate our findings will help shed light on the design of soft-bodied robots that adapt to the changing environments efficiently.

Keywords: Soft Robot Materials and Design, Soft Robot Applications, Service Robots
Abstract: In this study, a new type of sling for assisting bedridden patients is developed using a pneumatic growing mechanism. Growing Sling focuses on minimizing the labor input of the caregivers by automating the sling insertion and retraction process while maintaining safety and comfort. Improvements over the typical growing mechanism were made by reinforcing the sling with shafts and filament tape for restricting the height of the sling to ensure its design purpose. Analysis of forces exerted on the structure was made to interpret the driving power of the automated insertion process and to ensure the structural integrity of components. Experiments on materials and prototype devices were conducted to determine the quantitative load that the sling needs to endure and what type of material is suitable for fabrication. Further, we propose a fabrication process for the Growing Sling, including its dimensions, and validate the performance of the fabricated prototype.

Keywords: Soft Robot Materials and Design, Soft Robot Applications
Abstract: Pneumatically operated soft growing robots that extend via tip eversion are well-suited for navigation in confined spaces. Adding the ability to interact with the environment using sensors and tools attached to the robot tip would greatly enhance the usefulness of these robots for exploration in the field. However, because the material at the tip of the robot body continually changes as the robot grows and retracts, it is challenging to keep sensors and tools attached to the robot tip during actuation and environment interaction. In this paper, we analyze previous designs for mounting to the tip of soft growing robots, and we present a novel device that successfully remains attached to the robot tip while providing a mounting point for sensors and tools. Our tip mount incorporates and builds on our previous work on a device to retract the robot without undesired buckling of its body. Using our tip mount, we demonstrate two new soft growing robot capabilities: (1) pulling on the environment while retracting, and (2) retrieving and delivering objects. Finally, we discuss the limitations of our design and opportunities for improvement in future soft growing robot tip mounts.

Keywords: Soft Robot Materials and Design, Soft Sensors and Actuators, Soft Robot Applications
Abstract: The widespread use of fluidic actuation for soft robots creates a high demand for soft pumps and compressors. However, current off-the-shelf pumps are usually rigid, noisy, and cumbersome. As a result, it is hard to integrate most commercial pumps into soft robotic systems, which restricts the autonomy and portability of soft robots. This paper presents the novel design of a soft pump based on bellow structure and super-coiled polymer (SCP) artificial muscles. The pump is flexible, lightweight, modular, scalable, quiet, and low cost. The pumping mechanism and fabrication process of the proposed soft pump is demonstrated. A pump prototype is fabricated to verify the proposed design and characterize its performance. From the characterization results, the pump can reach an output flow rate of up to 54 ml/min and delivers pressure up to 2.63 kPa. The pump has potential applications in untethered soft robots and wearable devices.

Keywords: Soft Robot Applications, Soft Sensors and Actuators
Abstract: Traditional soft robots require separate sensors and actuators to precisely control their motion. A twisted-and-coiled actuator (TCA) is a new artificial muscle with both actuation and self-sensing capability that can simultaneously serve both as a sensor and an actuator allowing to control the motion of TCAs without external sensors. This paper investigates the integrated sensing and actuation for TCAs, and the self-sensing function is realized by only measuring the TCA's electrical resistance change. The closed-loop control of a single TCA is realized, and an innervated soft finger that can respond to external load without extra sensors is demonstrated. Our results will lay a foundation for integrated sensing and control by directly using the actuator, paving the way for self-contained smart robotic systems (e.g., untethered soft robots).

Keywords: Soft Robot Materials and Design, Soft Sensors and Actuators, Flexible Robots
Abstract: Tendrils are common stable structures in nature and are used for sensing, actuation, and geometrical stiffness modulation. In this paper, for the first time we exploit the helical geometry of a shape memory alloy (SMA) tendril as a simple to fabricate highly dexterous robotic continuum tentacle that we called Active Tendril-Backbone Robot (ATBR). This is achieved via partial (120 deg) activation of single helix turns resulting in backbone directional bendings. A 141.5 mm prototype (130 mm when fully compressed) has been fabricated and a simple theoretical framework is proposed and experimentally validated for modeling of the tentacle configuration. The manipulator has five 2-DOF joints capable of reaching bending angles of up to 54.5 deg and angular speed of up to 6.8 deg/s. The dexterity of the manipulator is showcased empirically in reaching complex configurations and simple navigation through confined space of a curving path.

Keywords: Aerial Systems: Applications, Soft Robot Materials and Design, Grasping
Abstract: Taking inspiration in nature, the work presented in this paper aims to develop bio-inspired claws to be used for grasping and perching in flapping wing aerial systems. This claws can be 3D printed out of two different materials and will be capable of adapt to any shape. Also, they will be soft for avoiding undesired damages on the objects when performing manipulation. These claws will be actuated by shape memory alloys (SMA) springs to get rid of the weight of traditional servos. The design of all the components will be explained in this work. Also the challenges of being able to control SMA using only a LiPo battery on an aerial vehicle will be exposed. The solutions applied and electronics used will be also described. Lastly, experiments made both in test bench as on flight will be summarized.

Keywords: Soft Robot Applications, Soft Robot Materials and Design, Soft Sensors and Actuators
Abstract: Ionic polymer-metal composites (IPMC) actuators are popular because they can be driven at a low voltage, possess excellent responsiveness, and can perform soft motions similar to that of living creatures. Conventional IPMC soft robots are manufactured by cutting and assembling IPMC sheets. However, using this conventional process to stably manufacture three-dimensional (3D)-shaped soft robots is difficult. To mitigate this problem, we propose a new method for fabricating 3D IPMC actuators in which several surface electrodes are separately fabricated from a single ion-exchange membrane. We refer to our proposal as the simultaneous 3D forming and patterning (SFP) method. Unlike the conventional IPMC fabrication process, the SFP method requires only one step to fix the ion-exchange membrane to contact masks. First, we briefly describe IPMC actuators, before introducing the proposed SFP method in detail. Next, we describe our investigations of the patterning resolution for the surface electrode using the proposed method. We fabricated two soft robot prototypes using the proposed method. The first robot is a starfish-type soft robot. Its surface electrode can be patterned in a plane using the proposed method, and independent driving is possible by applying voltage individually to the divided electrodes. The second prototype is a sea anemone-type soft robot, wherein surface electrodes can be patterned on a 3D curved surface to form a 3D shape.

Keywords: Medical Robots and Systems, Modeling, Control, and Learning for Soft Robots, Surgical Robotics: Laparoscopy
Abstract: In this paper, we present an analytical modeling approach to address the problem of tension loss in a generic variable curvature tendon-driven continuum manipulators(TDCM)occurring due to the tendon-sheath distributed friction force. Despite the previous approaches in the literature, our presented model and the iterative solution algorithm do not rely on a priori known curvature/shape of the TD-CM and can be implemented on any TD-CM with constant/ variable curvatures with a continuous neutral axis function. The performance of the proposed modeling approach in predicting the distributed tendon tension and tension loss has been evaluated via simulation and experimental studies on a TD-CM with planar bending. Results demonstrate the outstanding and accurate performance of our novel modeling and the proposed solution algorithm.

Keywords: Soft Robot Applications
Abstract: The properties and applications of auxetics havebeen widely explored in the past years. Through properutilization of auxetic structures, designs with unprecedentedmechanical and structural behaviors can be produced. Takingadvantage of this, we present the development of novel and low-cost 3D structures inspired by a simple auxetic unit. The corepart, which we call the body in this paper, is a 3D realizationof 2D rotating squares. This body structure was formed byjoining four similar structures through softer material at thevertices. A monolithic structure of this kind is accomplishedthrough a custom-built multi-material 3D printer. The modelworks in a way that, when torque is applied along the face ofthe rotational squares, they tend to bend at the vertex of thesofter material, and due to the connected-ness of the design, aproper opening and closing motion is achieved. To demonstratethe potential of this part as an important component forrobots, two applications are presented: a soft gripper anda crawling robot. Vacuum-driven actuators move both theapplications. The proposed gripper combines the benefits oftwo types of grippers whose fingers are placed parallel andequally spaced to each other, in a single design. This gripperis adaptable to the size of the object and can grasp objectswith large and small cross- sections alike. A novel bendingactuator, which is made of soft material and bends in curvaturewhen vacuumed, provides the grasping nature of the gripper.Crawling robots, in addition to their versatile nature, providea better interaction with humans. The designed crawling robotemploys negative pressure-driven actuators to highlight linearand turning locomotion.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Sensors and Actuators, Soft Robot Materials and Design
Abstract: In this paper, we address the challenge of sensor fusion in Soft Robotics for estimating forces and deformations. In the context of intrinsic sensing, we propose the use of a soft capacitive sensor to find a contact’s location, and the use of pneumatic sensing to estimate the force intensity and the deformation. Using a FEM-based numerical approach, we integrate both sensing streams and model two Soft Robotics devices we have conceived. These devices are a Soft Pad and a Soft Finger. We show in an evaluation that external forces on the Soft Pad can be estimated and that the shape of the Soft Finger can be reconstructed.

Keywords: Soft Robot Materials and Design, Soft Robot Applications, Mechanism Design
Abstract: Abstract— Tuning the stiffness of soft robots is essential in order to extend usability and control the maneuverability of soft robots. In this paper, we propose a novel mechanism that can reconfigure the stiffness of tubular structures, using pinching to induce highly directional changes in stiffness. When pinched, these tubes can be then utilized as flexure hinges to create virtual joints on demand; the orientation of the hinge axis can additionally be selected via control of the distribution of pinch forces on the surface of the tube. Through proper material and geometry selection, passive shape recovery is observed when pinching forces are removed; a proposed active shape recovery technique can further assist the tube to recover its initial shape in order to re-configure the hinge in a new orientation. The proposed mechanism has been validated in FEA as well as experimentally, looking specifically at the relation between pinching force and curvature change, as well as comparing tube stiffness between pinched and unpinched configurations. The experimental prototype detailed in this paper – and demonstrated in the associated video – is capable of controlling the generation and recovery of flexure hinges at multiple orientations around the radial axis of tubes on demand.

Keywords: Soft Robot Applications, Soft Robot Materials and Design, Motion and Path Planning
Abstract: We present a methodology for designing, fabricating, and controlling rolling membrane-driven tensegrity robots. This methodology is enabled by pneumatic membrane actuators and a generalized path planning algorithm for rolling polyhedra. Membrane actuators are planar, assembled in a scalable fashion, and amenable to arbitrary geometries. Their deformation trajectories can be tuned by varying the stacking sequence and orientation of layers of unidirectional lamina placed on their surfaces. We demonstrate the application of the same set of membrane actuators consisting of polygonal faces of Platonic Solids to create polyhedral tensegrity variants. Three specific tensegrities in the forms of cube, dodecahedron, and rhombicuboctahedron are chosen to demonstrate the path planning algorithm, though the algorithm is generalizable to any uniform or non-uniform n-sided polyhedra. The membrane-driven tensegrities are able to roll in unique trajectories and circumvent obstacles contingent on the distribution and types of polygons which constitute their faces.

Keywords: Soft Robot Materials and Design, Mechanism Design
Abstract: Soft robots have attracted much attention in recent years owing to their high adaptability. Long articulated soft robots enable diverse operations, and tip-extending robots that navigate their environment through growth are highly effective in robotic search applications. Because the robot membrane extends from the tip, these robots can lengthen without friction from the environment. However, the flexibility of the membrane inhibits tip retraction. Two methods have been proposed to resolve this issue; increasing the pressure of the internal fluid to reinforce rigidity, and mounting an actuator at the tip. The disadvantage of the former is that the increase is limited by the membrane pressure resistance, while the second method adds to the robot complexity. In this paper, we present a tip-retraction mechanism without bending motion that takes advantage of the friction from the external environment. Water is used as the internal fluid to increase ground pressure with the environment. We explore the failure pattern of the retraction motion and propose plausible solutions by using a hydrostatic skeleton robot. Additionally, we develop a prototype robot that successfully retracts by using the proposed methodology. Our solution can contribute to the advancement of mechanical design in the soft robotics field with applications to soft snakes and manipulators.

Keywords: Force and Tactile Sensing, Grasping, Soft Sensors and Actuators
Abstract: Tactile sensors are used in robot manipulation to reduce uncertainty regarding hand-object pose estimation. However, existing sensor technologies tend to be bulky and provide signals that are difficult to interpret into actionable changes. Here, we achieve wireless tactile sensing with soft and conformable magnetic stickers that can be easily placed on objects within the robot's workspace. We embed a small magnetometer within the robot's fingertip that can localize to a magnetic sticker with sub-mm accuracy and enable the robot to pick up objects in the same place, in the same way, every time. In addition, we utilize the soft magnets' ability to exhibit magnetic field changes upon contact forces. We demonstrate the localization and force-feedback features with a 7-DOF Franka arm on deformable tool use and a key insertion task for applications in home, medical, and food robotics. By increasing the reliability of interaction with common tools, this approach to object localization and force sensing can improve robot manipulation performance for delicate, high-precision tasks.

Keywords: Soft Sensors and Actuators, Biomimetics, Agricultural Automation
Abstract: The growing consumer demand for large volumes of high quality fruit has generated an increasing need for automated fruit quality control during production. Optical methods have been proved successful in a few cases, but with limitations related to the variability of fruit colors and lighting conditions during tests. Tactile sensing provides a valuable alternative, although it comes with the need of a physical interaction that could damage the fruit. To overcome these limitations, we propose to use a recently developed soft tactile sensor for non-invasive fruit quality control. The ability of the sensor to detect very small forces and to finely characterize surfaces allows to collect relevant information about the fruit by performing a very delicate physical interaction, that does not cause any damage. We report experiments in which such information is used to determine whether apples and strawberries are ripe or senescent. We test different configurations of the sensor and different classification algorithms, achieving very good accuracy for both apples (96%) and strawberries (83%).

Keywords: Soft Sensors and Actuators, Force and Tactile Sensing, Perception for Grasping and Manipulation
Abstract: Electronic skins and tactile sensors can provide the sense of touch to robotic manipulators. These sensing modalities complement existing long range optical sensors and can provide detailed information before and after contact. However, integration with existing systems can be challenging due to size constraints, the interface geometry, and restrictions of external wiring used to interface with the sensor. Here, we introduce a low-profile, wireless electronic skin for direct integration with existing robotic manipulators. The flexible electronic skin combines pressure, optical proximity sensing, and a micro-LIDAR device in a small, low profile package. Each of the sensors are characterized individually and the system is demonstrated on Robonaut 2, an anthropomorphic robot designed to work in environments designed for humans. We demonstrate the sensor can be used for contact sensing, mapping of local unknown environments, and to provide medical monitoring during an emergency in a remote area.

Keywords: Soft Sensors and Actuators, Modeling, Control, and Learning for Soft Robots
Abstract: This paper presents a vision-based sensing approach for a soft linear actuator, which is equipped with an internal camera. The proposed vision-based sensing pipeline predicts the three-dimensional tip position of the actuator. To train and evaluate the algorithm, predictions are compared to ground truth data from an external motion capture system. An off-the-shelf distance sensor is integrated in a second actuator of the same type, providing only the vertical component of the tip position and used as a baseline for comparison. The camera-based sensing pipeline runs at 40 Hz in real-time on a standard laptop and is additionally used for closed loop elongation control of the actuator. It is shown that the approach can achieve comparable accuracy to the distance sensor for measuring the linear expansion of the actuator, but additionally provide the full three-dimensional tip position.

Keywords: Flexible Robots, Underactuated Robots, Motion and Path Planning
Abstract: Robotic manipulators can be found today in most industries, from autonomous warehouses to advanced assembly lines in factories. Most of these industrial robots are characterized by having non-flexible and highly rigid links. In dense and complex environments these manipulators require many degrees of freedom (DOFs) which complicates the mechanical structure of the manipulator, as well as the control and path planning algorithms. In this work we present a minimalistic approach to reduce the number of active DOFs by using non-rigid, Hyper-Flexible Manipulators (HFM). We introduce a dynamic model of the HFM as well as a control scheme to bring the end-effector to a desired position from known initial configuration. Finally, we present experiments that support the analytic part and simulative results of this paper.

Keywords: Flexible Robots, Compliance and Impedance Control, Industrial Robots
Abstract: Lightweight robots are known to be intrinsically elastic in their joints. The established classical approaches to control such systems are mostly based on motor-side coordinates since the joints are comparatively stiff. However, that inevitably introduces errors in the coordinates that actually matter: the ones on the link side. Here we present a new joint-torque controller that uses feedback of the link-side positions. Passivity during interaction with the environment is formally shown as well as asymptotic stability of the desired equilibrium in the regulation case. The performance of the control approach is experimentally validated on DLR's new generation of lightweight robots, namely the SARA robot, which enables this step from motor-side-based to link-sided-based control due to sensors with higher resolution and improved sampling rate.

Keywords: Soft Sensors and Actuators, Soft Robot Applications
Abstract: Active acoustic sensing is being widely used in various fields, with applications including shape estimation of soft pneumatic actuators. In a pneumatic system, air tubes are frequently adopted, and thus it is essential to detect failures along the air path. Although acoustic sensing has been used for detecting contact and identifying the contact position along a tube, it has not been applied to pneumatic systems. We devised an acoustic sensing method to this end for air tubes in a pneumatic system. As pneumatic system noise propagates through the air tube, we employed this type of noise instead of the conventional method of using a sound source or emitting vibration with an additional oscillator. We conducted several experiments that confirm the feasibility of the proposed method, succeeding to estimate the contact point on a 16 m air tube.

Keywords: Soft Sensors and Actuators, Soft Robot Applications
Abstract: We propose a twin coil spring-based soft actuator that can move forward and backward with extensibility and can bend left and right with flexibility. It is driven by two flexible ultrasonic motors, each consisting of a compact metallic stator and an elastic elongated coil spring. The position of the end effector is determined by the positional relationship of the two coils and can be kinetically controlled with a constant curvature model. In our design, the coil springs act not only as a flexible slider but also as a resistive positional sensor. Changes in the resistance between the stator and the coil spring end are converted to a voltage and used for position detection. Each flexible ultrasonic motor with the self-sensing is experimentally evaluated, and it has shown good response characteristics, high sensor linearity, and robustness, without losing flexibility and controllability. We build a twin coil spring-based flexible ultrasonic motor prototype and demonstrate feedback control of planar motion based on the constant curvature model.

Keywords: Biomimetics, Simulation and Animation, Dynamics
Abstract: Although the biologically flexible flippers of the cormorant (Phalacrocorax) are believed to be one of the most important features to achieve optimal swimming performance before take off, studies on a deformable bionic flipper with a non-uniformly distributed stiffness are rare. In this paper, we present a fully coupled fluid-structure interaction solver based on computational fluid dynamics (FSI-CFD), which can deal with the dynamic interplay between flexible aquatic animals and the ambient medium. The numerical solutions of the physical models are presented to gain the unsteady hydrodynamic distribution during the power stroke and recovery stroke. We quantified the three-axis component distribution, and the results show that the horizontal force of the fluid will not provide positive thrust during the initial take-off stage. Greater lift and forerake moment are generated, which brings the body off the water as soon as possible and reduces the angle of attack. As the angle of attack decreases, the positive thrust will be generated, and the forward velocity and the lift will be further increased. As the draft area of the cormorant is reduced, the wings will start flapping, further increasing the lift and thrust, and thus taking off. This solver will serve as a framework for the future bio-inspired studies involving active and passive control associated with complex structural material.

Keywords: Soft Sensors and Actuators, Soft Robot Materials and Design, Biological Cell Manipulation
Abstract: We used cells, which are the units that make up a living body, as building blocks to design a biomachine hybrid system and develop a tactile sensor that uses living cells as sensor receptors. We fabricated a novel cell tactile sensor with the electrodes formed using printed electronics technology. This sensor comprises elastic electrodes mounted on a soft material to acquire tactile information; similar to a conventional cell tactile sensor, it acquires signals through mechanical stimulation. Further, self-organization of cells can be induced, and logical processing such as selective responses to stimuli can be performed directly by the physical system, without any coding using programming languages. The proposed novel cell tactile sensor that uses printed electrodes is small enough to mount on robots. Interestingly, we confirmed the self-healing properties of the proposed sensor after cells were injured mechanically.

Keywords: Haptics and Haptic Interfaces, Soft Sensors and Actuators, Soft Robot Applications
Abstract: In this paper, we report a self-sensing soft tactile actuator based on Dielectric elastomer actuator (DEA) for wearable haptic interface. DEAs are one of electroactive polymer actuators, which are reported to have large area strain and fast response speed. A soft tactile actuator is constructed of a multi-layered DEA membrane layer, a passive membrane layer, and an inner circular pillar. The soft actuator was optimized by varying the geometry, and the force and displacement tests were conducted under a frequency range of 0 to 30 Hz. The selected actuator produces an output force up to 0.9 N, with a displacement of 1.43 mm. To provide accurate physical force feedback to the user, the actuator is integrated with a 1.1 mm thick film-type soft force sensor that enables feedback control. Under the pressure, touch layer contacts with the core, and the light inside the core scatters to the touch layer. A fabricated soft force sensor can measure the force in a range of 0 to 1.25 N under various frequency ranges. Our wearable prototype exhibits high output force of 0.9 N, as well as flexibility, conformity, and light-weight structure (3.2 g).

Keywords: Additive Manufacturing, Biomimetics, Nanomanufacturing
Abstract: With reduction in the scale of unmanned air vehicles, there is an increasing need for lightweight, compact, low-power sensors and alternate sensing modalities to facilitate flight control and navigation. This paper presents a novel method to fabricate a micro-scale artificial hair sensor that is capable of directional airflow sensing. The sensor consists of a high-aspect ratio hair structure attached to a thin flexible membrane. When subjected to airflow, the hair deflection induces a deformation of the membrane. Two pairs of perpendicular electrodes are attached to the membrane, which allow the sensing of airflow amplitude and direction through the measurement of differential capacitance. The sensor structure is fabricated by using two photon polymerization, which is integrated onto a miniature PCB circuit board to allow simple measurement. The sensor's responses to static displacement loading from different directions were characterized, and are in good agreement with the simulation results. Finally, the sensor's capability for directional airflow measurement was demonstrated with a clear correlation between flow speed and sensor output.

Keywords: Soft Sensors and Actuators, Soft Robot Materials and Design, Sensor Networks
Abstract: Thin and imperceptible soft skins that can detect internal de-formations as well as external forces, can go a long way to ad-dress perception and control challenges in soft robots. However, decoupling proprioceptive and exteroceptive stimuli is a chal-lenging task. In this paper, we present a silicone-based, capaci-tive E-skin for exteroception and proprioception (SCEEP). This soft and stretchable sensor can perceive stretch as along with touch at 100 different points via its 100 tactels. In this paper, we present a novel algorithm that decouples global strain from local indentations due to external forces. The soft skin is 10.1cm in length and 10cm in width and can be used to accurately measure the global strain of up to 25% with an error of under 3%; while at the same time, can determine the amplitude and position of local indentations. This is a step towards a fully soft electronic skin that can act as a proprioceptive sensor to meas-ure internal states while measuring external forces.

Keywords: Soft Robot Applications, Soft Sensors and Actuators, Novel Deep Learning Methods
Abstract: CCD camera-based tactile sensors provide high-resolution information about the deformation of soft and elastic interfaces. However, they have poor scalability as it is difficult to sense a large surface area without increasing the distance between the camera and the interface or using multiple processing chips. For example, using such tactile sensors for a whole robotic arm is not yet possible. In this work, we demonstrate a data driven method that can reconstruct the high-resolution information about deformation of the soft interface while keeping the space requirements and power consumption relatively low. Our modified tactile sensor incorporates two independent sensing techniques, one low- and one high-resolution, and we learn to map to the latter from the former. As a low-resolution sensor, we use liquid-filled channels that transmit the information from the location of the tactile interaction to a rigid display, where the liquid displacements are tracked by a CCD camera. Simultaneously, the same interaction is measured by tracking the markers on the bottom of the sensor using a second CCD camera. After data collection, we train two different machine learning models to reconstruct the time series of the high-resolution sensor. By training a convolutional autoencoder (CAE) and attaching it to the recurrent neural network (RNN), we demonstrate the reconstruction of high-resolution video frames using only the time series of the low-resolution sensor.

Keywords: AI-Based Methods
Abstract: Personalized online machine learning allows a very accurate modelling of individual behavior and demands. In particular, a system that dynamically adapts during runtime can initiate a continuous collaboration with its user where both alternately adjust to each other to maximize the system's utility. However, in application scenarios based on supervised learning it is often unclear how to obtain the required ground truth for such dynamic systems. In this paper, we focus on applications where a real-time classification of sequential data is crucial. Concretely, we propose to adapt an online personalized model solely based on pseudo-ground-truth information which is provided by another machine learning model. This model has the advantage to classify sequences in retrospective with a small delay and thus is able to achieve a higher performance than real-time systems. In particular, it is a pre-trained offline model, which means that no ground-truth information is necessary during runtime. We apply the proposal on the task of online action classification, for which the benefits of personalization have been recently emphasized.

Keywords: Deep Learning for Visual Perception, Surveillance Systems, RGB-D Perception
Abstract: We present the sequential correlation network (SCN) to improve concurrent activity detection. SCN combines a recurrent neural network and a correlation model hierarchically to model the complex correlations and temporal dynamics of concurrent activities. SCN has several advantages that enable effective learning even from a small dataset for real-world deployment. Unlike the majority of approaches assuming that each subject performs one activity at a time, SCN is end-to-end trainable, ie, it can automatically learn the inclusive or exclusive relations of concurrent activities. SCN is lightweight in design using only a small set of learnable parameters to model the spatio-temporal correlations of activities. This also enhances the explainability of the learned parameters. Furthermore, the learning of SCN can benefit from the initialization using semantically meaningful priors. We evaluate the proposed method against the state-of-the-art method on two benchmark datasets with human skeletal data, SCN achieves comparable performance to the SOTA but with much faster inference speed and less memory usage.

Keywords: Health Care Management, Novel Deep Learning Methods
Abstract: Electronic Health Record (EHR) and healthcare claim data provide rich clinical information for time series analysis. In this work, we provide a different angle of solving healthcare multivariate time series classification by turning it into a computer vision problem. We propose a Convolutional Feature Engineering (CFE) methodology, that can effectively extract long sequence dependency time series features. Combined with LightGBM, it can achieve the state-of-the-art results with 35X speed acceleration compared with LSTM based approaches on MIMIC-III In Hospital Mortality benchmark task. We deploy CFE based LightGBM into our Mortality Early Warning System at Humana, and train it on 1 million member samples. The offline metrics shows that this new approach generates better-quality predictions than previous LSTM based approach, and meanwhile greatly decrease the training and inference time.

Keywords: Big Data in Robotics and Automation, Computer Vision for Transportation
Abstract: This work studies the problem of predicting the sequence of future actions for surrounding vehicles in real-world driving scenarios. To this aim, we make three main contributions. The first contribution is an automatic method to convert the trajectories recorded in real-world driving scenarios to action sequences with the help of HD maps. The method enables automatic dataset creation for this task from large-scale driving data. Our second contribution lies in applying the method to the well-known traffic agent tracking and prediction dataset Argoverse, resulting in 228,000 action sequences. We also manually annotated 2,245 action sequences for testing. The third contribution is to propose a novel action sequence prediction method by integrating past positions and velocities of the traffic agents, map information and social context into a single end-to-end trainable neural network. Our experiments prove the merit of the data creation method and the value of the created dataset -- prediction performance improves consistently with the size of the dataset and shows that our action prediction method outperforms comparing models.

Keywords: Novel Deep Learning Methods, Representation Learning, Big Data in Robotics and Automation
Abstract: Sensor-based activity recognition for construction vehicles is useful for evaluating the skills of the operator, measuring work efficiency, and many other use cases. Therefore, many researches have explored robust activity-recognition models. However, it remains a challenge to apply the model to many construction sites because of the imbalance of the dataset. While it is natural to employ multi-label representation on imbalanced data with a large number of activity categories, multi-label robust classification for activity recognition has yet to be resolved because of the nature of the time-series property. In this work, we propose a novel multi-label long short-term memory (LSTM) model, which is effective for the sequence multi-labeling problem. The proposed model has connections to the temporal direction and attribute direction, which exploit both the temporal pattern and co-occurrence among attributes. In addition, by providing a bidirectional connection structure in the attribute direction, the model enables us to alleviate the dependency of the chain order in what we call ``classifier chain``, which is a classical approach to multi-label classification.

To validate our methods, we conduct experiments using real-world construction-vehicle dataset.

Keywords: Big Data in Robotics and Automation, Humanoid Robot Systems, Cognitive Human-Robot Interaction
Abstract: The Everyday Activities Science and Engineering(EASE) Collaborative Research Consortium’s mission to enhance the performance of cognition-enabled robots establishes its foundation in the EASE Human Activities Data Analysis Pipeline. Through collection of diverse human activity information resources, enrichment with contextually relevant annotations, and subsequent multimodal analysis of the combined data sources, the pipeline described will provide a rich resource for robot planning researchers, through incorporation in the OPENEASE cloud platform.

Keywords: Calibration and Identification, RGB-D Perception
Abstract: This paper describes a calibration method for RGB-D camera networks consisting of not only static overlapping, but also dynamic and non-overlapping cameras. The proposed method consists of two steps: online visual odometry-based calibration and depth image-based calibration refinement. It first estimates the transformations between overlapping cameras using fiducial tags, and bridges non-overlapping camera views through visual odometry that runs on a dynamic monocular camera. Parameters such as poses of the static cameras and tags, as well as dynamic camera trajectory, are estimated in the form of the pose graph-based online landmark SLAM. Then, depth-based ICP and floor constraints are added to the pose graph to compensate for the visual odometry error and refine the calibration result. The proposed method is validated through evaluation in simulated and real environments, and a person tracking experiment is conducted to demonstrate the data integration of static and dynamic cameras.

Keywords: Calibration and Identification, Formal Methods in Robotics and Automation, Multi-Modal Perception
Abstract: To fuse information from a 3D Light Detection and Ranging (LiDAR) sensor and a camera, the extrinsic transformation between the sensor coordinate systems needs to be known. Therefore, an extrinsic calibration must be performed, which is usually based on features extracted from sensor data. Naturally, sensor errors can affect the feature extraction process, and thus distort the calibration result. Unlike previous works, which do not consider the uncertainties of the sensors, we propose a set-membership approach that takes all sensor errors into account. Since the actual error distribution of off-the-shelf sensors is often unknown, we assume to only know bounds (or intervals) enclosing the sensor errors and accordingly introduce novel error models for both sensors. Next, we introduce interval-based approaches to extract corresponding features from images and point clouds. Due to the unknown but bounded sensor errors, we cannot determine the features exactly, but compute intervals guaranteed to enclose them. Subsequently, these feature intervals enable us to formulate a Constraint Satisfaction Problem (CSP). Finally, the CSP is solved to find a set of solutions that is guaranteed to contain the true solution and simultaneously reflects the accuracy of the calibration. Experiments using simulated and real data validate our approach and show its advantages over existing methods.

Keywords: Multi-Modal Perception, Sensor Fusion, Calibration and Identification
Abstract: In this paper we perform an extensive experimental evaluation of three planar target based 3D-LIDAR camera calibration algorithms, on a sensor suite consisting multiple 3D-LIDARs and cameras, assessing their robustness to random initialization and by using metrics like Mean Line Re-projection Error (MLRE) and Factory Stereo Calibration Error. We briefly describe each method and provide insights into practical aspects like ease of data collection. We also show the effect of noisy sensor on the calibration result and conclude with a note on which calibration algorithm should be used under what circumstances.

Keywords: Range Sensing, Sensor Networks, Localization
Abstract: This paper presents the development of a Kalman filter-based range estimation technique to precisely calculate the inter-node ranges of Ultra-Wide Band (UWB) modules. Relative clock tracking filters running between every anchor pair tracks relative clock dynamics while estimating the time of flight as a filter state. Both inbound and outbound message timestamps are used to update the filter to make the time of flight observable in the chosen state space design. A faster relative clock filter convergence has been achieved with the inclusion of the clock offset ratio as a measurement additional to the timestamps. Furthermore, a modified gradient clock synchronization algorithm is used to achieve global clock synchronization throughout the network. A correction term is used in the gradient clock synchronization algorithm to enforce the global clock rate to converge at the average of individual clock rates while achieving asymptotic stability in clock rate error state. Experiments are conducted to evaluate synchronization and ranging accuracy of the proposed range estimation approach.

Keywords: Sensor Fusion, Calibration and Identification, Omnidirectional Vision
Abstract: In this paper, we propose a unified calibration method for multi-camera multi-LiDAR systems. Only using a single planar checkerboard, the captured checkerboard frames by each sensor are classified as either global frames if they are observed by at least two sensors, or a local frame if observed by a single camera. Both global and local frames of each camera are used to estimate its intrinsic parameters, whereas the global frames between sensors are for computing their relative poses. The global frames are used to optimize intrinsic parameters of the cameras, and global poses of the sensors and the checkerboards while the local frames are used to optimize intrinsic parameters of the cameras to prevent their over-fitting to the global frames. In contrast to the previous methods that simply combine the pairwise poses (e.g., camera-to-camera or camera-to-LiDAR) that are separately estimated, we further optimize the sensor poses in the system globally using all observations as the constraints in the optimization problem. We find that the point-to-plane distances are effective as camera-to-LiDAR constraints where the points are 3D positions of the checkerboard corners and the planes are estimated from the LiDAR point-cloud. Also, abundant corner observations in the local frames enable the joint optimization of intrinsic and extrinsic parameters in a unified framework. In contrast to previous calibration methods which optimize relative poses between a pair of sensors (e.g., camera to camera, or camera to LiDAR) and merging them, we jointly optimize extrinsic parameters of the sensors in the global coordinate system via bundle adjustment. The proposed calibration method utilizes entire observations in a unified global optimization framework, and it significantly reduces the error caused by the simple composition of the relative sensor poses. The proposed global optimization framework can decrease cumulative errors from merging the relative poses by maximizing the utilization of meaningful sensor observations. We extensively evaluate the proposed algorithm qualitatively and quantitatively using real and synthetic datasets. We plan to make the implementation open to the public with the paper publication.

Keywords: Industrial Robots, Deep Learning in Grasping and Manipulation
Abstract: A practical robotic bin-picking system requires a high grasp success rate for various objects. Also, the system must be capable of coping with various constraints and their changes flexibly. To resolve these issues, this study proposes a novel deep learning-based method that exploits a simulator to generate desired grasping actions. The features of this method are as follows: (1) Grasping conditions for any object can be flexibly customizable in the simulated environment to improve the real-world grasping actions. (2) Sensor input (RGB image) is directly regressed to grasping actions by using convolutional processing. Owing to these features, the system using the proposed method can grasp objects with geometric variations, semi-transparent objects, and objects with a biased center of gravity. Experimental results on a real robot system show that the proposed method exhibits a high grasp success rate for four types of different objects and adapts to two additional constraints of grasping condition.

Keywords: Compliant Assembly, Factory Automation, Assembly
Abstract: A new approach to precision mating of heavy objects suspended from overhead cranes is presented. We have found through experiments that heavy shafts suspended with multiple cables at specific positions and orientations can be inserted into a chamfered hole despite a small clearance. This will allow an overhead crane, although limited in positioning accuracy, to execute precision assembly of a heavy shaft simply by holding it with multiple cables and lowering it onto the chamfered hole of a fixed object. Unlike the well-known Remote Center Compliance (RCC) hand, this method does not use a two-layer elastic structure but exploits the physical properties of cables. Specifically, cables go slack when a compressive load is applied. This unidirectional load bearing property is exploited to suspend a heavy shaft such that it is not over-constrained during insertion. Conditions for the cable attachment position and orientation for successful insertion are obtained. A proof-of-concept prototype is developed and experimental verification of the principle and analysis are presented..

Keywords: Assembly, Contact Modeling, Reinforecment Learning
Abstract: In this paper, we propose a novel sim-to-real framework to solve bolting tasks with tight tolerance and complex contact geometry which are hard to be modeled. The sim-to-real has desirable features in terms of cost and safety, however, that of the assembly task is rare due to the lack of simulator, which can robustly render multi-contact assembly. We implement the sim-to-real transfer of nut tightening policy which is adaptive to uncertain bolt positions. This can be realized through developing a novel contact model, which is fast and robust to complex assembly geometry, and novel hierarchical controller with reinforcement learning (RL), which can perform the tasks with a narrow and complicated path. The fast and robust contact model is achieved by utilizing configuration space abstraction and passive midpoint integrator (PMI), which render the simulator robust even in a high stiffness contact condition. And we use sampling-based motion planning to construct a path library and design linear quadratic tracking controller as a low-level controller to be compliant and avoid local optima. Additionally, we use the RL agent as a high-level controller to make it possible to adapt to the bolt position uncertainty, thereby realizing sim-to-real. Experiments are performed to verify our proposed sim-to-real framework.

Keywords: Assembly, Compliant Assembly
Abstract: Current trends in industrial automation favor agile systems that allow adaptation to rapidly changing task requirements and facilitate customized production in smaller batches. This work presents a flexible manufacturing system relying on compliance control, CAD based localization, and a multi-modal gripper to enable fast and efficient task programming for assembly operations. CAD file processing is employed to extract component pose data from 3D assembly models, while the system's active compliance compensates for errors in calibration or positioning. To minimize retooling delays, a novel gripper design incorporating both a parallel jaw element and a rotating module is proposed. The developed system placed first in the manufacturing track of the Robotic Grasping and Manipulation Competition of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2019, experimentally validating its efficiency.

Keywords: Manipulation Planning, Learning from Demonstration, Assembly
Abstract: Enabling robots to quickly learn manipulation skills is an important, yet challenging problem. Such manipulation skills should be flexible, e.g., be able adapt to the current workspace configuration. Furthermore, to accomplish complex manipulation tasks, robots should be able to sequence several skills and adapt them to changing situations. In this work, we propose a rapid robot skill-sequencing algorithm, where the skills are encoded by object-centric hidden semi-Markov models. The learned skill models can encode multimodal (temporal and spatial) trajectory distributions. This approach significantly reduces manual modeling efforts, while ensuring a high degree of flexibility and re-usability of learned skills. Given a task goal and a set of generic skills, our framework computes smooth transitions between skill instances. To compute the corresponding optimal end-effector trajectory in task space we rely on Riemannian optimal controller. We demonstrate this approach on a 7~DoF robot arm for industrial assembly tasks.

Keywords: Factory Automation, Reinforecment Learning, Assembly
Abstract: Recent progress in deep reinforcement learning has enabled agents to autonomously learn complex control strategies from scratch. Model-free approaches like Deep Deterministic Policy Gradients (DDPG) seem promising for applications with intricate dynamics, such as contact-rich manipulation tasks. However, these methods typically require large amounts of training data or meticulous hyperparameter tuning, limiting their usefulness for real-world robotics applications. In this paper, we evaluate and benchmark our recently proposed approach for improving model-free reinforcement learning with DDPG through Qgraph-based bounds in temporal difference learning. We directly apply the algorithm to a challenging real-world industrial insertion task and assess its performance (see https://youtu.be/Z_GcNbCWE-E). Empirical results show that the insertion task can be learned despite significant frictional forces and uncertainty, even in sparse-reward settings. We present an in-depth comparison based on a large number of experiments and demonstrate the advantages and performance of Qgraph-bounded DDPG: the learning process can be significantly sped up, robustified against bad choices of hyperparameters and runs with less memory requirements. Lastly, the presented results extend the current theoretical understanding of the link between data graph structure and soft divergence in DDPG.

Keywords: Intelligent and Flexible Manufacturing, Cooperating Robots, Planning, Scheduling and Coordination
Abstract: Multi-functional cells with cooperating teams of robots promise to be flexible, robust, and efficient and, thus, are a key to future factories. However, their programming is tedious and AI-based planning for multiple robots is computionally expensive. In this work, we present a modular and efficient twolayer planning approach for multi-robot assembly. The goal is to generate the program for coordinated teams of robots from an (enriched) 3D model of the target assembly. Although the approach is both motivated and evaluated with LEGO, which is a challenging variant of blocks world, the approach can be customized to different kinds of assembly domains.

Keywords: Redundant Robots, Whole-Body Motion Planning and Control, Service Robots
Abstract: In this paper an experimental study of set-based task-priority kinematic control for a dual-arm mobile robot is developed. The control strategy for the coordination of the two manipulators and the mobile base relies on the definition of a set of elementary tasks to be properly handled depending on their functional role. In particular, the tasks have been grouped into three categories: safety, operational and optimization tasks. The effectiveness of the resulting task hierarchy has been validated through experiments on a Kinova Movo robot, in a domestic use case scenario.

Keywords: Cooperating Robots, Dual Arm Manipulation, Multi-Robot Systems
Abstract: Cooperative manipulation offers many advantages over single-arm manipulation. However, this comes at a cost of added complexity, both in modeling and control of multi-arm systems. Much research has been focused on determining optimal load distribution strategies based on several objective functions, some of which include manipulability, energy consumption and joint torque minimization. This paper presents an internal loading strategy that is subject to the estimate of the external disturbances along the body of one or more of the arms involved in the manipulation process. The authors of this paper propose a reaction strategy to external disturbances by transforming the disturbance forces into internal forces on the object through appropriate load distribution on the cooperative arms. The goal is to have a set-point on the object, track a given trajectory while compensating for external disturbances along the links of some of the robot arms involved in the cooperative manipulation.

Keywords: Mobile Manipulation, Optimization and Optimal Control, Distributed Robot Systems
Abstract: We present a distributed algorithm to enable a group of robots to collaboratively manipulate an object to a desired configuration while avoiding obstacles. Each robot solves a local optimization problem iteratively and communicates with its local neighbors, ultimately converging to the optimal trajectory of the object over a receding horizon. The algorithm scales efficiently to large groups, with a convergence rate constant in the number of robots, and can enforce constraints that are only known to a subset of the robots, such as for collision avoidance using local online sensing. We show that the algorithm converges many orders of magnitude faster, and results in a tracking error two orders of magnitude lower, than competing distributed collaborative manipulation algorithms based on Consensus alternating direction method of multipliers (ADMM).

Keywords: Distributed Robot Systems, Multi-Robot Systems, Networked Robots

Keywords: Multifingered Hands, Visual Servoing, Manipulation Planning
Abstract: In this paper, we propose a "multi-vision hand" system which has a number of small high-speed cameras arranged on its surface. And we propose visual serving control using the multi-eye system. As a target of control with the multi-vision hand system, the ball catching motion control. In the proposed catching control, the catch position of the ball estimated is corrected by the multi-vision hand in realtime. In experiments, catch operation was successful by the correction of the catch position, confirming the effectiveness of the multi-vision hand system.

Keywords: Grippers and Other End-Effectors, Grasping, Compliance and Impedance Control
Abstract: In this study, Magripper, a highly backdrivable gripper, is developed to achieve high-speed hitting grasping executed seamlessly from reaching. The gripper is designed to achieve both high speed and environmental adaptability. The key element is backdrivability in terms of both hardware and control. In Magripper, a magnetic gear is introduced to passively absorb shock in the moment of contact as a means of hardware backdrivability, and backdrive control is implemented based on the Zener model. After developing a hitting grasping framework, high-speed hitting grasping tasks with a wood block, a wood cylinder, and a plastic coin are conducted using only servo control without sensors, such as cameras and tactile sensors. In particular, coin grasping with high-speed movement is very difficult because collisions with environmental objects such as the floor and desk, are likely, which may break a robot.

Keywords: Dexterous Manipulation, Mechanism Design, Factory Automation
Abstract: We present the design of an active two-phase finger for mechanically mediated dexterous manipulation. The finger enables re-orientation of a grasped object by using a pneumatic braking mechanism to transition between free-rotating and fixed (i.e., braked) phases. Our design allows controlled high-bandwidth (5 Hz) phase transitions independent of the grasping force for manipulation of a variety of objects. Moreover, its thin profile (1 cm) facilitates picking and placing in clutter. Finally, the design features a sensor for measuring fingertip rotation to support feedback control. We experimentally characterize the finger's load handling capacity in the brake phase and rotational resistance in the free phase. We also demonstrate several pick-and-place manipulations common to industrial and laboratory automation settings that are simplified by our design.

Keywords: Grippers and Other End-Effectors, In-Hand Manipulation, Deep Learning in Grasping and Manipulation
Abstract: The ability to perform in-hand manipulation still remains an unsolved problem; having this capability would allow robots to perform sophisticated tasks requiring repositioning and reorienting of grasped objects. In this work, we present a novel non-anthropomorphic robot grasper with the ability to manipulate objects by means of active surfaces at the fingertips. Active surfaces are achieved by spherical rolling fingertips with two degrees of freedom (DoF) - a pivoting motion for surface reorientation - and a continuous rolling motion for moving the object. A further DoF is in the base of each finger, allowing the fingers to grasp objects over a range of size and shapes. Instantaneous kinematics was derived and objects were successfully manipulated both with a custom handcrafted control scheme as well as one learned through imitation learning, in simulation and experimentally on the hardware.

Keywords: Grippers and Other End-Effectors, Grasping, In-Hand Manipulation
Abstract: Robotic hands with anthropomorphism considerations are of prominent popularity in humancentered environment. Existing anthropomorphic robotic hands achieving part or most of human hand comparable dexterity have been applied as various robotic endeffectors and prosthetics. However, two deficiencies are evident that the design for a dexterous anthropomorphic hand is largely based on the intuition of designers and the dexterity of robotic hand is hard to evaluate. To tackle these two challenges, this paper firstly summarizes 50 hand dexterity benchmarks (HD-marks) to evaluate hand dexterity comprehensively from three perspectives. Secondly, a novel 22-DOFs soft robotic hand (S-22) replicates human hand kinematics is used to demonstrate all the 50 HD-marks. Thirdly, 7 critical joint-based kinematic motions (K-motions) and their correlation with the 50 HD-marks are established. Therefore, a clear robotic hand design guideline is built by mapping the hand functional dexterity to the required joint kinematics.

Keywords: In-Hand Manipulation, Force and Tactile Sensing, Deep Learning in Grasping and Manipulation
Abstract: The use of tactile information is one of the most important factors for achieving stable in-grasp manipulation. Especially with low-cost robotic hands that provide low-precision control, robust in-grasp manipulation is challenging. Abundant tactile information could provide the required feedback to achieve reliable in-grasp manipulation also in such cases. In this research, soft distributed 3-axis skin sensors (``uSkin'') and 6-axis F/T (force/torque) sensors were mounted on each fingertip of an Allegro Hand to provide rich tactile information. These sensors yielded 78 measurements for each fingertip (72 measurements from the uSkin and 6 measurements from the 6-axis F/T sensor). However, such high-dimensional tactile information can be difficult to process because of the complex contact states between the grasped object and the fingertips. Therefore, a convolutional neural network (CNN) was employed to process the tactile information. In this paper, we explored the importance of the different sensors for achieving in-grasp manipulation. Successful in-grasp manipulation with untrained daily objects was achieved when both 3-axis uSkin and 6-axis F/T information was provided and when the information was processed using a CNN.

Keywords: Entertainment Robotics, Dexterous Manipulation, Model Learning for Control
Abstract: Juggling manipulation is one of difficult manipulation to acquire because some of such manipulation is unstable and also its physical model is unknown due to the complex nonprehensile manipulation. To acquire these unstable unknown-dynamics juggling manipulation, we propose a method for designing the predictive model of such manipulation with a deep neural network, and also a real-time optimal control law with some robustness and adaptability using backpropagation of the network. In this study, we applied this method to diabolo orientation stabilization, which is one of unstable unknown-dynamics juggling manipulation. We verify the effectiveness of the proposed method by comparing with basic controllers such as P Controller or PID Controller, and also check the adaptability of the proposed controller by some experiments with a real life-sized humanoid robot.

Keywords: Grasping, Manipulation Planning, Optimization and Optimal Control
Abstract: In this paper, we study the problem of computing grasping forces for quasi-static manipulation of large and heavy objects, by exploiting object-environment contacts. We present a general formulation of this problem as a Second-Order Cone Program (SOCP) that considers (i) contact friction constraints at the object-manipulator contacts and object-environment contacts, (ii) force/moment equilibrium constraints, and (iii) manipulator joint torque constraints. The SOCP formulation implies that the optimal grasping forces for manipulating objects with the help of the environment can be computed efficiently. Different optimization objectives like minimizing contact forces at the object-manipulator contacts or minimizing joint torques of manipulators can be considered. We evaluated our method by simulations in two different scenarios.

Keywords: Multifingered Hands
Abstract: Synergy provides a practical approach for expressing various postures of a multi-fingered hand. However, a conventional synergy defined for reproducing grasping postures cannot perform in-hand manipulation, e.g., tasks that involve simultaneously grasping and manipulating an object. Locking the position of particular fingers of a multi-fingered hand is essential for in-hand manipulation tasks either to hold an object or to fix unnecessary fingers. When using conventional synergy based control to manipulate an object, which requires locking some fingers, the coordination of joints is heavily restricted, decreasing the dexterity of the hand. We propose a functionally divided manipulation synergy (FDMS) method, which provides a synergy-based control to achieves both dimensionality reduction and in-hand manipulation. In FDMS, first, we define the function of each finger of the hand as either ``manipulation" or ``fixed." Then, we apply synergy control only to the fingers having the manipulation function, so that dexterous manipulations can be realized with a few control inputs. Furthermore, we propose the Synergy Switching Framework as a method for applying a finely defined FDMS to sequential task changes. The effectiveness of our method is experimentally verified.

Keywords: Grasping, Dynamics, Contact Modeling
Abstract: One of the key advantages of robots is the high speeds at which they can operate. In industrial settings, increased velocities can lead to higher throughputs and improved efficiency. Some manipulation tasks might require the robot to perform highly dynamic operations such as shaking, or swinging while grasping an object. These fast movements may produce high accelerations and thus give rise to inertial forces that can cause a grasped object to slip. In this paper a method is proposed to determine the inertial forces that arise on a grasped object during a trajectory, find the instances at which the object might slip, and avoid these slippages by changing the trajectory, namely the orientation of the object. To exemplify the usage of this approach, two grasping tasks are realised: a prehensile and a non-prehensile grasp, and strategies to successfully perform these tasks without changing the overall duration of the trajectory are defined and evaluated.

Keywords: Dexterous Manipulation, In-Hand Manipulation, Manipulation Planning
Abstract: We address the problem of controlling a partially constrained trajectory of the manipulation frame–an arbitrary frame of reference rigidly attached to the object–as the desired motion about this frame is often underdefined. This may be apparent, for example, when the task requires control only about the translational dimensions of the manipulation frame, with disregard to the rotational dimensions. This scenario complicates the computation of the grasp frame trajectory, as the mobility of the mechanism is likely limited due to the constraints imposed by the closed kinematic chain. In this letter, we address this problem by combining a learned, object-agnostic manipulation model of the gripper with Model Predictive Control (MPC). This combination facilitates an approach to simple vision-based control of robotic hands with generalized models, enabling a single manipulation model to extend to different task requirements. By tracking the hand-object configuration through vision, the proposed framework is able to accurately control the trajectory of the manipulation frame along translational, rotational, or mixed trajectories. We provide experiments quantifying the utility of this framework, analyzing its ability to control different objects over varied horizon lengths and optimization iterations, and finally, we implement the controller on a physical system.

Keywords: Grasping, Soft Robot Materials and Design, Biomimetics
Abstract: Stable grip of wet, deformable objects is a challenging task for robotic grasping and manipulation, especially for food products’ handling. The wet, slippery interfaces between the object and robotic fingers may require larger gripping force, resulting in higher risk of damaging the grasped object. This research aims to evaluate the role of micro-patterned soft pad on enhancement of wet adhesion in grasping a food sample in wet environment. We showcased this scenario with a tofu block 19.6mm×19.6mm×15mm that is soft, and deformable object, gripped by a soft robotic gripper with two fingers. Each fingertip’s surface, which directly makes contact with the tofu, was deposited soft pads in two cases: normal pads (flat surface) and a micropatterned pads. The micropatterned pad comprises of 14400 square cells, each cell has four 85 μm edges, surrounded by a channel network with 44 μm in depth. We conducted estimation of grasped force generated by pads in two cases, then verified by actual setup in griping the tofu block. Both estimated and experimental results reveal that the micropatterned pad decreased necessary load acting on the tofu’s surface 2.2 times lower than that of the normal one, while maintaining the stability of the grasped tofu. The showcase in this paper supported the potential of micro patterns on soft fingertip in grasping deformable objects in wet environments without complicated control strategy, promising wider applications for robot in service section or food industry.