Keywords: Aerial Systems: Applications
Abstract: VIJAY KUMAR is the Nemirovsky Family Dean of Penn Engineering with appointments in the Departments of Mechanical Engineering and Applied Mechanics, Computer and Information Science, and Electrical and Systems Engineering at the University of Pennsylvania. He received his Bachelors of Technology from the Indian Institute of Technology, Kanpur and his Ph.D. from The Ohio State University in 1987. He has been on the Faculty in the Department of Mechanical Engineering at the University of Pennsylvania since1987.

Dr. Kumar served as the Deputy Dean for Research in the School of Engineering and Applied Science from 2000-2004. He directed the GRASP Laboratory, a multidisciplinary robotics and perception laboratory, from 1998-2004. He was the Chairman of the Department of Mechanical Engineering and Applied Mechanics from 2005-2008. He then served as the Deputy Dean for Education in the School of Engineering and Applied Science from 2008-2012. He has served as the assistant director of robotics and cyber physical systems at the White House Office of Science and Technology Policy (2012 – 2013).

Dr. Kumar is a Fellow of the American Society of Mechanical Engineers (ASME) and the Institute of Electrical and Electronic Engineers (IEEE). He has served on the editorial boards of the IEEE Transactions on Robotics and Automation, IEEE Transactions on Automation Science and Engineering, ASME Journal of Mechanical Design, the ASME Journal of Mechanisms and Robotics and the Springer Tract in A

Keywords: Manipulation Planning
Abstract: Matthew T. Mason has been working on robotic manipulation since the 1970's. He earned the BS, MS, and PhD degrees in Computer Science and Artificial Intelligence at MIT, finishing his PhD in 1982. Since that time he has been on the faculty at Carnegie Mellon University. He was Director of the Carnegie Mellon Robotics Institute from 2004 to 2014, and is presently Professor of Robotics and Computer Science. He is a Fellow of the AAAI, and a Fellow of the IEEE. He is a winner of the System Development Foundation Prize, the IEEE RAS Pioneer Award, and the 2018 IEEE Robotics and Automation Award.

Keywords: Education Robotics
Abstract: Dutch FashionTech designer Anouk Wipprecht creates designs ahead of her time; combining the latest in science and technology to make fashion an experience that transcends mere appearances. She wants her garments to facilitate and augment the interactions we have with ourselves and our surroundings. Her Spider Dress is a perfect example of this aesthetic, where sensors and moveable arms on the dress help to create a more defined boundary of personal space while employing a fierce style. Partnering up with companies such as Intel, AutoDesk, Google, Arduino, Microsoft, Adidas, Cirque Du Soleil, Audi, Swarovski, and 3D printing companies Shapeways and Materialise she researches how our future would look as we continue to embed technology into what we wear, and more importantly – how this will change our perspective on how we will interface with technology.

Keywords: Marine Robotics, Probability and Statistical Methods, Field Robots
Abstract: Underwater robots are subject to position drift due to the effect of ocean currents and the lack of accurate localisation while submerged. We are interested in exploiting such position drift to estimate the ocean current in the surrounding area, thereby assisting navigation and planning. We present a Gaussian process (GP)-based expectation maximisation (EM) algorithm that estimates the underlying ocean current using sparse GPS data obtained on the surface and dead-reckoned position estimates. We first develop a specialised GP regression scheme that exploits the incompressibility of ocean currents to counteract the underdetermined nature of the problem. We then use the proposed regression scheme in an EM algorithm that estimates the best-fitting ocean current in between each GPS fix. The proposed algorithm is validated in simulation and on a real dataset, and is shown to be capable of reconstructing the underlying ocean current field. We expect to use this algorithm to close the loop between planning and estimation for underwater navigation in unknown ocean currents.

Keywords: Marine Robotics, Localization, Autonomous Vehicle Navigation
Abstract: An adaptive sensing method is presented to control the pinging interval of a downward-looking sonar on an Autonomous Underwater Vehicle. The goal is to conserve energy via adjusting the pinging rate automatically without reducing the localization accuracy when using terrain-aided navigation (TAN). In this paper, the TAN is implemented using a particle filter and a bias velocity estimator developed based on a Kalman filter. The adaptation on the sonar pinging interval is determined based on the depth variation of local seafloor topography which is quantified using a modified Teager Kaiser energy operator. As a result, more measurements are collected on high relief regions, and less measurements are obtained on relatively flat and smooth regions. We evaluated the adaptive sensing method in a simulated environment and applied it to a field data set. The results show that the adaptive sensing method produces an improved navigational accuracy compared to the missions with fixed sonar pinging rates. In the offline field missions, the energy consumed by the altimeter is reduced to about 30% in the adaptive sensing missions compared to continuously sensing missions where the altimeter is pinging consistently without switching off.

Keywords: Marine Robotics, SLAM, Field Robots
Abstract: Synthetic Aperture Sonar (SAS) is a technique to improve the spatial resolution from a moving set of receivers by extending the array in time, increasing the effective array length and aperture. This technique is limited by the accuracy of the receiver position estimates, necessitating highly accurate, typically expensive aided-inertial navigation systems for submerged platforms. We leverage simultaneous localization and mapping to fuse acoustic and navigational measurements and obtain accurate pose estimates even without the benefit of absolute positioning for lengthy underwater missions. We demonstrate a method of formulating the well-known SAS problem in a SLAM framework, using acoustic data from hydrophones to simultaneously estimate platform and beacon position. An empirical probability distribution is computed from a conventional beamformer to correctly account for uncertainty in the acoustic measurements. The non-parametric method relieves the familiar Gaussian-only assumption currently used in the localization and mapping discipline and fits effectively into a factor graph formulation with conventional factors such as ground-truth priors and odometry. We present results from field experiments performed on the Charles River with an autonomous surface vehicle which demonstrate simultaneous localization of an unknown acoustic beacon and vehicle positioning, and provide comparison to GPS ground truths.

Keywords: Marine Robotics, Field Robots, Environment Monitoring and Management
Abstract: Water as a medium poses a number of challenges for robots, limiting the progress of research in underwater robotics vis-a-vis ground or aerial robotics. The primary challenges are satellite based positioning and radio communication being unusable due to high attenuation of electromagnetic waves in water. We have developed miniature, agile, easy to carry and deploy Autonomous Underwater Vehicles (AUVs) equipped with a suite of sensors for underwater environmental sensing. We previously demonstrated adaptive sampling and feature tracking, and gathered data from a lake for limnological research, with the AUV performing inertial navigation. In this paper, we demonstrate a new underwater acoustic positioning system, which allows on-board estimation of AUV position. Our system uses absolute time information from GNSS for initial clock synchronization and uses one-way-travel-time for range measurements, which makes it scalable in the number of robots. It is easily deployable and does not rely on any installed infrastructure in the environment. We describe various hardware and software components of our system, and present results from experiments in Lake Geneva.

Keywords: Marine Robotics, Mapping, SLAM
Abstract: In this paper, we propose a novel approach for underwater simultaneous localization and mapping using a multibeam imaging sonar for 3D terrain mapping tasks. The high levels of noise and the absence of elevation angle information in sonar images present major challenges for data association and accurate 3D mapping. Instead of repeatedly projecting extracted features into Euclidean space, we apply optical flow within bearing-range images for tracking extracted features. To deal with degenerate cases, such as when tracking is interrupted by noise, we model the subsea terrain as a Gaussian Process random field on a Chow–Liu tree. Terrain factors are incorporated into the factor graph, aimed at smoothing the terrain elevation estimate. We demonstrate the performance of our proposed algorithm in a simulated environment, which shows that terrain factors effectively reduce estimation error. We also show ROV experiments performed in a variable-elevation tank environment, where we are able to construct a descriptive and smooth height estimate of the tank bottom.

Keywords: Localization, SLAM, Marine Robotics
Abstract: In this work, we propose a novel method for underwater localization using natural visual landmarks above the water surface. High-accuracy, drift-free pose estimates are necessary for inspection tasks in underwater indoor environments, such as nuclear spent pools. Inaccuracies in robot localization degrade the quality of its obtained map. Our framework uses sparse features obtained via an onboard upward-facing stereo camera to build a global ceiling feature map. However, adopting the pinhole camera model without explicitly modeling light refraction at the water-air interface contributes to a systematic error in observations. Therefore, we use refraction-corrected projection and triangulation functions to obtain true landmark estimates. The SLAM framework jointly optimizes vehicle odometry and point landmarks in a global factor graph using an incremental smoothing and mapping backend. To the best of our knowledge, this is the first method that observes in-air landmarks through water for underwater localization. We evaluate our method via both simulation and real-world experiments in a test-tank environment. The results show accurate localization across various challenging scenarios.

Keywords: Object Detection, Segmentation and Categorization, Computer Vision for Automation, Perception for Grasping and Manipulation
Abstract: In order to solve robotic bin-picking problem in many industrial applications, accurate 6D object pose estimation is one of fundamental technologies. This paper presents a method for accurate 6D pose estimation from a single RGB image for texture-less industrial parts. These objects are common but still challenging to deal with, due to the fact that poor surface texture and brightness makes difficult to compute discriminative local appearance descriptors. The proposed method mainly consists of two stages, which ranges from the detection stage to the optimization stage. Firstly, all known objects in the RGB image are detected with 2D bounding box via a tiny convolutional neural network. Then, the second stage will optimize the 6D pose in the Edge image given several coarse initializations. These coarse initializations are generated from the Edge image via a hypothesis-evaluation scheme. Furthermore, the proposed method is validated by achieving state-of-the-art results of texture-less industrial parts for RGB input. According to practical experiments, the proposed method is accurate and robust enough to be applied on the robotic manipulation platform to complete a simple assembly task.

Keywords: Computer Vision for Automation, Omnidirectional Vision, Localization
Abstract: Pose estimation is one of the most important problems in computer vision. It can be divided in two different categories - absolute and relative - and may involve two different types of camera models: central and non-central. State-of-the-art methods have been designed to solve separately these problems. This paper presents a unified framework that is able to solve any pose problem by alternating optimization techniques between two set of parameters, rotation and translation. In order to make this possible, it is necessary to define an objective function that captures the problem at hand. Since the objective function will depend on the rotation and translation it is not possible to solve it as a simple minimization problem. Hence the use of Alternating Minimization methods, in which the function will be alternatively minimized with respect to the rotation and the translation. We show how to use our framework in three distinct pose problems. Our methods are then benchmarked with both synthetic and real data, showing their better balance between computational time and accuracy.

Keywords: Computer Vision for Automation, Computer Vision for Manufacturing, RGB-D Perception
Abstract: Accurate pose estimation is often a requirement for robust robotic grasping and manipulation of objects placed in cluttered, tight environments, such as a shelf with multiple objects. When deep learning approaches are employed to perform this task, they typically require a large amount of training data. However, obtaining precise 6 degrees of freedom for ground-truth can be prohibitively expensive. This work therefore proposes an architecture and a training process to solve this issue. More precisely, we present a weak object detector that enables localizing objects and estimating their 6D poses in cluttered and occluded scenes. To minimize the human labor required for annotations, the proposed detector is trained with a combination of synthetic and a few weakly annotated real images, for which a human provides only a list of objects present in each image (no time-consuming annotations, such as bounding boxes, segmentation masks and object poses). To close the gap between real and synthetic images, we use multiple domain classifiers trained adversarially. During the inference phase, the resulting class-specific heatmaps of the weak detector are used to guide the search of 6D poses of objects. Our proposed approach is evaluated on several publicly available datasets for pose estimation. The results clearly indicate that this approach could provide an efficient way toward fully automating the training process of computer vision models used in robotics.

Keywords: Kinematics, Motion and Path Planning
Abstract: We present a discrete-optimization technique for finding feasible robot arm trajectories that pass through provided 6-DOF Cartesian-space end-effector paths with high accuracy, a problem called pathwise-inverse kinematics. The output from our method consists of a path function of joint-angles that best follows the provided end-effector path function, given some definition of ``best''. Our method, called Stampede, casts the robot motion translation problem as a discrete-space graph-search problem where the nodes in the graph are individually solved for using non-linear optimization; framing the problem in such a way gives rise to a well-structured graph that affords an effective best path calculation using an efficient dynamic-programming algorithm. We present techniques for sampling configuration space, such as diversity sampling and adaptive sampling, to construct the search-space in the graph. Through an evaluation, we show that our approach performs well in finding smooth, feasible, collision-free robot motions that match the input end-effector trace with very high accuracy, while alternative approaches, such as a state-of-the-art per-frame inverse kinematics solver and a global non-linear trajectory-optimization approach, performed unfavorably.

Keywords: Wearable Robots, Virtual Reality and Interfaces, Haptics and Haptic Interfaces
Abstract: Accurate real-time estimation of the pose and configuration of the human hand attached to a dexterous haptic input device is crucial to improve the interaction possibilities for teleoperation and in virtual and augmented reality. In this paper, we present an approach to reconstruct the pose of the human hand and the joint angles of the fingers when wearing a novel fixed-base (grounded) hand exoskeleton. Using a kinematic model of the human hand built from MRI data, we can reconstruct the hand pose and joint angles without sensors on the human hand, from attachment points on the first three fingers and the palm. We test the accuracy of our approach using motion capture as a ground truth. This reconstruction can be used to determine contact geometry and force-feedback from virtual or remote objects in virtual reality or teleoperation.

Keywords: Computer Vision for Manufacturing, Perception for Grasping and Manipulation, Deep Learning in Robotics and Automation
Abstract: Most of industrial robotic assembly tasks today require fixed initial conditions for successful assembly. These constraints induce high production costs and low adaptability to new tasks. In this work we aim towards flexible and adaptable robotic assembly by using 3D CAD models for all parts to be assembled. We focus on a generic assembly task - the Siemens Innovation Challenge - in which a robot needs to assemble a gear-like mechanism with high precision into an operating system. To obtain the millimeter-accuracy required for this task and industrial settings alike, we use a depth camera mounted near the robot’s end-effector. We present a high-accuracy two-stage pose estimation procedure based on deep convolutional neural networks, which includes detection, pose estimation, refinement, and handling of near- and full symmetries of parts. The networks are trained on simulated depth images with means to ensure successful transfer to the real robot. We obtain an average pose estimation error of 2.16 millimeters and 0.64 degree leading to 91% success rate for robotic assembly of randomly distributed parts. To the best of our knowledge, this is the first time that the Siemens Innovation Challenge is fully addressed, with all the parts assembled with high success rates.

Keywords: Localization, SLAM, Visual-Based Navigation
Abstract: Extending our recent work that focuses on the observability analysis of aided inertial navigation systems (INS) using homogeneous geometric features including points, lines and planes, in this paper, we complete the analysis for the general aided INS using different combinations of geometric features (i.e., points, lines and planes). We analytically show that the linearized aided INS with different feature combinations generally possess the same observability properties as those with same features, i.e., 4 unobservable directions, corresponding to the global yaw rotation and the global position of the sensor platform. During the analysis, we particularly propose a novel minimal representation of line features, i.e., the ``closest point'' parameterization, which uses a 4D Euclidean vector to describe a line and is proved to preserve the same observability properties. Based on that, for the first time, we provide two sets of unified representations for points, lines and planes, i.e., the quaternion form and the closest point (CP) form, and perform extensive observability analysis with analytically-computed Jacobians for these unified parameterizations. We validate the proposed CP representations and observability analysis with Monte-Carlo simulations, in which EKF-based vision-aided INS (VINS) with combinations of geometrical features in CP form are developed and compared.

Keywords: Localization, SLAM, Mapping
Abstract: Enabling real-time visual-inertial navigation in unknown environments while achieving bounded-error performance holds great potentials in robotic applications. To this end, in this paper, we propose a novel linear-complexity EKF for visual-inertial localization, which can efficiently utilize loop closure constraints, thus allowing for long-term persistent navigation. The key idea is to adapt the Schmidt-Kalman formulation within the multi-state constraint Kalman filter (MSCKF) framework, in which we selectively include keyframes as nuisance parameters in the state vector for loop closures but do not update their estimates and covariance in order to save computations while still tracking their cross-correlations with the current navigation states. As a result, the proposed Schmidt-MSCKF has only O(n) computational complexity while still incorporating loop closures into the system. The proposed approach is validated extensively on large-scale real-world experiments, showing significant performance improvements when compared to the standard MSCKF, while only incurring marginal computational overhead.

Keywords: Localization, SLAM, Failure Detection and Recovery
Abstract: In this paper, we present a real-time multi-IMU visual inertial navigation system (mi-VINS) that utilizes the information from multiple inertial measurement units (IMUs) and thus is resilient to IMU sensor failures. In particular, in the proposed mi-VINS formulation, one of the IMUs serves as the "base" of the system, while the rest act as auxiliary sensors aiding in state estimation. A key advantage of this architecture is the ability to seamlessly "promote" an auxiliary IMU as a new base, for example, upon detection of the base IMU failure, thus being resilient to the single point of sensor failure as seen in conventional VINS. Moreover, in order to properly fuse the information of multiple IMUs, both the spatial (relative pose) and temporal (time offset) calibration parameters between each sensor and the base IMU are estimated online. The proposed mi-VINS with online spatial and temporal calibration is validated in both simulations and real-world experiments, and is shown to be able to provide accurate localization and calibration even in scenarios with IMU sensor failures.

Keywords: Localization, Visual Learning, Deep Learning in Robotics and Automation
Abstract: Inspired by the cognitive process of humans and animals, Curriculum Learning (CL) trains a model by gradually increasing the difficulty of the training data. In this paper, we study whether CL can be applied to complex geometry problems like estimating monocular Visual Odometry (VO). Unlike existing CL approaches, we present a novel CL strategy for learning the geometry of monocular VO by gradually making the learning objective more difficult during training. To this end, we propose a novel geometry-aware objective function by jointly optimizing relative and composite transformations over small windows via bounded pose regression loss. A cascade optical flow network followed by recurrent network with a differentiable windowed composition layer, termed CL-VO, is devised to learn the proposed objective. Evaluation on three real-world datasets shows superior performance of CL-VO over state-of-the-art feature-based and learning-based VO.

Keywords: Localization, SLAM, Sensor Fusion
Abstract: This paper focuses on the localization and mapping problem on ground vehicles using odometric and monocular visual sensors. To improve the accuracy of vision based estimation on ground vehicles, researchers have exploited the constraint of approximately planar motion, and usually implemented it as a stochastic constraint on an SE(3) pose. In this paper, we propose a simpler algorithm that directly parameterizes the ground vehicle poses on SE(2). The out-of-SE(2) motion perturbations are not neglected, but incorporated into an integrated noise term of a novel SE(2)-XYZ constraint, which associates an SE(2) pose and a 3D landmark via the image feature measurement. For odometric measurement processing, we also propose an efficient preintegration algorithm on SE(2). Utilizing these constraints, a complete visual-odometric localization and mapping system is developed, in a commonly used graph optimization structure. Its superior performance in accuracy and robustness is validated by real-world experiments in industrial indoor environments.

Keywords: Localization, Sensor Fusion, Field Robots
Abstract: This paper proposes an approach for fusing direct radiometric data from a thermal camera with inertial measurements to extend the robotic capabilities of aerial robots for navigation in GPS-denied and visually degraded environments in the conditions of darkness and in the presence of air-borne obscurants such as dust,fog and smoke. An optimization based approach is developed that jointly minimizes the re-projection error of 3D landmarks and inertial measurement errors. The developed solution is extensively verified against both ground-truth in an indoor laboratory setting, as well as inside an underground mine under severely visually degraded conditions.

Keywords: Space Robotics and Automation, Flexible Robots, Dynamics
Abstract: The parameter estimation of space manipulator systems on orbit is studied, whose manipulators are subject to joint flexibilities. To improve path planning and tracking capabilities, advanced control strategies that benefit from the knowledge of system parameters are required. These parameters include the system inertial parameters as well as the stiffness and damping parameters, which describe joint flexibilities. During operation some of these parameters may change or be unknown. Estimation methods based on the equations of motion are sensitive to noise, while methods based on the angular momentum conservation, while they are tolerant to noise, they cannot estimate the parameters that describe joint flexibilities. A parameter estimation method, based on the energy balance, applied during the motion of a space flexible-joint manipulator system in the free-floating mode, is developed. The method is tolerant to noise and can reconstruct the system full dynamics. It is shown that the parameters estimated by the proposed method can describe the system dynamics fully. The application of the developed method is valid for spatial systems; it is illustrated by a planar 7 degrees of freedom (DoF) example system.

Keywords: Space Robotics and Automation, Motion Control, Dynamics
Abstract: This paper addresses the coordinated control of the spacecraft's attitude and the end-effector pose of a manipulator-equipped space robot. A controller is proposed to simultaneously regulate the spacecraft's attitude, the global center-of-mass (CoM), and the end-effector pose. The control is based on a triangular actuation decomposition that decouples the end-effector task from the spacecraft's force actuator, increasing fuel efficiency. The strategy is validated in hardware using a robotic motion simulator composed of a seven degrees-of-freedom (DOF) arm mounted on a 6DOF base. The trade-off between control requirements and fuel consumption is discussed.

Keywords: Space Robotics and Automation, Motion Control, Compliance and Impedance Control
Abstract: We introduce the concept of Momentum Driven Structures (MDS) made of inertially actuated units linked together by compliant elements as a potential solution for rough environments exploration. We propose a control method for MDS based on the bio-inspired concept of Central Pattern Generator (CPG) and study in simulation the impact of compliance distribution on locomotion performance using population based optimization techniques. Our results suggest that compliant structures outperform their rigid counterparts in terms of distance traveled. In addition, we show that co-evolved structures are only performing marginally better than their control-only optimized equivalent, highlighting the fact that compliance modulation may not be a significant asset in such experiment, considering the related hardware complexity it introduces.

Keywords: Space Robotics and Automation, Deep Learning in Robotics and Automation
Abstract: Planetary rovers, such as those currently on Mars, face difficult path planning problems, both before landing during the mission planning stages as well as once on the ground. In this work we present a new approach to these planning problems based on inverse reinforcement learning (IRL) using deep convolutional networks and value iteration networks as important internal structures. Value iteration networks are an approximation of the value iteration (VI) algorithm implemented with convolutional neural networks to make VI fully differentiable. We propose a modification to the value iteration recurrence, referred to as the soft value iteration network (SVIN). SVIN is designed to produce more effective training gradients through the value iteration network. It relies on an internal soft policy model, where the policy is represented with a probability distribution over all possible actions, rather than a deterministic policy that returns only the best action. We demonstrate the effectiveness of our proposed architecture in both a grid world dataset as well as a highly realistic synthetic dataset generated from currently deployed rover mission planning tools and real Mars imagery.

Keywords: Space Robotics and Automation, Contact Modeling, Grasping
Abstract: This paper presents a contact-event-triggered filter using only a force-torque sensor with impedance control for non-cooperative, rotating, heavy object capture. Contact events are modeled for prediction, and detected to trigger the particle filter’s updating process. By combining these features, a computationally efficient, contact-event-triggered filter is proposed. For our purpose of capture using impedance control, expected contact events, collisions and sliding are defined for prediction and detection. This novel method is implemented in an air bearing robotic system, and has demonstrated its superiority with the highest success rate (100%) for sliding contact mode cases, whereas the previous method could only yield a success rate of 87.9%. The computation resource is demonstrated to be limited, with a computation time of 4.2 milliseconds on average and 8.3 milliseconds at worst.

Keywords: Space Robotics and Automation, Multi-Robot Systems
Abstract: In this work we design a novel control strategy for a space manipulator operating on a controlled base. The proposed controllers resolve the tracking of the end-effector and the regulation of the base. In particular, we focus on the effects due to the different sampling rates of the manipulator and the base controllers which can generate stability issues. These effects are analysed from an energetic perspective and passivity-based controllers are designed for the base and the manipulator to avoid instability. The method is validated with simulations and experiments on a robotic facility, the OOS-Sim.

Keywords: Deep Learning in Robotics and Automation, Grasping, Sensor-based Control
Abstract: Grasping objects under uncertainty remains an open problem in robotics research. This uncertainty is often due to noisy or partial observations of the object pose or shape. To enable a robot to react appropriately to unforeseen effects, it is crucial that it continuously takes sensor feedback into account. While visual feedback is important for inferring a grasp pose and reaching for an object, contact feedback offers valuable information during manipulation and grasp acquisition. In this paper, we use model-free deep reinforcement learning to synthesize control policies that exploit contact sensing to generate robust grasping under uncertainty. We demonstrate our approach on a multi-fingered hand that exhibits more complex finger coordination than the commonly used two- fingered grippers. We conduct extensive experiments in order to assess the performance of the learned policies, with and without contact sensing. While it is possible to learn grasping policies without contact sensing, our results suggest that contact feedback allows for a significant improvement of grasping robustness under object pose uncertainty and for objects with a complex shape.

Keywords: Deep Learning in Robotics and Automation, Force and Tactile Sensing, Model Learning for Control
Abstract: To achieve a dexterous robotic manipulation, we need to endow our robot with tactile feedback capability, i.e. the ability to drive action based on tactile sensing. In this paper we specifically address the challenge of tactile servoing, i.e. given the current tactile sensing and a target/goal tactile sensing --- memorized from a successful task execution in the past --- what is the action that will bring the current tactile sensing to move closer towards the target tactile sensing at the next time step. We develop a data-driven approach to acquire a dynamics model for tactile servoing by learning from demonstration. Moreover, our method represents the tactile sensing information as to lie on a surface --- or a 2D manifold --- and perform a manifold learning, making it applicable to any tactile skin geometry. As a proof of concept, we evaluate our method on a robot equipped with a tactile finger.

Keywords: Deep Learning in Robotics and Automation, Grasping, Perception for Grasping and Manipulation
Abstract: In this paper, we propose an end-to-end grasp evaluation model to address the challenging problem of localizing robot grasp configurations directly from the point cloud. Compared to recent grasp evaluation metrics that are based on handcrafted depth features and a convolutional neural network (CNN), our proposed PointNetGPD is light-weighted and can directly process the 3D point cloud that locates within the gripper for grasp evaluation. Taking the raw point cloud as input, our proposed grasp evaluation network can capture the complex geometric structure of the contact area between the gripper and the object even if the point cloud is very sparse. To further improve our proposed model, we generate a larger-scale grasp dataset with 350k real point cloud and grasps with YCB object set for training. The performance of the proposed model is quantitatively measured both in simulation and on robotic hardware. Experiments on object grasping and clutter removal show that our proposed model generalizes well to novel objects and outperforms state-of-the-art methods.

Keywords: Deep Learning in Robotics and Automation, Dexterous Manipulation, Learning from Demonstration
Abstract: Multi-fingered dexterous hands are versatile and capable of acquiring a diverse set of skills such as grasping, in-hand manipulation, and tool use. To fully utilize their versatility in real-world scenarios, we require algorithms and policies that can control them using on-board sensing capabilities, without relying on external tracking or motion capture systems. Cameras and tactile sensors are the most widely used on-board sensors that do not require instrumentation of the world. In this work, we demonstrate an imitation learning based approach to train deep visuomotor policies for a variety of manipulation tasks with a simulated five fingered dexterous hand. These policies directly control the hand using high dimensional visual observations of the world and propreoceptive observations from the robot, and can be trained efficiently with a few hundred expert demonstration trajectories. We also find that using touch sensing information enables faster learning and better asymptotic performance for tasks with high degree of occlusions.

Keywords: Deep Learning in Robotics and Automation, Force and Tactile Sensing
Abstract: Much of the literature on robotic perception focuses on the visual modality. Vision provides a global observation of a scene, making it broadly useful. However, in the domain of robotic manipulation, vision alone can sometimes prove inadequate: in the presence of occlusions or poor lighting, visual object identification might be difficult. The sense of touch can provide robots with an alternative mechanism for recognizing objects. In this paper, we study the problem of touch-based instance recognition. We propose a novel framing of the problem as multi-modal recognition: the goal of our system is to recognize, given a visual and tactile observation, whether or not these observations correspond to the same object. To our knowledge, our work is the first to address this type of multi-modal instance recognition problem on such a large-scale with our analysis spanning 98 different objects. We employ a robot equipped with two GelSight touch sensors, one on each finger, and a self-supervised, autonomous data collection procedure to collect a dataset of tactile observations and images. Our experimental results show that it is possible to accurately recognize object instances by touch alone, including instances of novel objects that were never seen during training. Our learned model outperforms other methods on this complex task, including that of human volunteers.

Keywords: Deep Learning in Robotics and Automation, Dexterous Manipulation
Abstract: Dexterous multi-fingered robotic hands can perform a wide range of manipulation skills, making them an appealing component for general-purpose robotic manipulators. However, such hands pose a major challenge for autonomous control, due to the high dimensionality of their configuration space and complex intermittent contact interactions. In this work, we propose deep reinforcement learning (deep RL) as a scalable solution for learning complex, contact rich behaviors with multi-fingered hands. Deep RL provides an end-to-end approach to directly map sensor readings to actions, without the need for task specific models or policy classes. We show that contact-rich manipulation behavior with multi-fingered hands can be learned by directly training with model-free deep reinforcement learning algorithms in the real world, with minimal additional assumption and without the aid of simulation. We learn a variety of complex behaviors on two different low-cost hardware platforms. We show that each task can be learned entirely from scratch, and further study how the learning process can be further accelerated by using a small number of human demonstrations to bootstrap learning. Our experiments demonstrate that complex multi-fingered manipulation skills can be learned in the real world in about 3-5 hours, and that demonstrations can decrease this to 2-3 hours, indicating that direct deep RL training in the real world is a viable and practical alternative to simulation and model-based contr

Keywords: Product Design, Development and Prototyping, Flexible Robots, Biomimetics
Abstract: Soft-bodied robots are getting attention from researchers as their potential in designing compliant and adaptive robots. However, soft-bodied robots also pose many challenges not only in non-linear controlling but also in design and fabrication. Especially, the non-compatibility between soft materials and rigid sensors/actuators makes it more difficult to design a fully compliant soft-bodied robot. In this paper, we propose an all-printed sensor and actuator for designing soft-bodied robots by printing silver nano-particle ink on top of a flexible plastic film. We can print bending sensors and thermal based actuators instantly with home-commodity inkjet printers without any pre/post-processing. We exemplify the application of this fabrication method with an all-printed paper caterpillar robots which can inch forward and sense its body bending angle.

Keywords: Product Design, Development and Prototyping, Grippers and Other End-Effectors, Mechanism Design
Abstract: The main task of robotic grippers, holding an object, does not require work theoretically. Yet grippers consume significant amounts of energy in practice. This paper presents an approach for designing an energy-saving drive for robotic grippers employing a Statically Balanced Force Amplifier (SBFA) and a Non-backdrivable mechanism (NBDM). A novel metric (Grip Performance Metric) to systematically evaluate drives regarding their energy consumption, is used in the design phase; afterwards, the realization and testing of a prototype (REED, Robotic Energy-Efficient Drive) are presented. Results show that the actuation force can be reduced by 92%, resulting in energy-savings of 86% for an example task. This shows the potential of drives based on SBFAs and NBDMs to achieve energy-neutral grippers.

Keywords: Simulation and Animation, Computational Geometry, Additive Manufacturing
Abstract: Procedural generation of elastic structures provides the fundamental basis for controlling and designing 3D printed deformable object behaviors. The automation through generative algorithms provides flexibility in how design and functionality can be seamlessly integrated into a cohesive process that generates 3D prints with variable elasticity. Generative deformation introduces an automated method for perforating existing volumetric structures, promoting simulated deformations, and integrating stress analysis into a cohesive pipeline model that can be used with existing consumer-level 3D printers with elastic material capabilities. In this work, we present a consolidated implementation of the design, simulate, refine, and 3D print procedure based on the automated generation of heterogeneous lattice structures. We utilize Finite Element Analysis (FEA) metrics to generate perforated deformation models that adhere to deformation behaviors created within our design environment. We present the core algorithms, automated pipeline, and 3D print deformations of various objects. Quantitative results illustrate how the heterogeneous geometric structure can influence elastic material behaviors towards design objectives. Our method provides an automated open-source tool for quickly prototyping elastic 3D prints.

Keywords: Education Robotics, Product Design, Development and Prototyping
Abstract: This paper introduces the KUKA Robot Learning Lab at KIT - a remotely accessible robotics testbed. The motivation behind the laboratory is to make state-of-the-art industrial lightweight robots more accessible for education and research. Such expensive hardware is usually not available to students or less privileged researchers to conduct experiments. This paper describes the design and operation of the Robot Learning Lab and discusses the challenges that one faces when making experimental robot cells remotely accessible. Especially safety and security must be ensured, while giving users as much freedom as possible when developing programs to control the robots. A fully automated and efficient processing pipeline for experiments makes the lab suitable for a large amount of users and allows a high usage rate of the robots.

Keywords: Agricultural Automation, Product Design, Development and Prototyping, Object Detection, Segmentation and Categorization
Abstract: This paper presents a prototype of an automated seedling height assessment system for tree nurseries. The proposed system can acquire and store real-time 3D point-cloud data of seedlings; and perform offline identification, measurement, and report generation of seedling heights with an overall system accuracy that meets a 5,mathrm{mm} accuracy specification. Periodic growth information of seedlings allows quantifying effects of different factors on the overall seedling development process for research and production optimization purposes. However, current manual sampling approaches used at these facilities produce quite limited data samples, and the process is rather time-consuming and labor-intensive for industrial scale operations. In contrast, the proposed system is capable of significantly increasing the measurement sample size, measurement resolution, and frequency of measurement by automating the seedling measurement process using a scanning laser profilometer and an application specific point-cloud processing algorithm. The performance of the proposed profilometry solution for point-cloud generation is compared with several other point-cloud generation methods such as a 3D structured light sensing, light intensity detection and ranging (LiDAR), stereovision, and photogrammetry. These comparison results demonstrate a superior performance of the laser-profilometer over other sensing solutions available for seedling height measurement. The proposed system is experimen

Keywords: Product Design, Development and Prototyping, Micro/Nano Robots, Soft Material Robotics
Abstract: We propose a novel adsorption pad for wall climbing robot and irregular surface object using capillary force and water sealing. We call this adsorption pad as Super Wet Adsorption pad. The SWA pad has a porous part and a capillary part. The porous part is made by salt reaching method. When the SWA pad adsorbs to the wall which some sand and dust are attached, water comes from the porous part to avoid vacuum breaking. The capillary part is connected to the porous part to supply and stock the water. In this paper, we show the design of the porous part and the capillary part, fabrication process of each parts, and perform the evaluation experiment of the capillary force and adsorption of uneven surfaces, demonstration of wall climbing robot and adsorption of irregular surface foods.

Keywords: Legged Robots, Humanoid and Bipedal Locomotion, Biomimetics
Abstract: In this paper, we investigated how the stiffness and damping properties of soft robotic feet affect the stability and energetic economy of bipedal robotic walking. To this end, we manufactured four different spherical feet from the following materials: hollow rubber, Sorbothane, Norsorex, and Neoprene. The materials were specifically chosen to cover a wide range of stiffness and damping values. The impact response of each design was first characterized in a drop test rig. We then evaluated the performance of each foot in an extensive series of walking experiments on the planar bipedal robot RAMone. Our results showed that, at low speeds, the feet with lower damping had a smaller energy cost of walking, possibly due to greater return of mechanical energy at lift-off. However, at speeds above 0.5 m/s, the feet with lower damping started to exhibit a bouncing behaviour which led to higher walking instability and increased the energy cost of walking. Additionally, we found the feet with lower stiffness to be more economical across all walking speeds. Our results provide insight into the role of foot properties in bipedal walking and may help with the design of walking robots.

Keywords: Legged Robots, Humanoid and Bipedal Locomotion, Optimization and Optimal Control
Abstract: Dynamic bipedal robot locomotion has achieved remarkable success due in part to recent advances in trajectory generation and nonlinear control for stabilization. A key assumption utilized in both theory and experiments is that the robot's stance foot always makes no-slip contact with the ground, including at impacts. This assumption breaks down on slippery low-friction surfaces, as commonly encountered in outdoor terrains, leading to failure and loss of stability. In this work, we extend the theoretical analysis and trajectory optimization to account for stick-slip transitions at point foot contact using Coulomb's friction law. Using AMBER-3M planar biped robot as an experimental platform, we demonstrate for the first time a slippery walking gait which can be stabilized successfully both on a lubricated surface and on a rough no-slip surface. We also study the influence of foot slippage on reducing the mechanical cost of transport, and compare energy efficiency in both numerical simulation and experimental measurement.

Keywords: Legged Robots, Humanoid and Bipedal Locomotion, Humanoid Robots
Abstract: Jumping can be an effective way of locomotion to overcome small terrain gaps or obstacles. In this paper we propose two different approaches to perform jumps with a humanoid robot. Specifically, starting from a pre-defined CoM trajectory we develop the theory for a velocity controller and for a torque controller based on an optimization technique for the evaluation of the joints input. The controllers have been tested both in simulation and on the humanoid robot iCub. In simulation the robot was able to jump using both controllers, while the real system jumped with the velocity controller only. The results highlight the importance of controlling the centroidal angular momentum and they suggest that the joint performances, namely maximum power, of the legs and torso joints, and the low level control performances are fundamental to achieve acceptable results.

Keywords: Robust/Adaptive Control of Robotic Systems, Humanoid and Bipedal Locomotion, Legged Robots
Abstract: Complex robot motions are frequently generated by composing simpler primitive movements. We use this approach to formulate robot motion plans as sequences of primitives to be executed one after the other. When dealing with dynamical movement primitives, besides accomplishing the high-level objective, planners must also reason about the effect of the plan's execution on the safety of the platform. This task is exacerbated by the presence of disturbances, such as non-vanishing external forces. To address this issue, we present a framework that builds on rigorous control-theoretic tools to generate safely executable motion plans for externally excited robotic systems. We illustrate the proposed framework on adapting the motion of a 3D bipedal robot model to persistent external forcing by switching among dynamic movement primitives, each corresponding to a limit-cycle walking gait.

Keywords: Humanoid Robots, Humanoid and Bipedal Locomotion
Abstract: Balancing is a critical feature for a robot interacting with an unstructured environment. The balancing control should account for unknown perturbation forces that might destabilize the robot when performing the intended tasks. In the case of humanoid robots, this challenge is higher, due to the inherent difficulties of balancing a robot on two legs resulting in a rather small footprint. Approaches for enabling a good balancing behavior on humanoid robots traditionally rely on whole-body balancing approaches. This paper extends a passivity-based whole-body balancing framework to guarantee the equilibrium of a humanoid robot while performing different interaction tasks where the (high) task forces acting on the robot are difficult to foresee. Instead of controlling the center of mass, the proposed controller directly uses information from the Gravito-Inertial Wrench Cone to guarantee the feasibility of the balancing forces. The performance of the approach is validated in a number of successful experimental tests.

Keywords: Humanoid and Bipedal Locomotion, Telerobotics and Teleoperation, Haptics and Haptic Interfaces
Abstract: This paper presents a method to achieve human and legged robot dynamic synchronization through bilateral feedback teleoperation. Our study shows how we can explore the interplay between human Extrapolated Center ofMass and the contact forces with the environment in order to transmit to the robot the underlying balancing and stepping strategy. All the necessary key equations for the frontal plane coupled dynamics are presented along with the human feedback law derived from the proposed state normalization in length and time. Here, we pay special attention to how the natural frequency of each system influences the resulting motion and analyze how the coupled system responds to various robot sizes. Experiments in which a human operator controls a simulated bipedal robot show how the Balance Feedback Interface force varies according to different scales and responds to external disturbances. Finally, we show the method’s robustness to uneven terrain and how we can allow the point feet robot to synchronously take steps with the operator. This is an introductory study that aims to grant legged robotsmotor capabilities for power manipulation comparable to humans.

Keywords: Physical Human-Robot Interaction, Mobile Manipulation, Sensor Fusion
Abstract: Service robots are expected to autonomously and efficiently work in human-centric environments. For this type of robots, object perception and manipulation are challenging tasks due to need for accurate and real-time response. This paper presents an interactive open-ended learning approach to recognize multiple objects and their grasp affordances concurrently. This is an important contribution in the field of service robots since no matter how extensive the training data used for batch learning, a robot might always be confronted with an unknown object when operating in human-centric environments. The paper describes the system architecture and the learning and recognition capabilities. Grasp learning associates grasp configurations (i.e., end-effector end-effector positions and orientations) to grasp affordance categories. The grasp affordance category and the grasp configuration are taught through verbal and kinesthetic teaching, respectively. A Bayesian approach is adopted for learning and recognition of object categories and an instance-based approach is used for learning and recognition of affordance categories. An extensive set of experiments has been performed to assess the performance of the proposed approach regarding recognition accuracy, scalability and grasp success rate on challenging datasets and real-world scenarios.

Keywords: Physical Human-Robot Interaction, Compliance and Impedance Control
Abstract: This paper presents a novel sensing approach for the physical interaction between a human user and a serial robotic arm. The approach is inspired from the concept of macro-mini robot architecture. The framework is developed for a general multi-degree-of-freedom serial robot and a corresponding impedance control scheme is proposed. In order to illustrate the concept, a five-degree-of-freedom robotic arm was built as well as a six-degree-of-freedom low-impedance sensing device that is used to control the robot. Experimental results are provided.

Keywords: Physical Human-Robot Interaction, Collision Avoidance, Sensor-based Control
Abstract: Human-robot collaboration in industrial settings calls for implementing safety measures to ensure there is no risk to humans working in such an environment. In human-robot physical collaboration, an object or a load is handled by both human and the robot. Developing a safety framework for the robot is a requirement for preventing collisions during performing a task. In this paper, force myography (FMG) data are used to develop a control scheme for the robot such that it can work with the human worker while avoiding collisions. Force myography quantifies the activities of human muscles when applying force to handle an object. A neural network-based approach is then used to select the most informative features of the FMG signal. The proposed control scheme then incorporates the FMG data and the robot dynamics to obtain a prediction about the next step of the cooperation task and to plan the robot motion accordingly. The proposed approach is evaluated experimentally in real time in a moving objects task which requires appropriate complementary actions from the robot and the human user. The results of this study show that the proposed scheme can successfully plan the robot motion based on the actions of the human user.

Keywords: Physical Human-Robot Interaction, Force and Tactile Sensing, Robot Safety
Abstract: In the future, human-robot interaction will include collaboration in close-quarters where the environment geometry is partially unknown. As a means for enabling such interaction, this paper presents a multi-modal sensor array capable of contact detection and localization, force sensing, proximity sensing, and mapping. The sensor array integrates Hall effect and time-of-flight (ToF) sensors in an I²C communication network. The design, fabrication, and characterization of the sensor array for a future in-situ collaborative continuum robot are presented. Possible perception benefits of the sensor array are demonstrated for accidental contact detection, mapping of the environment, selection of admissible zones for bracing, and constrained motion control of the end effector while maintaining a bracing constraint with an admissible rolling motion.

Keywords: Physical Human-Robot Interaction, Compliance and Impedance Control, Hydraulic/Pneumatic Actuators
Abstract: This paper proposes an intrinsically backdrivable electro-hydraulic torque actuator (EHTA) and associated control algorithms that could lead to the design of a high-performance interactive robot system. The EHTA consists of (electro-hydraulic) backdrivable servovalves and double vane rotary hydraulic actuators. It is represented as a flexible actuator model considering servovalve dynamics and fluid dynamics. In the flexible actuator structure, the interaction stability is not guaranteed because the EHTA is a non-collocated system. However, due to the use of backdrivable servovalves, the EHTA is rigidly represented in the finite frequency region that a robot operates in daily life; it has the property of finite frequency passivity. Based on this property, a disturbance observer (DOB) was designed to compensate for friction effects and model uncertainties. Then, the compliance control was applied to the friction-free actuator. Consequently, we achieved the actuator with a torque sensitivity of 0.1 Nm and a maximum torque output of approximately 100 Nm. The proposed actuators and controllers were evaluated through experiments.

Keywords: Physical Human-Robot Interaction, Dexterous Manipulation, Biologically-Inspired Robots
Abstract: This study examined strategies humans chose to manipulate an object with complex (nonlinear, underactuated) dynamics, such as liquid sloshing in a cup of coffee. The problem was simplified to the well-known cart-and-pendulum system moving on a horizontal line. This model was implemented in a virtual environment and human subjects manipulated the object via a robotic manipulandum. The task was to maneuver the system from rest to arrive at a target position such that no residual oscillations of the pendulum bob remained. Our goal was to test whether humans simplified control by employing dynamic primitives, specifically submovements. Experimental velocity profiles of the human movements were compared to those predicted by three different control models. Two models used continuous optimization-based control, the third control model was based on Input Shaping. Input Shaping is a method for controlling flexible objects by convolving a motion profile with impulses of appropriate amplitude and timing. To evaluate whether humans used Input Shaping, we decomposed the velocity profiles recorded from humans into submovements, as proxies for the convolved impulses. Comparing the motion profiles from the 3 models with the experimentally measured human profiles showed superior performance of the Input Shaping model. These initial results are consistent with our hypothesis that combining dynamic primitives, submovements, is a competent description of human performance and may provide a simp

Keywords: Perception for Grasping and Manipulation, Compliant Assembly
Abstract: This paper introduces a Bayesian state estimator for contact-rich manipulation tasks with application in non-prehensile manipulation, industrial assembly or in-hand localization. The core idea of our approach is to explicitly model both the contact dynamics and a torque-based robot controller as part of the underlying system model. Our approach is capable of estimating the state of movable objects for various robot kinematics and geometries of robots and objects. This includes complex scenarios with multiple robots, multiple objects and articulated objects. We have validated our approach in simulation and on a physical robot. The experiments show that multi-modal distributions of six degrees of freedom object poses can be accurately tracked in real-time in a complex manipulation scenario.

Keywords: Perception for Grasping and Manipulation, Reactive and Sensor-Based Planning, Sensor-based Control
Abstract: We describe a single fingertip-mounted sensing sys- tem for robot manipulation that provides proximity (pre-touch), contact detection (touch), and force sensing (post-touch). The sen- sor system consists of optical time-of-flight range measurement modules covered in a clear elastomer. Because the elastomer is clear, the sensor can detect and range nearby objects, as well as measure deformations caused by objects that are in contact with the sensor and thereby estimate the applied force. We examine how this sensor design can be improved with respect to invariance to object reflectivity, signal-to-noise ratio, and continuous operation when switching between the distance and force measurement regimes. By harnessing time-of-flight technology and optimizing the elastomer-air boundary to control the emitted light’s path, we develop a sensor that is able to seamlessly transition between measuring distances of up to 50 mm and contact forces of up to 10 newtons. We demonstrate that our sensor improves manipulation accuracy in a block unstacking task. Thorough instructions for manufacturing the sensor from inexpensive, commercially available components are provided, as well as all relevant hardware design files and software sources.

Keywords: Perception for Grasping and Manipulation, Force and Tactile Sensing
Abstract: Humans can form an impression of how a new object feels simply by touching its surfaces with the densely innervated skin of the fingertips. Many haptics researchers have recently been working to endow robots with similar levels of haptic intelligence, but these efforts almost always employ hand-crafted features, which are brittle, and concrete tasks, such as object recognition. We applied unsupervised feature learning methods, specifically K-SVD and Spatio-Temporal Hierarchical Matching Pursuit (ST-HMP), to rich multi-modal haptic data from a diverse dataset. We then tested the learned features on 19 more abstract binary classification tasks that center on haptic adjectives such as smooth and squishy. The learned features proved superior to traditional hand-crafted features by a large margin, almost doubling the average F₁ score across all adjectives. Additionally, particular exploratory procedures (EPs) and sensor channels were found to support perception of certain haptic adjectives, underlining the need for diverse interactions and multi-modal haptic data.

Keywords: Perception for Grasping and Manipulation, Force and Tactile Sensing, Dexterous Manipulation
Abstract: This work studies the problem of shape reconstruction and object localization using a vision-based tactile sensor, GelSlim.

The main contributions are the recovery of local shapes from contact, an approach to reconstruct the tactile shape of objects from tactile imprints, and an accurate method for object localization of previously reconstructed objects. The algorithms can be applied to a large variety of 3D objects and provide accurate tactile feedback for in-hand manipulation.

Results show that by exploiting the dense tactile information we can reconstruct the shape of objects with high accuracy and do on-line object identification and localization, opening the door to reactive manipulation guided by tactile sensing. We provide videos and supplemental information in the project's website web.mit.edu/mcube/research/tactile_localization.html .

Keywords: Perception for Grasping and Manipulation, Sensor-based Control, Grasping
Abstract: In this paper, we propose an approach to detect incipient slip, i.e. predict slip, by using a high-resolution vision-based tactile sensor, GelSlim. The sensor dynamically captures the tactile imprints of the grasped object and their changes with a soft gel pad. The method assumes the object is mostly rigid and expects the motion of object's imprint on the sensor surface to be a 2D rigid-body motion. We use the deviation of the true motion field from that of a 2D planar rigid transformation as a measure of slip. The output is a dense slip field which we monitor in real time to detect when small areas of the contact patch start to slip (incipient slip). The method can detect incipient slip in any direction without any prior knowledge of the object at 24 Hz. We test the method on 10 objects for 240 times and achieve 86.25% detection accuracy with the vast majority of failure cases occurring when grasping highly deformable objects. We further show how the slip feedback can be used to adjust the gripping force to avoid slip with a closed-loop bottle-cap screwing and unscrewing experiment. The method can be used to enable many manipulation tasks in both structured and unstructured environments.

Keywords: Force and Tactile Sensing, Deep Learning in Robotics and Automation
Abstract: Deep learning has the potential to have the impact on robot touch that it has had on robot vision. Optical tactile sensors act as a bridge between the subjects by allowing techniques from vision to be applied to touch. In this paper, we apply deep learning to an optical biomimetic tactile sensor, the TacTip, which images an array of papillae (pins) inside its sensing surface analogous to structures within human skin. {color{black}Our main result is that the application of a deep CNN can give reliable edge perception and thus a robust policy for planning contact points to move around object contours.} Robustness is demonstrated over several irregular and compliant objects with both tapping and continuous sliding, using a model trained only by tapping onto a disk. These results relied on using techniques to encourage generalization to tasks beyond which the model was trained. We expect this is a generic problem in practical applications of tactile sensing that deep learning will solve.

Keywords: Intelligent Transportation Systems, Sensor Fusion
Abstract: In this paper, we propose a road detection method with LiDAR-camera fusion in a novel conditional random field (CRF) framework to exploit both range and color information. In the LiDAR based part, a fast height-difference based scanning strategy is applied in the 2D LiDAR range-image domain and a dense road detection result in camera image domain can be obtained through geometric upsampling given the LiDAR-camera calibration parameters. In the camera based part, a fully convolutional network is applied in the camera image domain. Finally, we fuse the dense and binary road detection results from both LiDAR and camera in a single CRF framework. Experiments show that using a single thread of CPU, the proposed LiDAR based part can operate at a frequency of over 250Hz with sparse output in range image and 40Hz with dense result in camera image for the 64-beam Velodyne scanner. Our CRF fusion method achieves competitive performance on the KITTI-Road dataset.

Keywords: Intelligent Transportation Systems, Autonomous Vehicle Navigation
Abstract: In this work a graph-based, semantic mapping approach for indoor robotics applications is presented, which is extending OpenStreetMap (OSM) with robotic-specific, semantic, topological, and geometrical information. Models are introduced for basic indoor structures such as walls, doors, corridors, elevators, etc. The architectural principles support composition with additional domain and application-specific knowledge. As an example, a model for an area is introduced, and it is explained how this can be used in navigation. A key advantage of the proposed graph-based map representation is that it allows exploiting the hierarchical structure of the graphs. Finally, the compatibility of the approach with existing, grid-based motion planning algorithms is shown.

Keywords: Intelligent Transportation Systems, Deep Learning in Robotics and Automation
Abstract: Probabilistic vehicle trajectory prediction is essential for robust safety of autonomous driving. Current methods for long-term prediction cannot guarantee the physical feasibility of predicted distribution. Moreover, their models cannot adapt to the driving policy of the predicted target human driver. In this work, we propose to overcome these two shortcomings by a Bayesian recurrent neural network model consisting of Bayesian-neural-network-based policy model and known physical model of the scenario. Bayesian neural network can ensemble complicated output distribution, enabling rich family of trajectory distribution. The embedded physical model ensures feasibility of the distribution. Moreover, the adopted gradient-based training method allows direct optimization for better performance in long prediction horizon. Furthermore, a particle-filter-based parameter adaptation algorithm is designed to adapt the policy Bayesian neural network to the predicted target online. Effectiveness of the proposed methods is verified with a toy example with multi-modal stochastic feedback gain and naturalistic car following data.

Keywords: Intelligent Transportation Systems, Planning, Scheduling and Coordination, Path Planning for Multiple Mobile Robots or Agents
Abstract: Mobility-on-demand (MoD) systems are revolutionizing urban transit with the introduction of ride-sharing. Such systems have the potential to reduce vehicle congestion and improve accessibility of a city's transportation infrastructure. Recently developed algorithms can compute routes for vehicles in real-time for a city-scale volume of requests while allowing vehicles to carry multiple passengers at the same time. However, these algorithms focus on optimizing the performance for a given fleet of vehicles and do not tell us how many vehicles are needed to service all the requests. In this paper, we present an offline method to optimize the vehicle distributions and fleet sizes on historical demand data for MoD systems that allow passengers to share vehicles. We present an algorithm to determine how many vehicles are needed, where they should be initialized, and how they should be routed to service all the travel demand for a given period of time. Evaluation using 23,529,740 historical taxi requests from one month in Manhattan shows that on average 2864 four passenger vehicles are needed to service all of the taxi demand in a day with an average added travel delay of 2.8 mins.

Keywords: Intelligent Transportation Systems, Automation Technologies for Smart Cities, Computer Vision for Transportation
Abstract: This paper presents a vision-based technique and a system developed for the global reconstruction of three-dimensional (3-D) road surfaces. Using the system, the technique globally reconstructs 3-D road surfaces by estimating the global camera pose using the Adaptive Extended Kalman Filter (AEKF) and integrating it with existing local road surface reconstruction techniques. The AEKF adaptively updates the covariance of uncertainties such that the estimation works well even in environments with varying uncertainties. Numerical results show the efficacy of the proposed technique over the EKF-based technique by 50% in accuracy, and the on-road test has demonstrated the ability of the proposed technique for the real-world global 3D road surface reconstruction.

Keywords: Intelligent Transportation Systems, Computer Vision for Transportation, Deep Learning in Robotics and Automation
Abstract: We present a deep metadata fusion approach that connects image data and heterogeneous metadata inside a Convolutional Neural Network (CNN). This approach enables us to assign all relevant traffic lights to their associated lanes. To achieve this, a common CNN topology is trained by downsampled and transformed input images to predict an indication vector. The indication vector contains the column positions of all the relevant traffic lights that are associated with lanes. In parallel, we fuse prepared and adaptively weighted Metadata Feature Maps (MFM) with the convolutional feature map input of a selected convolutional layer. The results are compared to rule-based, only-metadata, and only-vision approaches. In addition, human performance of the traffic light to ego-vehicle lane assignment has been measured by a subjective test. The proposed approach outperforms all other approaches. It achieves about 93.0 % average precision for a real world dataset. In a more complex dataset, 87.1 % average precision is achieved. In particular, the new approach reaches significantly higher results with 93.7 % to 91.0 % average accuracy for a real world dataset in contrast to lower human performance.

Keywords: Surgical Robotics: Planning, Model Learning for Control, Learning from Demonstration
Abstract: Tissue manipulation is a frequently used fundamental subtask of any surgical procedures, and in some cases it may require the involvement of a surgeon's assistant. The complex dynamics of soft tissue as an unstructured environment is one of the main challenges in any attempt to automate the manipulation of it via a surgical robotic system. Two AI learning based model predictive control algorithms using vision strategies are proposed and studied: (1) reinforcement learning and (2) learning from demonstration. Comparison of the performance of these AI algorithms in a simulation setting indicated that the learning from demonstration algorithm can boost the learning policy by initializing the predicted dynamics with given demonstrations. Furthermore, the learning from demonstration algorithm is implemented on a Raven IV surgical robotic system and successfully demonstrated feasibility of the proposed algorithm using an experimental approach. This study is part of a profound vision in which the role of a surgeon will be redefined as a pure decision maker whereas the vast majority of the manipulation will be conducted autonomously by a surgical robotic system. A supplementary video can be found at: http://bionics.seas.ucla.edu/research/surgeryproject17.html

Keywords: Medical Robots and Systems, Surgical Robotics: Laparoscopy
Abstract: Fetoscopy is a technically challenging surgery, due to the dynamic environment and low diameter endoscopes often resulting in a limited field of view. In this paper, we report on the design and operation of a robotic multimodal endoscope with optical ultrasound and white light stereo camera. The manufacture and control of the endoscope is presented, along with large area (80 mm x 80 mm) surface visualisations of a placenta phantom using the optical ultrasound sensor. The repeatability of the surface visualisations were found to be 0.446±0.139 mm and 0.267±0.017 mm for a raster and spiral scan, respectively.

Keywords: Medical Robots and Systems, Force and Tactile Sensing, Robot Safety
Abstract: Retinal microsurgery is technically demanding and requires high surgical skill with very little room for manipulation error. Any unexpected manipulation could cause extreme tool-sclera contact force (scleral force), it could in turn lead to sclera damage. The introduction of robotic assistance could enhance and expand a surgeon's manipulation capabilities. However, the potential intraoperative danger coming from surgeon's mis-operations cannot be filtered or interrupted by the existing robotic systems. Therefore, we propose a method to predict the upcoming unsafe manipulation in robot-assisted retinal surgery, this predicted information is then fed back to the surgeon via auditory substitution. By this way the surgeon could react to the possible unsafe events in advance. The sclera safety is focused in this work. A force-sensing tool is fabricated and calibrated to measure the scleral force. A recurrent neural network is designed and trained to predict the scleral force status 500 milliseconds early. The auditory substitution is implemented to feedback the predicted force status to the surgeon. A vessel following manipulation is designed and performed on a dry eye phantom to simulate the retinal surgery and is further used to examine the proposed method. Five users are involved in the validation experiments. The results show that the early warning could help to reduce the number of the unsafe manipulation events.

Keywords: Surgical Robotics: Steerable Catheters/Needles, Telerobotics and Teleoperation, Medical Robots and Systems
Abstract: Robotic bronchoscopy has the potential to improve the early detection of lung cancer. For the technology to be broadly adopted, the physician needs to be able to control the robotic bronchoscope in an instinctive and effective manner. In this paper, we describe the algorithms used to manipulate/drive the Auris Monarch Platform, a 10 degree-of-freedom bronchoscope and sheath, using a 3 degree-of-freedom user input. We introduce the concept of paired driving where the devices co-insert and co-articulate depending on their relative insertion. The paper presents safety algorithms such as auto-relax on retract and tension monitoring. The drive modes were developed, optimized, and clinically tested in lung models, human cadavers and live porcine models prior to their commercial release. Clinical studies show that the physician is able to reach significantly deeper in the lung than with classic bronchoscopes.

Keywords: Surgical Robotics: Laparoscopy, Force Control, Cooperative Manipulators
Abstract: Surgeons performing laparoscopic surgery experience distortion when perceiving the stiffness of a patient’s tissues. This is due to the lever effect induced by the introduction of instruments in their patient’s body through a fulcrum. To address this problem, we propose to use the comanipulation paradigm. A robotic device is connected to the handle of the instrument while simultaneously being held by the surgeon. This device applies a force on the handle that reflects the force measured at the tool tip, with a gain that depends on the lever ratio. The implementation of this method is presented on an experimental setup and a preliminary assessment experiment is presented.

Keywords: Field Robots, Localization, Robotics in Hazardous Fields
Abstract: This paper presents the overall architecture of the VIKINGS robot, one of the five contenders in the ARGOS challenge and winner of two competitions. The VIKINGS robot is an autonomous or remote-operated robot for the inspection of oil and gas sites and is able to assess various petrochemical risks based on embedded sensors and processing.

The VIKINGS robot is able to autonomously monitor all the elements of a petrochemical process on a multi-storey oil platform (reading gauges, state of the valves, proper functioning of the pumps) while facing many hazards (leaks, obstacles or holes in its path).

In this article is presented the major components of the robot's architecture and the algorithms we developed for some functions (localization, gauge reading, etc). We also present the methodology that we adopted and that allowed us to succeed in this challenge.

Keywords: Space Robotics and Automation, Motion Control
Abstract: Dual-arm space robots have attracted increasing attention to perform on-orbit servicing missions autonomously or telerobotically. Coordinated control of the arms motion and the spacecraft base attitude considering coupling dynamics between them is essential for successful space operations. In this work both arms of the space robot perform as mission arms aimed at accomplishing secure capture of a floating target. Two types of controllers, i.e. a smoothed quasi-continuous second-order sliding mode controller (SQC2S) and an adaptive variable structure controller (AVSC) are developed and compared in three cases: path tracking, set-point regulation and inaccurate system model. The AVSC controller has been demonstrated to achieve higher tracking accuracy in all scenarios. Furthermore, AVSC controller saves a large amount of energy compared to SQC2S. This is a critical advantage since energy is limited on-board spacecraft and is quite valuable. The SQC2S controller cannot deal with large value system uncertainties reflected by large pose tracking errors of the end-effectors; whereas AVSC can restrict the position tracking error within a much smaller value. Therefore, AVSC has the ability to realize fast and high-accuracy tracking operation of space robots in space missions.

Keywords: Robotics in Hazardous Fields, Field Robots, Mapping
Abstract: Nuclear facilities can often require continuous monitoring to ensure there is no contamination of radioactive materials that might lead to safety or environmental issues. The current approach to radiological monitoring is to use human operators, which is both time consuming and cost inefficient, and as with many repetitive, routine tasks, there are considerable opportunities for the process to be improved through the utilization of autonomous robotic systems. This paper describes the design and development of an autonomous, ground based radiological monitoring robot, Continuous Autonomous Radiation Monitoring Assistance (CARMA) and how it was able to detect and locate a fixed alpha source that was embedded into the floor when it was deployed into an active area on the Sellafield nuclear site. This deployment was the first time that a fully autonomous robot had ever been deployed at Sellafield, the largest nuclear site in Europe.

Keywords: Field Robots, Autonomous Vehicle Navigation, Space Robotics and Automation
Abstract: Robotic foraging is a rich and potentially fruitful research field. Many robotics applications can be modeled as foraging problems, such as search and rescue, wildlife tracking, crop pollination and harvesting, mining and in-situ resource utilization, and scientific data/sample collection. In this article, the design of an autonomous foraging robot, named Cataglyphis, that won NASA’s Sample Return Robot Challenge in 2014, 2015, and 2016, is presented. The main goal of this article is to share the thinking process behind some of the key choices made during the design of Cataglyphis. The general framework that enabled autonomous sample return as described in this article may also be adapted to robots performing many other foraging-like applications.

Keywords: Robotics in Construction, Field Robots, Human-Centered Automation
Abstract: Interior painting of industrial developments is labor-intensive and performed with conventional techniques which are no doubt timeconsuming and tiresome We present a cooperative painting robot called Pictobot, Which provides a way to combine the benefits of automation in construction with those of human dexterity and ingenuity. Thus it relieves workers of the tiresome task and considerable climbing, bending, kneeling, and reaching, freeing them to supervise the robot. Pictobot is empowered with a sensor-driven painting system via in-situ 3D scanning and spray-gun motion planning which adapts to the uncertainties of construction environment and robot deployment from various positions. Besides, we employ human perception to decompose a broad functional area to smaller workspaces and to position the robot approximately at the expected places with the help of both visual and data feedbacks. So the human-Pictobot system works collaboratively, in which the worker's judgment and perception become the upper robot planner, and the robot adjusts the spray gun path and the painting plans from various deployed positions. The robot is tested in two actual industrial developments successfully. Pictobot enables achieving higher spraying transfer efficiency in comparison with manual sparing that means reducing paint dust, paint waste and human exposure to harmful chemicals. It also allows the existing workforce to achieve more with consistent coat quality and higher productivity.

Keywords: Telerobotics and Teleoperation, Robotics in Hazardous Fields, Motion Control
Abstract: There is a substantial financial incentive for in-situ repair of industrial assets. However, the need for highly rained mechanics to travel to the location of a repair often results in inconveniently long downtimes. The emergence of robots capable of replicating human interventions on industrial equipment can be coupled with remote control strategies to reduce the response time from several days to a few hours. This work outlines the design and remote control strategy for a novel robotic system to carry out repairs on aeroengine compressors in-situ via the internet. A high-level control computer serves as an interface with the skilled operator. A low-level controller receives instruction packets from the high-level controller via the internet and uses them to determine the necessary movements to carry out a machining operation. The robot, comprising a combination of rotary, prismatic and flexible (continuum) joints, was designed to replicate the degree of freedom of hand-held tools. Sensors and encoders on the robot enable the low-level controller to independently detect faults and stop all motion despite the high latency of internet communications. The remote control system was tested by machining stress relief features on eleven compressor blades with a median RMS error of 0.064 mm between the desired and measured blends. A successful demonstration on a production aeroengine shows the capability of the system.

Keywords: Flexible Robots, Soft Material Robotics
Abstract: Robotic arms built from stiffness-adjustable, continuously bending segments serially connected with revolute joints have the ability to change their mechanical architecture and workspace, thus allowing high flexibility and adaptation to different tasks with less than six degrees of freedom, a concept that we call malleable robots. Known stiffening mechanisms may be used to implement suitable links for these novel robotic manipulators; however, these solutions usually show a reduced performance when bending due to structural deformation. By including an inner support structure this deformation can be minimised, resulting in an increased stiffening performance. This paper presents a new multi-material spine-inspired flexible structure for providing support in stiffness-controllable layer-jamming-based robotic links of large diameter. The proposed spine mechanism is highly movable with type and range of motions that match those of a robotic link using solely layer jamming, whilst maintaining a hollow and light structure. The mechanics and design of the flexible spine are explored, and a prototype of a link utilising it is developed and compared with limb segments based on granular jamming and layer jamming without support structure. Results of experiments verify the advantages of the proposed design, demonstrating that it maintains a constant central diameter across bending angles and presents an improvement of more than 203% of resisting force at 180 degrees.

Keywords: Flexible Robots, Soft Material Robotics
Abstract: Inherent compliance plays an enabling role in soft robots, which rely on it to mechanically conform to the environment. However, it also limits the payload of the robots. Various variable stiffness approaches have been adopted to limit compliance and provide structural stability, but most of them can only achieve stiffening of discrete fixed regions which means compliance cannot be precisely adjusted for different needs. This paper offers an approach to enhance the payload with finely adjusted compliance for different needs. We have developed a manipulator that incorporates a novel variable stiffness mechanism and a sliding layer mechanism. The variable stiffness mechanism can achieve a 6.4 stiffness changing ratio with a miniaturized size (10mm diameter for the testing prototype) through interlocking jamming layers with a honeycomb core. The sliding layer mechanism can actively shift the position of the stiffening regions through sliding of jamming layers. A model to predict the robot shape is derived with verifications via an experiment. The stiffening capacity of the variable stiffness mechanism is also empirically evaluated. A case study of a potential application in laparoscopic surgeries is showcased. The payload of the manipulator is investigated, and the prototype shows up to 57.8 percentage decrease of the vertical deflection due to an external load after reconfigurations.

Keywords: Soft Material Robotics, Hydraulic/Pneumatic Actuators, Mechanism Design
Abstract: This article describes an innovative design of variable stiffness soft actuator, which can potentially be utilized for manipulation and locomotion of soft robots. The new actuator is a combination of two types of actuations: soft pneumatic actuation and muscle-like supercoiled polymer (SCP) actuation. Soft pneumatic actuator has two roles: first is to generate bending motions and second is to increase the stiffness of the whole actuator together with SCP artificial muscles. SCP artificial muscles are exploited to generate pre-load to resist the whole actuator from (excessive) deformation when external load is applied. These two types of actuations are arranged antagonistically to realize stiffness tuning of the whole actuator. At a given bending position, stiffness of the actuator could be tuned by controlling the pressure inside the air chamber and the tension on the SCP artificial muscles. In experimental section, tests are conducted to characterize the applied SCP artificial muscles before they are applied to the proposed actuator. Afterwards, tests of proposed actuator are performed to examine its variable stiffness capability. From experimental results, the proposed actuator can achieve 3.47 times stiffness variation ratio from 0.0312 N/mm (40 kPa air pressure and no SCP actuation) to 0.1083 N/mm (82 kPa air pressure and SCP actuation at 0.143 W/cm) at the same position (bending angle of 56 degree). This study exhibits the potential of applying SCP artificial muscles to

Keywords: Soft Material Robotics, Haptics and Haptic Interfaces
Abstract: Haptic devices use touch to enable communication in a salient and private manner. While most haptic devices are held or worn at the hand, there is recent interest in developing wearable haptic devices for the arms. This frees the hands for manipulation tasks, but creates challenges for wearability. One approach is to use pneumatically driven soft haptic devices that, compared to rigid devices, can be more readily worn due to their form factor and light weight. We propose a two-degree of freedom (2-DOF) pneumatic soft linear tactor that can be mounted on the forearm and provide shear force. The tactor is comprised of four soft fiber-constrained linear pneumatic actuators connected to a dome-shaped tactor head. The tactor can provide fast, repeatable forces on the order of 1 N in shear, in various directions in the plane of the skin surface. We demonstrate the tradeoffs of two housing schemes, one soft and one rigid, that mount the pneumatic soft linear actuator to the forearm. A user study demonstrated the performance of both versions of the device in providing directional cues, highlighting the challenges and importance of grounding soft wearable devices and the difficulties of designing haptic devices given the perceptual limits of the human forearm.

Keywords: Soft Material Robotics, Grippers and Other End-Effectors, Mechanism Design
Abstract: Pneumatic soft grippers have nonlinear continuum deformation enabling them to adapt to irregular object shapes. Most existing pneumatic soft grippers use only open loop control. Attempts for close loop control of soft grippers are all based on air pressure regulation, which is inaccurate and cumbersome due to nonlinear performance of the soft actuator and the compressible nature of air. In this paper, we presents a controllable soft gripper based on pre-charged pneumatic (PCP) soft actuators. The soft actuators, with the same design as a normal bending pneumatic soft actuator (PSA), are pre-charged with air to a preset pressure through a one way check valve. The bending angle and bending speed of soft actuators are controlled by servomotors through tendons based on feedback data from force and proximity sensors. Kinematic models of the soft actuators are developed. Dynamic properties are experimentally studied. A prototype gripper with close loop control is developed for controllable grasping demonstration.

Keywords: Soft Material Robotics, Motion Control, Model Learning for Control
Abstract: Soft robots attract research interests worldwide. However, its control remains challenging due to the difficulty in sensing and accurate modeling. In this paper, we propose a novel iterative learning model predictive control (ILMPC) method for soft bending actuators. The uniqueness of our approach is the ability to improve model accuracy gradually. In this method, a pseudo-rigid-body model is used to take an initial guess of the bending behavior of the actuator and the model accuracy is improved with iterative learning. Compared with conventional model free iterative learning control (ILC), the proposed method significantly reduces the learning curve. Compared with the model predictive control (MPC), the proposed method does not rely on an accurate model and it will output a satisfactory model after the learning process. A soft-elastic composite actuator (SECA) is used to validate the proposed method. Both simulation and experimental results show that the proposed method outperforms the conventional MPC and ILC.

Keywords: Haptics and Haptic Interfaces, Mechanism Design, Tendon/Wire Mechanism
Abstract: Haptic joysticks for man-machine interaction used in aerospace flight control have highly demanding requirements of reliability, force density, and high dynamics that can hardly be met with conventional electromagnetic actuators. This work explores the potential of using an alternative actuation strategy based on hyper-redundant MR clutches that modulate the force of a tendon-driven 2-degree-of-freedom spherical gimbal. A system design and its closed-loop force control scheme are proposed. Experimental results for an open-loop characterization, static force control and dynamic force control are set out and compared with typical requirements for such devices from the literature. Results show that the proposed architecture leads to one of the lightest systems reported in the literature that has the potential to meet reliability requirements by providing a jam-free design with duplex fault tolerance, and yet, can generate high force levels while providing enough force resolution. The approach is promising and can extend to high-performance collaborative robot applications.

Keywords: Wearable Robots, Hydraulic/Pneumatic Actuators, Haptics and Haptic Interfaces
Abstract: Supernumerary Robotic Limbs (SRLs) are wearable robots augmenting human capabilities by acting as a co-worker, reaching objects, support human arms, etc. However, existing SRLs lack the mechanical backdrivability and bandwidth required for tasks where the interaction forces must be controlled such as painting, drilling, manipulating objects, etc. Being highly backdrivable with a high bandwidth while minimizing weight presents a major technological challenge imposed by the limited performances of conventional electromagnetic actuators. This paper studies the feasibility of using magnetorheological (MR) clutches coupled to a low-friction hydrostatic transmission to provide a highly capable, but yet lightweight, force-controllable SRL. A 2.7 kg 2-DOFs wearable robotic arm is designed and built. Shoulder and elbow joints are designed to deliver 39 and 25 Nm, with 115 and 180° of range of motion. Experimental studies conducted on a one-DOF test bench and validated analytically demonstrate a high force bandwidth (>25 Hz) and a good ability to control interaction forces even when interacting with an external impedance. Furthermore, three force-control approaches are studied and demonstrated experimentally: open-loop, closed-loop on force, and closed-loop on pressure. All three methods are shown to be effective. Overall, the proposed MR-Hydrostatic actuation system is well-suited for a lightweight SRL interacting with both human and environment that add unpredictable disturbances.

Keywords: Haptics and Haptic Interfaces, Wearable Robots, Tendon/Wire Mechanism
Abstract: Grounding of kinesthetic feedback against a user's hand can increase the portability and wearability of a haptic device. However, the effects of different hand-grounding locations on haptic perception of a user are unknown. In this letter, we investigate the effects of three different hand-grounding locations—back of the hand, proximal phalanx of the index finger, and middle phalanx of the index finger—on haptic perception using a newly designed wearable haptic device. The novel device can provide kinesthetic feedback to the user's index finger in two directions: along the finger-axis and in the finger's flexion-extension movement direction. We measure users’ haptic perception for each grounding location through a user study for each of the two feedback directions. Results show that among the studied locations, grounding at proximal phalanx has a smaller average just noticeable difference for both feedback directions, indicating a more sensitive haptic perception. The realism of the haptic feedback, based on user ratings, was the highest with grounding at the middle phalanx for feedback along the finger axis, and at the proximal phalanx for feedback in the flexion-extension direction. The results provide insights for designing next-generation wearable hand-grounded kinesthetic devices to achieve better haptic performance and user experience in virtual reality and teleoperated robotic applications.

Keywords: Surgical Robotics: Laparoscopy, Force and Tactile Sensing, Haptics and Haptic Interfaces
Abstract: This paper evaluates the feasibility of a novel optical sensing concept to measure forces applied at the tip of daVinci EndoWrist instruments. An optical slit is clamped onto the instrument shaft, in-line with an infrared LED-bicell pair. Deflection of the shaft moves the slit with respect to the LED-bicell pair and modulates the light incident on each active element of the bicell. The differential photocurrent is conditioned and monitored to estimate the tip forces. The feasibility evaluation consists of a flexible beam model to quantify the required sensor performance, experimental results with a 3D printed prototype and estimation of the sensor limitations including the measurement bandwidth due to the structural dynamics. The proposed approach requires no modifications to the instrument, is adaptable to different instruments and robot platforms, and leads to high-resolution, high-dynamic range sensing without hysteresis.

Keywords: Wearable Robots, Mechanism Design
Abstract: In this study, a human-chair model was developed as the basis for a wearable chair design. A prototype chair, HUST-EC, was fabricated and evaluated. Employing the optimization under an inner point penalty function, an optimized simulation of the operating mode with the lowest chair height was implemented. The solid models were established by using the finite element analysis program embedded in Solidworks, which revealed that the support from the designed chair was steady to the user. An electromyography (EMG) test platform has been developed, consisting of four EMG sensors, a MATLAB-based acquisition software, and a loaded vest. Four healthy subjects participated in the evaluation experiment, in which EMGs were collected from the muscle groups of rectus femoris, biceps femoris, vastus medialis, and vastus lateralis under different loads and chair angles. The experimental data demonstrate that (1) the HUST-EC can greatly reduce muscle activation at a variety of loads and bending angles; (2) under the same load, the muscle activation decreases slightly with an increased bending angle; and (3) at the same bending angle, muscle activation increases slightly with an increased load. The results show that the designed chair can effectively reduce the physical burden in workers and may improve work efficiency.

Keywords: Haptics and Haptic Interfaces, Virtual Reality and Interfaces, Soft Material Robotics
Abstract: Virtual reality experiences via immersive optics and sound are becoming ubiquitous; there are several consumer systems (e.g., Oculus Rift and HTC Vive) now available with these capabilities. Other sensory experiences, such as that of touch remain elusive in this field. The most successful examples of haptic sensation (e.g., Nintendo 64's Rumble Pack and its descendants) are vibrotactile, which do not afford for persistent, morphological shape experiences. This paper presents work on the development of a 12 DOF fluidically pressurized soft actuator for persistent and kinesthetic haptic sensations, a hardware controller for operating it, and software interface with NVIDIA's game VR Funhouse.

Keywords: SLAM, Sensor Fusion, Kinematics
Abstract: Dense visual SLAM methods are able to estimate the 3D structure of an environment and locate the observer within them. They estimate the motion of a camera by matching visual information between consecutive frames, and are thus prone to failure under extreme motion conditions or when observing texture-poor regions. The integration of additional sensor modalities has shown great promise in improving the robustness and accuracy of such SLAM systems. In contrast to the popular use of inertial measurements we propose to tightly-couple a dense RGB-D SLAM system with kinematic and odometry measurements from a wheeled robot equipped with a manipulator. The system has real-time capability while running on GPU. It optimizes the camera pose by considering the geometric alignment of the map as well as kinematic and odometric data from the robot. Through experimentation in the real-world, we show that the system is more robust to challenging trajectories featuring fast and loopy motion than the equivalent system without the additional kinematic and odometric knowledge, whilst retaining comparable performance to the equivalent RGB-D only system on easy trajectories.

Keywords: Robot Audition, Localization
Abstract: We present a novel sound localization algorithm for a non-line-of-sight (NLOS) sound source in indoor environments. Our approach exploits the diffraction properties of sound waves as they bend around a barrier or an obstacle in the scene. We combine a ray tracing-based sound propagation algorithm with a Uniform Theory of Diffraction (UTD) model, which simulate bending effects by placing a virtual sound source on a wedge in the environment. We precompute the wedges of a reconstructed mesh of an indoor scene and use them to generate diffraction acoustic rays to localize the 3D position of the source. Our method identifies the convergence region of those generated acoustic rays as the estimated source position based on a particle filter. We have evaluated our algorithm in multiple scenarios consisting of static and dynamic NLOS sound sources. In our tested cases, our approach can localize a source position with an average accuracy error of 0.7m, measured by the L2 distance between estimated and actual source locations in a 7m×7m×3m room. Furthermore, we observe 37% to 130% improvement in accuracy over a state-of-the-art localization method that does not model diffraction effects, especially when a sound source is not visible to the robot.

Keywords: SLAM, Deep Learning in Robotics and Automation, Mapping
Abstract: While the keypoint-based maps created by sparse monocular Simultaneous Localisation and Mapping (SLAM) systems are useful for camera tracking, dense 3D reconstructions may be desired for many robotic tasks. Solutions involving depth cameras are limited in range and to indoor spaces, and dense reconstruction systems based on minimising the photometric error between frames are typically poorly constrained and suffer from scale ambiguity. To address these issues, we propose a 3D reconstruction system that leverages the output of Convolutional Neural Networks (CNNs) to produce fully dense depth maps for keyframes that include metric scale.

Our system, DeepFusion, is capable of producing dense reconstructions at real-time on a GPU. It fuses the output of a semi-dense multiview stereo algorithm with the depth and gradient predictions of a CNN in a probabilistic fashion, using learned uncertainties produced by the network. While the network only needs to be run once per keyframe, we are able to optimise for the depth map with each new frame so as to constantly make use of new geometric constraints. Based on its performance on synthetic and real world datasets, we demonstrate that DeepFusion is capable of performing at least as well as other comparable systems.

Keywords: SLAM, Mapping, Visual Learning
Abstract: In this paper, we proposed a new deep learning based dense monocular SLAM method. Compared to existing methods, the proposed framework constructs a dense 3D model via a sparse to dense mapping using learned surface normals. Together with single view learned depth estimation as prior for monocular visual odometry, both accurate positioning and quality depth reconstruction are obtained. The depth and normal are predicted by a single network trained in a tightly coupled manner.

Experimental results show that our method significantly improves the performance of visual tracking and depth prediction in comparison to the state-of-the-art in deep monocular dense SLAM.

Keywords: SLAM, Localization, Mapping
Abstract: We propose a novel semi-direct approach for monocular simultaneous localization and mapping (SLAM) that combines the complementary strengths of direct and feature-based methods. The proposed pipeline loosely couples direct odometry and feature-based SLAM to perform three levels of parallel optimizations: (1) photometric bundle adjustment (BA) that jointly optimizes the local structure and motion, (2) geometric BA that refines keyframe poses and associated feature map points, and (3) pose graph optimization to achieve global map consistency in the presence of loop closures. This is achieved in real-time by limiting the feature-based operations to marginalized keyframes from the direct odometry module. Exhaustive evaluation on two benchmark datasets demonstrates that our system outperforms the state-of-the-art monocular odometry and SLAM systems in terms of overall accuracy and robustness.

Keywords: Mapping, Field Robots, Learning and Adaptive Systems
Abstract: This paper addresses the problem of learning instantaneous occupancy levels of dynamic environments and predicting future occupancy levels. Due to the complexity of most real environments, such as urban streets or crowded areas, the efficient and robust incorporation of temporal dependencies into otherwise static occupancy models remains a challenge. We propose a method to capture the uncertainty of moving objects and incorporate this uncertainty information into a continuous occupancy map represented in a rich high-dimensional feature space. This data-efficient model not only allows us to learn the occupancy states incrementally, but also makes predictions about what the future occupancy states will be. Experiments performed using 2D and 3D laser data collected from crowded unstructured outdoor environments show that the proposed methodology can accurately predict occupancy states for areas of around 1000 m^2 at 10 Hz, making the proposed framework ideal for online applications under real-time constraints.

Keywords: Mapping
Abstract: In this work, we present an information based exploration strategy tailored for the generation of high resolution 3D maps. We employ RGBD panoramas because they have been shown to provide memory efficient high quality representations of space. Robots explore the environment by selecting locations with maximal Cauchy-Schwarz Quadratic Mutual Information (CSQMI) computed on an angle enhanced occupancy grid to collect these RGBD panoramas. By employing the angle enhanced occupancy grid, the resulting exploration strategy emphasizes perspective in addition to binary coverage. Furthermore, the goal selection strategy is improved by using image morphology to reduce the search space over which CSQMI is computed. We present experimental results demonstrating the improved performance in perception related tasks by capturing panoramas using this approach, near frontier exploration, and a control of logging images at regular intervals while teleoperating the robot through the workspace. Collect imagery was passed through an object detection library with our perspective aware approach yielding a greater number of successful detections compared to near frontier exploration.

Keywords: Mapping, Learning and Adaptive Systems, Autonomous Vehicle Navigation
Abstract: Robot navigation in outdoor environments is exposed to detrimental factors such as vibrations or power consumption due to the different terrains on which the robot navigates. In this paper, we address the problem of actively improving navigation by planning paths that aim at reducing over time phenomena such as vibrations during traversal. Our approach uses a Gaussian Process~(GP) mixture model and an aerial image of the environment to learn and improve continuously a place-dependent model of such phenomena from the experiences of the robot. We use this model to plan paths that trade-off the exploration of unknown promising regions and the exploitation of known areas where the impact of the detrimental factors on navigation is low, leading to an improved navigation over time. We implemented our approach and thoroughly tested it using real-world data. Our experiments suggest that our approach with no initial information leads the robot, after few runs, to follow paths along which it experiences similar vibrations or energy consumption as if it was following the optimal path computed given the ground truth information.

Keywords: Mapping, Learning and Adaptive Systems
Abstract: Mapping the occupancy of an environment is central for robot autonomy. Traditional occupancy grid maps discretise the environment into independent cells, neglecting important spatial correlations, and are unable to capture the continuous nature of the real world. With these drawbacks of grid maps in mind, Hilbert Maps (HM) and more recently Bayesian Hilbert Maps (BHMs), were introduced as a continuous representation of the environment. In this paper we propose a method to merge Bayesian Hilbert Maps built by a team of robots in a decentralised manner. The training of BHMs requires the inversion of a large covariance matrix, incurring cubic complexity. We introduce an approximation, Fast Bayesian Hilbert Maps (Fast-BHM), which reduces the time complexity to below quadratic. Such speed-ups allow the building and merging of Bayesian Hilbert Map models to be practical, opening the door for multi-robot Hilbert Map systems which can be much faster and more robust than an individual robot. By merging several individual Fast-BHMs in a decentralised manner we obtain a unified model of the environment which is itself a Fast-BHM. We conduct experiments to show that global Fast-BHM models do not deteriorate after repeated merging and training. We then empirically demonstrate, due to its the compact representation, fused Fast-BHMs outperform fusion methods involving discretising continuous representations, when the amount of information communicated is limited.

Keywords: Mapping, Range Sensing, SLAM
Abstract: In this letter, we propose a method which can be used to regenerate the 3D normal distributions transform (NDT) for target lattice. When a pose is updated by SLAM, the lattice at the pose is also transformed. Given that NDT is a Gaussian mixture model generated by regular cells, the fusion of NDTs transformed with updated poses can distort the shapes of the Gaussian components (GCs). Moreover, when robots without information about other robots' initial poses share and fuse NDT maps, the simple fusion of NDT maps built in different lattices can distort GCs. To overcome this problem, we propose a method by which GCs are subdivided into truncated GCs by the target lattices on each axis iteratively, and the truncated GCs in the same target cell are fused. To determine whether the GC should be subdivided, we define a weight threshold assigned to the weight corresponding to the truncated GC. In an experiment, we evaluated the receiver operating characteristics, the accuracy, the L2 value, the mean error, the mean covariance distance based on Frechet distance to assess the similarity of the regenerated NDT and ground truth NDT. Also, we evaluated the computational performance of the proposed method. Moreover, we evaluated the application of map fusion. It was found that the NDT regenerated by the proposed method showed improvement in the L2 value, mean error, and mean covariance distance.

Keywords: Mapping, SLAM
Abstract: In this paper we present a novel global Structure from Motion (SfM) pipeline that is particularly effective in dealing with low-parallax scenes and camera motion collinear with the features that represent the environment structure. It is therefore particularly suitable in Urban SLAM, in which frequent road-facing motion poses many challenges to conventional SLAM algorithms. Our pipeline includes a recently explored bundle adjustment (BA) method that exploits a feature parameterization using parallax angle between on-manifold observation rays (PMBA). This BA stage is demonstrated to have a consistently stable optimization configuration for features with any parallax and therefore low-parallax features can stay in reconstruction without pre-filtering. To allow practical usage of PMBA, we provide a compatible initialization stage in the SfM to initialize all camera poses simultaneously, exhibiting friendliness to collinear motion. This is achieved by simplifying PMBA into a hybrid graph problem of high connectivity yet small node set, solved using a robust linear programming technique. Using simulations and a series of publicly available real datasets including “KITTI” and “BAL”, we demonstrate the robustness of the position initialization stage in handling collinear motion and outlier matches, superior convergence performance of the BA stage in presence of low-parallax features, and effectiveness of our pipleline to handle many sequential or out-of-order urban scenes.

Keywords: Aerial Systems: Mechanics and Control, Failure Detection and Recovery
Abstract: Compared to other vertical take-off and landing (VTOL) systems, a tailsitter minimizes the number of actuators and moving parts necessary. The downside of having a minimalistic actuation is its inherent low fault-tolerance. The failure of an actuator usually results in a loss of controllability, resulting in a crash. In this paper we analyze the possible actuator failures and the constraints they pose on the capabilities of the system. We further present light-weight adaptations to the nominal flight controller to make it fault-tolerant. The fault-tolerant controller is implemented on a small tailsitter VTOL aircraft and adjusted to the system by means of extensive experimental studies. Finally, the capabilities and performance under failures are demonstrated and analyzed.

Keywords: Aerial Systems: Mechanics and Control, Dynamics, Control Architectures and Programming
Abstract: This paper presents the modeling and control of a passively-coupled tilt-rotor vertical takeoff and landing aircraft. The aircraft consists of a quadrotor frame attached to a fixed-wing aircraft by an unactuated hinged mechanism. The platform is capable of smooth transitions from hover to forward flight without the use of tilting actuators. The transition from hover to forward flight is made possible by differential thrust between the fore and aft propellers of the quadrotor frame. In this paper, the coupled dynamics between the quadrotor frame and the aircraft frame are modeled as a constrained multi-body system. The equations of motion are established using a constrained Lagrangian approach and the model developed is used to build a realistic simulation environment for control design purpose. A cascaded control architecture based on P/PID controllers is proposed to achieve inner-loop attitude, height and forward velocity control. Simulated and experimental results are obtained with a close match for hover, transitions, forward flight, and banked turn maneuvers.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications
Abstract: In response to an abundance of applications, Unmanned Aerial Vehicles are being called upon to perform missions of high difficulty for increasingly long periods of time. Traditional paradigms of propeller design and actuation are reaching a design ceiling, motivating creative approaches to the design of propeller-based propulsion mechanisms. Within the last decade, one particular kind of mechanism, the variable pitch propeller, has been studied by researchers for its applications to the class of small UAVs. This paper pushes for new results in this area by exploring the use of Variable Pitch Propulsion (VPP) for maximizing efficiency for small, versatile UAVs. A control algorithm is presented to minimize the consumed electrical power during a quasi-steady propulsive state. In particular, the algorithm is not confined to operation in limited regions of the state space, but seeks to minimize power at whatever point in the state space a steady state is reached. Several experimental results are presented to validate the approach.

Keywords: Aerial Systems: Mechanics and Control, Biologically-Inspired Robots
Abstract: We present the design, fabrication, and feedforward control of a insect-sized (142~mg) aerial robot that is equipped with a bio-inspired inertial tail. A tail allows the robot to perform rapid inertial reorientation as well as to shift weight to modulate aerodynamic torques on its body. Here we present the first analysis of inertial reorientation using a piezo actuator, departing from previous work to date that has focused exclusively on actuation by DC electric motor. The primary difference is that unlike a geared motor system, the piezo-tail system operates as a resonant system, exhibiting slowly-decaying oscillations. We present a dynamic model of piezo-driven inertial reorientation, along with an open-loop feedforward controller that reduces excitation of the resonant mode. We validate our approach on a tethered testbed as well as a flight-capable prototype. Our results indicate that incorporating a tail can allow for more rapid dynamic maneuvers and could stabilize the robot during flight.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications
Abstract: In this paper the problem of contact-based navigation path planning for aerial robots is considered with the goal of enabling the autonomous in-contact operation on surfaces that can be highly anomalous. Such a capacity can prove critical in inspection through contact missions, as well as when a flying robot is tasked to operate in very narrow environments rendering safe free-flight impossible. To achieve this objective, beyond sliding in contact, a new locomotion primitive is introduced, namely that of azimuth rotations perpendicular to the surface under consideration. This new navigation mode, called flying cartwheel mode, offers navigation resourcefulness and resilience when the system is tasked to move in contact with surfaces that are otherwise non-traversable. The designed path planning method exploits both navigation modalities and a traversability metric to decide when to switch from sliding to flying cartwheel mode, and overall provides cost-optimal trajectories for in-contact navigation. The proposed approach is verified both in simulation, as well as experimentally using a surface presenting complex anomalies. It is highlighted that the proposed method does not assume any specialized contact mechanism or a control law tailored to physical interaction tasks, and hence is applicable to almost any micro aerial vehicle integrating protective shrouds around its propellers.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications, Dynamics
Abstract: In this paper, we introduce a cargo transportation method with a new type of multi-rotor UAV platform known as T3-multirotor, to achieve stable and constant flight performance regardless of the type of cargo attached to the fuselage. The T3-multirotor, which consists of the `Thrust Generating Part' and the `Fuselage Part', can directly control the relative attitude between the two parts using the novel servomechanism. By utilizing the servomechanism with the proposed relative attitude control strategy, the T3-multirotor with cargo attached to the fuselage part can behave as a multi-rotor with only the moment of inertia of the thrust generating part during entire transportation. This allows the T3-multirotor to achieve the reliable performance in the event of any cargo being attached to the fuselage, achieving stable platform motion control. Detailed hardware description and dynamic analysis of T3-Multirotor is performed in this paper, and the validity of the proposed control strategy is also analyzed. The feasibility of the proposed control strategy is verified through experimental results with analysis.

Keywords: Learning and Adaptive Systems, Aerial Systems: Applications, Biologically-Inspired Robots
Abstract: In this work, we present an application of a policy gradient algorithm to a real-time robotic learning problem, where the goal is to maximize the average lift generation of a dynamically scaled robotic wing at a constant Reynolds number (Re). Compared to our previous work, the merit of this work is two-fold. First, a central pattern generator model was used as the motion controller, which provided a smooth generation and transition of rhythmic wing motion patterns while the CPG was being updated by the policy gradient, thereby accelerating the sample generation and reducing the total learning time. Second, the kinematics included three degrees of freedom (stroke, deviation, pitching) and were also free of half-stroke symmetry constraint, together they yielded a larger kinematic space which later explored by the policy gradient to maximize the lift generation. The learned wing kinematics used the full range of stroke and deviation to maximize the lift generation, implying that the wing trajectories with larger disk area and lower frequencies were preferred for high lift generation at constant Re. Furthermore, the wing pitching amplitude converged to values between 45°-49° regardless of what the other parameters were. Notably, the learning agent was able to find two locally optimal wing motion patterns, which had distinct shapes of wing trajectory but generated similar cycle-averaged lift.

Keywords: Aerial Systems: Applications, Mobile Manipulation, Grippers and Other End-Effectors
Abstract: This paper is an initial step toward the realization of an aerial robot that can perform lateral physical work, such as drilling a hole or fastening a screw in a wall. Aerial robots are capable of high maneuverability and can provide access to locations that would be difficult or impossible for ground-based robots to reach. However, to fully utilize this mobility, systems would ideally be able to perform functional work in those locations, requiring the ability to exert lateral forces. To substantially improve a hovering vehicle's ability to stably deliver large lateral forces, we propose the use of a versatile suction-based gripper that can establish pulling contact on featureless surfaces. Such contact enables access to environmental forces that can be used to further stabilize the vehicle and also increase the lateral force delivered to the surface through a possible secondary mechanism. This paper introduces the concept, describes the design of a new self-sealing suction cup based on a previous design, details the design of a gripper using those cups, and describes the arm and flight vehicle. It then evaluates the cup and gripper performance in several ways, culminating in physical grasping demonstrations using the arm and gripper, including one in the presence of simulated flight noise based on data from preliminary indoor flight experiments.

Keywords: Aerial Systems: Applications, Force and Tactile Sensing
Abstract: This paper reports the design, fabrication and testing of light-weight whisker sensors intended for use on robotic platforms, especially drones, featuring multiple whisker fibres in an array. The whiskers transmit forces along the vibrissae fibres to a load plate bonded to embedded MEMS barometers potted in polyurethane rubber, which act as force sensors. This construction allows for directional sensing of forces with arrays of fibres weighing less than half a gram per whisker. Forces as low as 0.34 mg can be measured, and the whiskers are capable of sensing fluid stream velocities up to 7.5 ms−1. The whiskers are sufficiently sensitive so as to be able to detect the pressure wave of an approaching hand moving at 0.53 ms−1 from 20 mm away.

Keywords: Aerial Systems: Applications, Visual-Based Navigation, Sensor-based Control
Abstract: Redundant navigation systems are critical for safe operation of UAVs in high-risk environments. Since most commercial UAVs almost wholly rely on GPS, jamming, interference and multi-pathing are real concerns that usually limit their operations to low-risk environments and Visual Line-of-Sight. This paper presents a vision-based route-following system for the autonomous, safe return of UAVs under primary navigation failure such as GPS jamming. Using a Visual Teach and Repeat framework to build a visual map of the environment during an outbound flight, we show the autonomous return of the UAV by visually localising the live view to this map when a simulated GPS failure occurs, controlling the vehicle to follow the safe outbound path back to the launch point. Using gimbal-stabilised stereo vision and inertial sensing alone, without reliance on external infrastructure, VO and localisation are achieved at altitudes of 5-25 m and flight speeds up to 55 km/h. We examine the performance of the visual localisation algorithm under a variety of conditions and also demonstrate closed-loop autonomy along a complicated 450 m path.

Keywords: Aerial Systems: Applications, Human-Centered Robotics
Abstract: In this work, we address the problem of providing human-assisted quadrotor navigation using a set of eye tracking glasses. The advent of these devices (i.e., eye tracking glasses, virtual reality tools, etc.) provides the opportunity to create new, non-invasive forms of interactions between an human and robots. We show how a set of glasses equipped with gaze tracker, a camera, and an Inertial Measurement Unit (IMU) can be used to (a) estimate the relative position of the human with respect to a quadrotor, (b) decouple the gaze direction from the head orientation, and (c) allow the human {em spatially task} (i.e., send new 3D navigation waypoints to) the robot in an uninstrumented environment. We employ a combination of camera and IMU data to track the human's head orientation, which allows us to decouple the gaze direction from the head motion.In order to detect the flying robot, we train and use a deep neural network.We evaluate the proposed approach experimentally, and show that our pipeline is able to successfully achieve human-guided autonomy for spatial tasking. The proposed approach can be employed in a wide range of scenarios including inspection, first response, and it can be used by people with disabilities that affect their mobility.

Keywords: Aerial Systems: Applications, Motion and Path Planning, Autonomous Vehicle Navigation
Abstract: In this paper, we propose a novel motion planning framework for quadrotor teach-and-repeat applications. Instead of controlling the drone to precisely follow the teaching path, our method converts an arbitrary jerky human-piloted trajectory to a topologically equivalent one, which is guaranteed to be safe, smooth, and kinodynamically feasible with an expected aggressiveness. Our proposed planning framework optimizes the trajectory in both spatial and temporal aspects. In the spatial layer, a flight corridor is found to represent the free space which is topologically equivalent with the teaching path. Then a minimum-jerk piecewise trajectory is generated within the flight corridor. In the temporal layer, the trajectory is re-parameterized to obtain a minimum-time temporal trajectory under kinodynamic constraints. The spatial and temporal optimizations are both formulated as convex programs and are done iteratively. The proposed method is integrated into a complete quadrotor system and is validated to perform aggressive flights in challenging indoor and outdoor environments.

Keywords: Agricultural Automation, Robotics in Agriculture and Forestry, Field Robots
Abstract: Autonomous robotic weeding systems in precision farming have demonstrated their full potential to alleviate the current dependency on herbicides or pesticides by introducing selective spraying or mechanical weed removal modules, thus reducing the environmental pollution and improving the sustainability. However, most previous works require fast weed detection system to achieve real-time treatment. In this paper, a novel computer vision based weeding control system is presented, where a non-overlapping multi-camera system is introduced to compensate the indeterminate classification delays, thus allowing for more complicated and advanced detection algorithms, e.g. deep learning based methods. The suitable tracking and control strategies are developed to achieve accurate and robust in-row weed treatment, and the performance of the proposed system is evaluated in different terrain conditions in the presence of various delays.

Keywords: Automation in Life Sciences: Biotechnology, Pharmaceutical and Health Care, Human Detection and Tracking, Deep Learning in Robotics and Automation
Abstract: In this work, we present a novel framework for on-line human gait stability prediction of the elderly users of an intelligent robotic rollator using Long Short Term Memory (LSTM) networks, fusing multimodal RGB-D and Laser Range Finder (LRF) data from non-wearable sensors. A Deep Learning (DL) based approach is used for the upper body pose estimation. The detected pose is used for estimating the body Center of Mass (CoM) using Unscented Kalman Filter (UKF). An Augmented Gait State Estimation framework exploits the LRF data to estimate the legs' positions and the respective gait phase. These estimates are the inputs of an encoder-decoder sequence to sequence model which predicts the gait stability state as Safe or Fall Risk walking. It is validated with data from real patients, by exploring different network architectures, hyperparameter settings and by comparing the proposed method with other baselines. The presented LSTM-based human gait stability predictor is shown to provide robust predictions of the human stability state, and thus has the potential to be integrated into a general user-adaptive control architecture as a fall-risk alarm.

Keywords: Automation Technologies for Smart Cities, Swarms, Agent-Based Systems
Abstract: Modern cities are growing ecosystems that face new challenges due to the increasing population demands. One of the many problems they face nowadays is waste management, which has become a pressing issue requiring new solutions. Swarm robotics systems have been attracting an increasing amount of attention in the past years and they are expected to become one of the main driving factors for innovation in the field of robotics. The research presented in this paper explores the feasibility of a swarm robotics system in an urban environment. By using bio-inspired foraging methods such as multi-place foraging and stigmergy-based navigation, a swarm of robots is able to improve the efficiency and autonomy of the urban waste management system in a realistic scenario. To achieve this, a diverse set of simulation experiments was conducted using real-world GIS data and implementing different garbage collection scenarios driven by robot swarms. Results presented in this research show that the proposed system outperforms current approaches. Moreover, results not only show the efficiency of our solution, but also give insights about how to design and customize these systems.

Keywords: Automation in Life Sciences: Biotechnology, Pharmaceutical and Health Care
Abstract: This paper presents the first system for automated ex vivo perfusion of an isolated heart and regulating the heart’s aortic pressure (AoP). An adaptive controller was developed for AoP regulation and maintained the heart’s physiological aerobic metabolism. A mathematical model of the perfusion system was established based on a nonlinear equivalent circuit fluid flow model. The model combined with a virtual controller forms a reference model to generate the ideal trajectory of AoP. An adaptation algorithm tunes the control parameters based on the reference model and the isolated heart. Experiments were conducted using large animal hearts (55±5 kg porcine, n = 6) to validate the adaptive controller’s performance for stepwise and fast switching AoP references. The results confirmed that the the proposed controller is able to regulate the AoP of an isolated porcine heart in an accurate (mean error less than 2 mmHg) and fast (4~8 s of settling time) manner.

Keywords: Automation in Life Sciences: Biotechnology, Pharmaceutical and Health Care, Biological Cell Manipulation, Automation at Micro-Nano Scales
Abstract: During immune surveillance, cytotoxic T lymphocytes (CTL) can selectively identify and destroy tumor cells by recognizing tumor-specific peptides (neoantigens), bound to major histocompatibility complex molecules (pMHC) arrayed on cancer cell surfaces. CTL use the same machinery to destroy virally infected cells displaying pathogen-specific pMHC, while leaving intact healthy cells expressing normal self-pMHC. We present a robotic microscope that allows scientists to conduct highly sensitive and selective T cell-pMHC studies with high throughput. Our system manipulates micro-meter beads coated with particular pMHC, presents them to T cells and generates piconewton level intermolecular forces required to detect T cell acuity with a neoantigen.Our systemintegrates optical tweezers, precision nanomicro stages, and episcopic/diascopic illumination schemes at two magnifications.We create a coordinate referencing system to locate translucent T cells in three-dimensional, over a large space, based on the characteristic intensity change at each focal plane caused by the cells. Our systemperforms automated experiments to detect the level of T cell acuity with specific pMHCs, by measuring the downstream cellular responses. High acuity T cells can be selectively recovered for single cell analysis. Our new methodology and tool will have a significant impact on cancer immunotherapy and immunology research.

Keywords: Automation Technologies for Smart Cities, Big Data in Robotics and Automation, Autonomous Agents
Abstract: Generating multi-vehicle trajectories from existing limited data can provide rich resources for autonomous vehicle development and testing. This paper introduces a multivehicle trajectory generator (MTG) that can encode multivehicle interaction scenarios (called driving encounters) into an interpretable representation from which new driving encounter scenarios are generated by sampling. The MTG consists of a bi-directional encoder and a multi-branch decoder. A new disentanglement metric is then developed for model analyses and comparisons in terms of model robustness and the independence of the latent codes. Comparison of our proposed MTG with β-VAE and InfoGAN demonstrates that the MTG has stronger capability to purposely generate rational vehicle-to-vehicle encounters through operating the disentangled latent codes. Thus the MTG could provide more data for engineers and researchers to develop testing and evaluation scenarios for autonomous vehicles.

Keywords: Force and Tactile Sensing, Deep Learning in Robotics and Automation, Recognition
Abstract: n-shot learning, i.e., learning a classifier from only few or even one training samples per class, is the ultimate goal in minimizing the cost of sample acquisition. This is esp. important for active sensing tasks like tactile material classification. Achieving high classification accuracy from only few samples is typically possible only when pre-knowledge is used. In n-shot transfer learning, knowledge from pre-training on a large knowledge set with many classes and samples per class has to be transferred to support the training for a given task set with only few samples per new class. In this paper, we show for the first time that deep end-to-end transfer learning is feasible for tactile material classification. Based on the previously presented (TactNet-II) [1], a deep con- volutional neural network (CNN) which reaches superhuman tactile classification performance, we adapt state-of-the art deep transfer learning methods. We evaluate the resulting deep n-shot learning methods with a publicly available tactile material data set with 36 materials [1] in a 6-way n-shot learning task with 30 materials in the knowledge set. In 1-shot learning, our deep transfer learning method reaches 75.5% classification accuracy and in 10-shot more than 90%, outperforming classification without knowledge transfer by more than 40%. This results in an up to 15 time reduction in the number of samples needed to reach a desired accuracy level.

Keywords: Force and Tactile Sensing, Haptics and Haptic Interfaces
Abstract: The sense of touch is arguably the first human sense to develop. Empowering robots with the sense of touch may augment their understanding of interacted objects and the environment beyond standard sensory modalities (e.g., vision).This paper investigates the effect of hybridizing touch and sliding movements for tactile-based texture classification. We develop three machine-learning algorithms within a framework to discriminate between surface textures; the first two methods use hand-engineered tactile features, whilst the third leverages convolutional and recurrent neural network layers to learn feature representations from raw data. To compare these methods, we constructed a dataset comprising tactile data from 23 textures gathered using the iCub platform under a loosely constrained setup, i.e., with nonlinear motion. In line with findings from neuroscience, our experiments show that a good initial estimate can be obtained via touch data, which can be further refined via sliding; combining both in our framework achieves a 98% accuracy over unseen data.

Keywords: Force and Tactile Sensing, Haptics and Haptic Interfaces, Sensor Fusion
Abstract: The integration of visual-tactile stimulus is common while humans performing daily tasks. In contrast, using unimodal visual or tactile perception limits the perceivable dimensionality of a subject. However, it remains a challenge to integrate the visual and tactile perception to facilitate robotic tasks. In this paper, we propose a novel framework for the cross-modal sensory data generation for visual and tactile perception. Taking texture perception as an example, we apply conditional generative adversarial networks to generate pseudo visual images or tactile outputs from data of the other modality. Extensive experiments on the ViTac dataset of cloth textures show that the proposed method can produce realistic outputs from other sensory inputs. We adopt the structural similarity index to evaluate similarity of the generated output and real data and results show that realistic data have been generated. Classification evaluation has also been performed to show that the inclusion of generated data can improve the perception performance. The proposed framework has potential to expand datasets for classification tasks, generate sensory outputs that are not easy to access, and also advance integrated visual-tactile perception.

Keywords: Force and Tactile Sensing
Abstract: Manipulation tasks often require robots to be continuously in contact with an object. Therefore tactile perception systems need to handle continuous contact data. Shear deformation causes the tactile sensor to output path-dependent readings in contrast to discrete contact readings. As such, in some continuous-contact tasks, sliding can be regarded as a disturbance over the sensor signal. Here we present a shear-invariant perception method based on principal component analysis (PCA) which outputs the required information about the environment despite sliding motion. A compliant tactile sensor (the TacTip) is used to investigate continuous tactile contact. First, we evaluate the method offline using test data collected whilst the sensor slides over an edge. Then, the method is used within a contour-following task applied to 6 objects with varying curvatures; all contours are successfully traced. The method demonstrates generalisation capabilities and could underlie a more sophisticated controller for challenging manipulation or exploration tasks in unstructured environments.

Keywords: Force and Tactile Sensing, Haptics and Haptic Interfaces
Abstract: Intrinsic Tactile Sensing (ITS) is a well-established technique, relying on force/torque and geometric surface description to find contact centroids. The method works well for rigid surfaces. However, finding a solution for deformable surfaces is an open issue. This work presents two solutions to extend ITS to deformable surfaces, relying on force-deformation characteristics of the surface under exploration: (i) a closed-form approach that calculates the contact centroid using standard ITS, but on a shrunk geometry approximating the deformed surface; (ii) an iterative procedure that takes into account soft surface deformation, and force/torque equilibrium to minimize a cost function. We have tested both using ellipsoid silicone specimens, with different softness levels and indented along different directions. Both linear and quadratic fitting for the force-indentation behavior were employed. The two methods have distinct advantages and limitations. However, a combination of two methods, using one to produce the initial guess for the other, turns out to be very effective. Indeed, in our validation this solution showed convergence under 1ms, attaining errors lower than 1 mm. The proposed approaches were implemented in a ROS-based toolbox, integrating both solutions.

Keywords: Force and Tactile Sensing, Haptics and Haptic Interfaces, Force Control
Abstract: Accurate force sensing is essential for skilled robot motion, while the limitation of the dynamic range is one of the major issues for force sensing. This paper deals with the development of a miniature six-axis force sensor with the aim of enabling robotic force detection across a high dynamic range (HDR). A miniaturized structure for the multistage structure of the HDR force sensor is designed by horizontally arranging two bodies: a low-rigidity thin beam inside a high-rigidity thick beam. The proposed sensor demonstrated performance superior to that of a conventional force sensor in the low load region, even with the inexpensive and lightweight resin material. Although creep and stress relaxation degrade the sensor's accuracy, these effects could be reduced by 80 percent with a correction filter.

Keywords: Social Human-Robot Interaction, Deep Learning in Robotics and Automation, AI-Based Methods
Abstract: Co-speech gestures enhance interaction experiences between humans as well as between humans and robots. Most existing robots use rule-based speech-gesture association, but this requires human labor and prior knowledge of experts to be implemented. We present a learning-based co-speech gesture generation that is learned from 52 h of TED talks. The proposed end-to-end neural network model consists of an encoder for speech text understanding and a decoder to generate a sequence of gestures. The model successfully produces various gestures including iconic, metaphoric, deictic, and beat gestures. In a subjective evaluation, participants reported that the gestures were human-like and matched the speech content. We also demonstrate a co-speech gesture with a NAO robot working in real time.

Keywords: Social Human-Robot Interaction, Physical Human-Robot Interaction, Human-Centered Robotics
Abstract: A robot cannot be warm and friendly – or can it? To explore whether a robot can be a medical receptionist, we developed a robotic system for interacting with patients at a doctor’s clinic, including acting friendly. We designed the robot to interact naturally with patients at the start and finish of a clinic visit. We investigated people’s perceptions to the robot in a wizard-of-Oz study, where the participants interacted with the robot over four interactions. 40 participants evaluated the robot. The results indicate the participants thought the robot could be a friendly receptionist, especially after repeated interactions with the robot. However, the participants mainly thought the robot was friendly in a “professional” way, rather than a personal friend.

Keywords: Social Human-Robot Interaction, Cognitive Human-Robot Interaction
Abstract: In this paper, we present an autonomous AI system designed for a Human-Robot Interaction (HRI) study, set around a dice game scenario. We conduct a case study to answer our research question: Does a robot with a socially engaged personality lead to a higher acceptance than a competitive personality? The flexibility of our proposed system allows us to construct and attribute two different personalities to a humanoid robot: a socially engaged personality that maximizes its user interaction and a competitive personality that is focused on playing and winning the game. We evaluate both personalities in a user study, in which the participants play a turn-taking dice game with the robot. Each personality is assessed with four different evaluation tools: 1) the Godspeed Questionnaire, 2) the Mind Perception Questionnaire, 3) a custom questionnaire concerning the overall HRI experience, and 4) a Convolutional Neural Network analyzing the emotions on the participants' facial feedback throughout the game. Our results show that the socially engaged personality evokes stronger emotions among the participants and is rated higher in likability and animacy than the competitive one. We conclude that designing the robot with a socially engaged personality contributes to a higher acceptance within an HRI scenario.

Keywords: Social Human-Robot Interaction, Autonomous Agents, Intelligent Transportation Systems
Abstract: While intelligence of autonomous vehicles (AVs) has significantly advanced in recent years, accidents involving AVs suggest that these autonomous systems lack gracefulness in driving when interacting with human drivers. In the setting of a two-player game, we propose model predictive control based on social gracefulness, which is measured by the discrepancy between the actions taken by the AV and those that could have been taken in favor of the human driver. We define social awareness as the ability of an agent to infer such favorable actions based on knowledge about the other agent's intent, and further show that empathy, i.e., the ability to understand others' intent by simultaneously inferring others' understanding of the agent's self intent, is critical to successful intent inference. Lastly, through an intersection case, we show that the proposed gracefulness objective allows an AV to learn more sophisticated behavior, such as passive-aggressive motions that gently force the other agent to yield.

Keywords: Social Human-Robot Interaction, Robot Companions, Human-Centered Robotics
Abstract: This paper presents an exploratory social Human-Robot Interaction (HRI) study that investigates and compares the persuasive effectiveness of robots attempting to influence a user with different behavior strategies. Ten multimodal persuasive strategies were uniquely designed based on Compliance Gaining Behaviors (CGBs). These persuasive strategies were then compared using two competing social robots attempting to influence a participant’s estimate during a jelly bean guessing game. The results of our exploratory study with 200 participants showed that affective and logical strategies had a higher potential for persuasive influence and warrant further research.

Keywords: Cognitive Human-Robot Interaction, Learning from Demonstration, Human Factors and Human-in-the-Loop
Abstract: During human-robot interaction (HRI), we want the robot to understand us, and we want to intuitively understand the robot. In order to communicate with and understand the robot, we can leverage interactions, where the human and robot observe each other's behavior. However, it is not always clear how the human and robot should interpret these actions: a given interaction might mean several different things. Within today's state-of-the-art, the robot assigns a single interaction strategy to the human, and learns from or teaches the human according to this fixed strategy. Instead, we here recognize that different users interact in different ways, and so one size does not fit all. Therefore, we argue that the robot should maintain a distribution over the possible human interaction strategies, and then infer how each individual end-user interacts during the task. We formally define learning and teaching when the robot is uncertain about the human's interaction strategy, and derive solutions to both problems using Bayesian inference. In examples and a benchmark simulation, we show that our personalized approach outperforms standard methods that maintain a fixed interaction strategy.

Keywords: Localization, Object Detection, Segmentation and Categorization, SLAM
Abstract: Recent researches demonstrate that self-localization performance is a very useful measure of likelihood-of-change (LoC) for change detection. In this paper, this ``detection-by-localization" scheme is studied in a novel generalized task of object-level change detection. In our framework, a given query image is segmented into object-level subimages (termed ``scene parts"), which are then converted to subimage-level pixel-wise LoC maps via the detection-by-localization scheme. Our approach models a self-localization system as a ranking function, outputting a ranked list of reference images, without requiring relevance score. Thanks to this new setting, we can generalize our approach to a broad class of self-localization systems. We further propose an aggregation of different self-localization results from different queries so as to achieve higher precision. Our ranking based self-localization model allows to fuse self-localization results from different modalities via an unsupervised rank fusion derived from a field of multi-modal information retrieval (MMR). Our framework does not rely on the raw-score-merging hypothesis. Challenging experiments of cross-season change detection using the publicly available North Campus Long-Term (NCLT) dataset validates the efficacy of our proposed method.

Keywords: Object Detection, Segmentation and Categorization, Learning and Adaptive Systems
Abstract: to utilize the state-of-the-art object recognition/detection/segmentation methods to real applications, there are two big blocks. First, most of the deep learning models heavily depend on large amounts of labeled training data, which is expensive to obtain for each individual application. Second, the object categories must be pre-defined in the dataset, thus not practical to scenarios with varying object categories. To alleviate the reliance on pre-defined big data, this paper proposes a customized object recognition and segmentation method. It aims to recognize and segment any object defined by the user, given only one annotation. There are three steps in the proposed method. First, the robot takes an exemplar video of the target object, the user defines the object name, and mask its boundary on only one frame. Then the robot automatically propagates the annotation through the exemplar video based on a proposed data generation method. In the meantime, a segmentation model continuously updates itself on the generated data. Finally, only a light segmentation net is required at testing stage, to recognize and segment any object that the user defines.

Keywords: Object Detection, Segmentation and Categorization, RGB-D Perception, Sensor Fusion
Abstract: This paper proposes a SEG-VoxelNet that takes RGB images and LiDAR point clouds as inputs for accurately detecting 3D vehicles in autonomous driving scenarios, which for the first time introduces semantic segmentation technique to assist the 3D LiDAR point cloud based detection. Specifically, SEG-VoxelNet is composed of two sub-networks: an image semantic segmentation network (SEG-Net) and an improved-VoxelNet. The SEG-Net generates the semantic segmentation map which represents the probability of the category for each pixel. The improved-VoxelNet is capable of effectively fusing point cloud data with image semantic feature and generating accurate 3D bounding boxes of vehicles. Experiments on the KITTI 3D vehicle detection benchmark show that our approach outperforms the methods of state-of-the-art.

Keywords: Object Detection, Segmentation and Categorization, Sensor Fusion, Deep Learning in Robotics and Automation
Abstract: For autonomous driving applications it is critical to know which type of road users and road side infrastructure are present to plan driving manoeuvres accordingly. Therefore autonomous cars are equipped with different sensor modalities to robustly perceive its environment. However, for classification modules based on machine learning techniques it is challenging to overcome unseen sensor noise. This work presents an object classification module operating on unsupervised learned multi-modal features with the ability to overcome gradual or total sensor failure. A two stage approach composed of an unsupervised feature training and a uni-modal and multi-modal classifiers training is presented. We propose a simple but effective decision module switching between uni-modal and multi-modal classifiers based on the closeness in the feature space to the training data. Evaluations on the ModelNet 40 data set show that the proposed approach has a 14% accuracy gain compared to a late fusion approach operating on a noisy point cloud data and a 6% accuracy gain when operating on noisy image data.

Keywords: Object Detection, Segmentation and Categorization, Semantic Scene Understanding, AI-Based Methods
Abstract: Earlier work demonstrates the promise of deep-learning-based approaches for point cloud segmentation; however, these approaches need to be improved to be practically useful. To this end, we introduce a new model SqueezeSegV2. With an improved model structure, SqueezeSetV2 is more robust against dropout noises in LiDAR point cloud and therefore achieves significant accuracy improvement. Training models for point cloud segmentation requires large amounts of labeled data, which is expensive to obtain. To sidestep the cost of data collection and annotation, simulators such as GTA-V can be used to create unlimited amounts of labeled, synthetic data. However, due to domain shift, models trained on synthetic data often do not generalize well to the real world. Existing domain-adaptation methods mainly focus on images and most of them cannot be directly applied to point clouds. We address this problem with a domain-adaptation training pipeline consisting of three major components: 1) learned intensity rendering, 2) geodesic correlation alignment, and 3) progressive domain calibration. When trained on real data, our new model exhibits segmentation accuracy improvements of 6.0-8.6% over the original SqueezeSeg. When training our new model on synthetic data using the proposed domain adaptation pipeline, we nearly double test accuracy on real-world data, from 29.0% to 57.4%. Our source code and synthetic dataset are open sourced.

Keywords: Computer Vision for Automation, Deep Learning in Robotics and Automation, Object Detection, Segmentation and Categorization
Abstract: Automated factories use deep learning-based vision systems to accurately detect various products. However, training such vision systems, requires manual annotation of a significant amount of data to optimize the large number of parameters of the deep convolutional neural networks. Such manual annotation is very time-consuming and laborious. To reduce this burden, we propose a fully automated annotation approach without any manual intervention. To do this, we associate one visual marker with one object and capture them in the same image. However, if an image showing the marker is used for training, normally the neural network learns the marker as a feature of the object. By hiding the marker with a noise mask, we succeeded in reducing this erroneous learning. Experiments verified the effectiveness of the proposed method in comparison with manual annotation, both in terms of the time needed to collect training data and the resulting detection accuracy of the vision system. The time required for data collection was reduced from 16.1 hours to 1.87 hours. The accuracy of the vision system trained with the proposed method was 87.3%, which is higher than the accuracy of a vision system trained with the manual method.

Keywords: Computer Vision for Automation, Calibration and Identification, Surveillance Systems
Abstract: It is necessary to employ smart sensory systems in dynamic and mobile workspaces where industrial robots are mounted on mobile platforms. Such systems should be aware of flexible and non-stationary workspaces and able to react autonomously to changing situations. Building upon our previously presented RoPose-system [1], which employs a convolutional neural network architecture that has been trained on pure synthetic data to estimate the kinematic chain of an industrial robot arm system, we now present RoPose-Real. RoPose-Real extends the prior system with a comfortable and targetless extrinsic calibration tool, to allow for the production of automatically annotated datasets for real robot systems. Furthermore, we use the novel datasets to train the estimation network with real world data. The extracted pose information is used to automatically estimate the observing sensor pose relative to the robot system. Finally we evaluate the performance of the presented subsystems in a real world robotic scenario.

Keywords: Learning and Adaptive Systems, Visual Learning, RGB-D Perception
Abstract: The development of high-performance perception for mobile robotic agents is still challenging. Learning appropriate perception models usually requires extensive amounts of labeled training data that ideally follows the same distribution as the data an agent will encounter in its target task. Recent developments in gaming industry led to game engines able to generate photorealistic environments in real-time, which can be used to realistically simulate the sensory input of an agent.

We propose a novel framework which allows the definition of different learning scenarios and instantiates these scenarios in a high quality game engine where a perceptual agent can act and learn in. The scenarios are specified in a newly developed scenario description language that allows the parametrization of the virtual environment and the perceptual agent. New scenarios can be sampled from a task-specific object distribution that allows the automatic generation of extensive amounts of different learning environments for the perceptual agent.

We will demonstrate the plausibility of the framework by conducting object recognition experiments on a real robotic system which has been trained within our framework.

Keywords: Factory Automation, Industrial Robots, RGB-D Perception
Abstract: Fast Graspability Evaluation (FGE) has been proposed as a method for detecting grasping positions on objects and is now being used for industrial robots. FGE uses convolution of hand templates with regions on the target object to estimate the optimum grasping posture. However, the hand opening width and rotation angles must be set with high resolution to achieve highly accurate results and the computational load is high. To address that issue, we propose a method in which hand templates are represented in compact form for faster processing by using singular value decomposition. Applying singular value decomposition enables hand templates to be represented as linear combinations of a small number of eigenvalue templates and eigenfunctions. Eigenfunctions take discrete values, but response values can be calculated with arbitrary parameters by fitting a continuous function. Experimental results show that the proposed method reduces computation time by two thirds while maintaining the same detection accuracy as conventional FGE for both parallel hands and three-finger hands.

Keywords: Demining Systems, Field Robots
Abstract: Coverage performance in a coverage path planning problem depends both on the path created and on the footprint of the sensor used. The footprint can be increased either by increasing the size of the sensor, or by mounting the sensor on a robotic arm to allow scanning over larger areas as the platform moves, effectively creating a virtual sensor with a larger footprint than the physical sensor's. However, the virtual footprint comes at a cost requiring formulating an optimization problem for the area of interest. In this work, three common strategies to use a metal detector on a platform are discussed, their time and energy performances are formulated and the corresponding optima are found.

Keywords: Flexible Robots, Biologically-Inspired Robots, Motion and Path Planning
Abstract: Fruitful developments on continuum robots have been witnessed in recent years due to their movements and manipulation capabilities in confined spaces. Due to the nature that a continuum robot has an infinite number of DoFs (Degrees of Freedom), majority of the existing systems deployed abundant actuators such that the robot can be controlled in separately modeled and actuated segments with constant or variable curvature. As the shape of a continuum robot is always jointly determined by its actuation and the interactions from the environment, it is hence worth exploring the opposite approach that how a task can be accomplished with a minimal number of actuators. This paper presents the first step of such an investigation where a slim 3-actuator continuum robot is controlled to reach different spatial locations under gravity. As the gravity greatly affects the robot’s shape, a monocular camera, together with two UKFs (Unscented Kalman Filters), was used to concurrently estimate the robot’s shape and the feature depth. Then the estimated shape can be used in updating the kinematics model of the robot to achieve motion control. Experiments were conducted to validate the efficacy of the proposed shape estimation, which promises the motion control implementation in the near future.

Keywords: Computer Vision for Transportation, Human Detection and Tracking
Abstract: This paper presents a novel dataset titled PedX, a large-scale multimodal collection of pedestrians at complex urban intersections. PedX consists of more than 5,000 pairs of high-resolution (12MP) stereo images and LiDAR data along with providing 2D and 3D labels of pedestrians. We also present a novel 3D model fitting algorithm for automatic 3D labeling harnessing constraints across different modalities and novel shape and temporal priors. All annotated 3D pedestrians are localized into the real-world metric space, and the generated 3D models are validated using a mocap system configured in a controlled outdoor environment to simulate pedestrians in urban intersections. We also show that the manual 2D labels can be replaced by state-of-the-art automated labeling approaches, thereby facilitating automatic generation of large scale datasets.

Keywords: Underactuated Robots, Tendon/Wire Mechanism, Biologically-Inspired Robots
Abstract: This paper presents the development of a tendon-driven continuum robot segment with a modular design, simple construction and significant lifting capabilities. The segment features a continuous flexible core combined with rigid interlocking vertebrae evenly distributed along its length. This design allows bending in two degrees of freedom while minimising torsional movement. The segment is actuated by two antagonistic tendon pairs, each of which is driven by a single geared DC motor. Modularity is achieved by embedding these motors in one end of the segment, avoiding the need for a bulky actuation unit and allowing variable numbers of segments to be connected. The design features a large hollow central bore which could be used as a vacuum channel for suction-assisted gripping or to allow ingress and egress of fluids. The design process goes through four iterations, the final two of which are subjected to quantitative experiments to evaluate workspace, lifting capabilities and torsional rigidity. All iterations are fabricated using multi-material 3D printing, which allows the entire structure to be printed as a pre-assembled unit with the rigid vertebrae fused to the flexible core. Assembly is then a simple case of inserting the motors and connecting the tendons. This unconventional manufacturing approach is found to be efficient, effective and relatively cheap.

Keywords: Underactuated Robots, Swarms, Automation at Micro-Nano Scales
Abstract: Consider many particles actuated by a uniform global external field (e.g. gravitational or magnetic fields). This paper presents analytical results using workspace obstacles and global inputs to reshape such a group of particles. Shape control of many particles is necessary for conveying information, construction, and navigation. First we show how the particles' characteristic angle of repose can be used to reshape the particles by controlling angle of attack and the magnitude of the driving force. These can then be used to control the force and torque applied to a rectangular rigid body. Next, we examine the full set of stable, achievable mean and variance configurations for the shape of a particle group in two canonical environments: a square and a circular workspace. Finally, we show how workspaces with linear boundary layers can be used to achieve a more rich set of mean and variance configurations.

Keywords: Motion and Path Planning, Underactuated Robots, Optimization and Optimal Control
Abstract: While much research has been conducted into the generation of smooth trajectories for underactuated unstable aerial vehicles such as quadrotors, less attention has been paid to the application of the same techniques to ground based omnidirectional dynamically balancing robots. These systems have more control authority over their linear accelerations than aerial vehicles, meaning trajectory smoothness is less of a critical design parameter. However, when operating in indoor environments these systems must often adhere to relatively low velocity constraints, resulting in very conservative trajectories when enforced using existing trajectory optimisation methods.

This paper makes two contributions; this gap is bridged by the extension of these existing methods to create a fast velocity constrained trajectory planner, with trajectory timing characteristics derived from the optimal minimum-time solution of a simplified acceleration and velocity constrained model. Next, a differentially flat model of an omnidirectional balancing robot utilizing a collinear Mecanum drive is derived, which is used to allow an experimental prototype of this configuration to smoothly follow these velocity constrained trajectories.

Keywords: Underactuated Robots, Flexible Robots, Dynamics
Abstract: In this contribution the problem of vibration control is studied on the basis of a fundamental oscillatory system consisting of a mass spring system and an additional mass. The proposed control strategy couples the orbits of the two masses such that both masses stop, while simultaneously stabilizing the second mass to a desired equilibrium. Using a coordinate and input transformation, the control strategy is directly transferred to an n-link manipulator mounted on a base with linear translational stiffness. Using semidefinite Lyapunov functions and a conditional stability argument, it is shown that the proposed control strategy damps out base vibrations, while additionally achieving a desired configuration in the task-space. Finally, the proposed method is compared to a state-of-the-art approach using numerical simulations.

Keywords: Model Learning for Control, Underactuated Robots, Learning and Adaptive Systems
Abstract: Control of underactuated balance robot requires external subsystem trajectory tracking and internal unstable subsystem balancing with limited control authority. We present a learning-based control approach for underactuated balance robots. The tracking and balancing control is designed the controller in fast- and slow-time scales. In the slow-time scale, model predictive control is adopted to plan desired internal state profile to achieve external trajectory tracking task. The internal state is then stabilized around the planned profile in the fast-time scale. The control design is based on a learned Gaussian process (GP) regression model without need of a priori knowledge about the robot dynamics. The controller also incorporates the GP model predicted variance to enhance robustness to modeling errors. Experiments are presented using a Furuta pendulum system.

Keywords: Underactuated Robots, Multi-Robot Systems, Swarms
Abstract: Abstract—This paper investigates strategies for multi-agent 3D position control using a shared control input and selfpropelled agents. The only control inputs allowed are rotation commands that rotate all agents by the same rotation matrix. In the 2D case, only two degrees-of-freedom (DOF) in position are controllable. We review controllability results in 2D, and then show that interesting things happen in 3D. We provide control laws for steering up to nine DOF in position, which can be mapped in various ways, including to control the x, y, z position of three agents, make four agents meet, or reduce the spread of n agents.

Keywords: Control Architectures and Programming, Industrial Robots, Social Human-Robot Interaction
Abstract: Collaborative robots (cobots) are a category of robots designed to work together with humans. By combining the fortes of the robot, such as precision and strength, with the dexterity and problem-solving ability of the human, it is possible to accomplish tasks that cannot be fully automated and improve the production quality and working conditions of employees. This article presents the results of the ClaXon project, which studies and implements interactions between humans and cobots in factories. The project has led to the integration of a cobot in the Audi car manufacturing plant in Brussels, Belgium. Proofs of concept were realized to study multimodal perceptions for human–robot interaction. The project addressed technical challenges regarding the introduction of cobots on the factory floor. Social experiments were conducted with factory workers to assess the social acceptance of cobots and to study the interactions between human and robot.

Keywords: Physical Human-Robot Interaction, Parallel Robots, Physically Assistive Devices
Abstract: In this paper we propose a novel passive mechanism and a macro-mini architecture for effective and intuitive physical human-robot interaction (pHRI). The macro-mini concept allows the use of a mini low-impedance passive mechanism (LIP) to effortlessly and intuitively control a macro high-impedance active (HIA) system such as a gantry manipulator. The proposed mini LIP design is based on a three-degree-of-freedom (3-dof) translational parallel mechanism, which makes it simple and compact, thereby adding little inertia to end-effector of the macro HIA mechanism. The kinematically and statically decoupled LIP mechanism is first described and analysed. Then, the kinematics of the macro-mini architecture is studied in order to establish the capabilities of the robot. A controller is then proposed that uses the passive joint coordinates of the LIP mechanism as input to control the motion of the HIA mechanism. Finally, experimental results are provided in order to illustrate the performance and intuitive behaviour of the robot, which is particularly suited for manufacturing applications.

Keywords: Intelligent and Flexible Manufacturing, Cognitive Human-Robot Interaction, AI-Based Methods
Abstract: Current market demands require an increasingly agile production environment throughout many manufacturing branches. Traditional automation systems and industrial robots, on the other hand, are often too inflexible to provide an economically viable business case for companies with rapidly changing products. The introduction of cognitive abilities into robotic and automation systems is, therefore, a necessary step toward lean changeover and seamless human-robot collaboration. In this article, we introduce the European Union (EU)-funded research project SMErobotics, which focuses on facilitating the use of robot systems in small and medium-sized enterprises (SMEs). We analyze open challenges for this target audience and develop multiple efficient technologies to address related issues. Real-world demonstrators of several end users and from multiple application domains show the impact these smart robots can have on SMEs. This article intends to give a broad overview of the research conducted in SMErobotics. Specific details of individual topics are provided through references to our previous publications.

Keywords: Software, Middleware and Programming Environments, Control Architectures and Programming, Social Human-Robot Interaction
Abstract: For many service robots, reactivity to changes in their surroundings is a must. However, developing software suitable for dynamic environments is difficult. Existing robotic middleware allows engineers to design behavior graphs by organizing communication between components. But because these graphs are structurally inflexible, they hardly support the development of complex reactive behavior. To address this limitation, we propose Playful, a software platform that applies reactive programming to the specification of robotic behavior. The front-end of Playful is a scripting language which is simple (only five keywords), yet results in the runtime coordinated activation and deactivation of an arbitrary number of higher-level sensory-motor couplings. When using Playful, developers describe actions of various levels of abstraction via behaviors trees. During runtime an underlying engine applies a mixture of logical constructs to obtain the desired behavior. These constructs include conditional ruling, dynamic prioritization based on resources management and finite state machines. Playful has been successfully used to program an upper-torso humanoid manipulator to perform lively interaction with any human approaching it.

Keywords: Social Human-Robot Interaction, Virtual Reality and Interfaces, Visual Tracking
Abstract: In this paper, we present a communication paradigm using a context-aware mixed reality approach for instructing human workers when collaborating with robots. The main objective of this approach is to utilize the physical work environment as a canvas to communicate task-related instructions and robot intentions in the form of visual cues. A vision-based object tracking algorithm is used to precisely determine the pose and state of physical objects in and around the workspace. A projection mapping technique is used to overlay visual cues on the tracked objects and the workspace. Simultaneous tracking and projection onto objects enable the system to provide just-in-time instructions for carrying out a procedural task. Additionally, the system can also inform and warn humans about the intentions of the robot and safety of the workspace. We hypothesized that using this system for executing a human-robot collaborative task will improve the overall performance of the team and provide a positive experience to the human partner. To test this hypothesis, we conducted an experiment involving human subjects and compared the performance (both objective and subjective) of the presented system with conventional forms of communication, namely printed and mobile display instructions. We found that projecting visual cues enabled human subjects to collaborate more effectively with the robot and resulted in higher efficiency in completing the task.

Keywords: Physically Assistive Devices, Prosthetics and Exoskeletons, Human Performance Augmentation
Abstract: The work presented in this paper is a lower-back exoskeleton prototype to provide back support for industrial workers, who are required to manually handle heavy materials. Reducing spinal loads during these tasks can thereby reduce the risk of work-related back injuries. Biomechanical studies show that compression of the lumbar spine is a key risk factor for musculoskeletal injuries. To address this issue, we present a wearable exoskeleton designed to provide back support and reduce lumbar spine compression. To provide effective assistance and avoid injury to muscles or tendons, we aim to apply a continuous torque of approximately 40 Nm on both hip joints, to actively assist both abduction/adduction (HAA) and flexion/extension (HFE). Each actuation unit includes a modular and compact series-elastic actuator (SEA) with a clutch. It provides mechanical compliance at the interface between the exoskeleton and the user, and the clutches can automatically disengage the torque between the exoskeleton and the user. These experimental results show that the exoskeleton can reduce lumbar compression by reducing the need for muscular activity in spine. Furthermore, powering both HFE and HAA can effectively reduce the lumbar spinal loading user experience when lifting and lowering objects while in a twisted posture.

Keywords: Planning, Scheduling and Coordination, Path Planning for Multiple Mobile Robots or Agents, Multi-Robot Systems
Abstract: The article proposes a new heuristic for task allocation and routing of heterogeneous robots. Specifically, we consider a path planning problem where there are two (structurally) heterogeneous robots that start from distinctive depots and a set of targets to visit. The objective is to find a tour for each robot in a manner that enables each target location to be visited at least once by one of the robots while minimizing the maximum travel cost. A solution for Multiple Depot Heterogeneous Traveling Salesman Problem (MDHTSP) with min-max objective is in great demand with many potential applications, because it can significantly reduce the job completion duration. However, there are still no reliable algorithms that can run in short amount of time. As an initial idea of solving min-max MDHTSP, we present a heuristic based on a primal-dual technique that solves for a case involving two robots while focusing on task allocation. Based on computational results of the implementation, we show that the proposed algorithm produces a good quality of feasible solution within a relatively short computation time.

Keywords: Planning, Scheduling and Coordination, Task Planning, Multi-Robot Systems
Abstract: We consider the problem of minimizing the total duration of missions by heterogeneous vehicles. The problem contains constraints related to vehicles’ capabilities and energy. The goal is to determine the best routes of each vehicle deployed by choosing which waypoints to pass and which observations to perform. Each vehicle has a particular distance matrix and a limited energy. In order to provide high quality solutions within reasonable computational time, two decomposition-based approximate methods were implemented: (i) the Multiphase heuristic, and (ii) the Two-Phase iterative heuristic. The performance of the methods is evaluated against the Branch-and-Cut algorithm using generated instances.

Keywords: Formal Methods in Robotics and Automation, Distributed Robot Systems, Process Control
Abstract: Designing robot controllers that correctly react to changes in the environment is a time-consuming and error-prone process. An alternative is to use “correct-by- construction” synthesis approaches to automatically generate controller designs from high-level specifications. In particular, Generalized Reactivity(1) or GR(1) specifications are well-suited to express specifications for robots that must act in dynamic environments, and approaches to generate controller designs from GR(1) specifications are highly computationally efficient.

Toward that end, this paper presents Salty, a domain-specific language for GR(1) specifications. While tools exist to synthesize system designs from GR(1) specifications, Salty makes such specifications easier to write and debug by supporting features such as richer input and output types, user-defined macros, common specification patterns, and specification optimization and sanity checking. Salty interfaces with the separately developed synthesis tool Slugs to produce a system or controller design, and Salty translates this design to a software implementation in a variety of languages. We demonstrate Salty on an application involving coordination of multiple unmanned air vehicles (UAVs) and provide a workflow for connecting synthesized UAV controllers to freely available UAV planning and simulation software suites UxAS and AMASE.

Keywords: Task Planning, Mapping, Motion and Path Planning
Abstract: This paper proposes a method to deploy teams of robots with constrained energy capacities to persistently maintain a map of an uncertain environment. Typical occupancy map approaches assume a static world; however, we introduce a decay in confidence that degrades the occupancy probability of grid cells and promotes revisitation. Further, sections of the map whose occupancy differs between observations are visited more frequently, while unchanging areas are scheduled less frequently. While naive planning is intractable through the entire space of multi-agent spatio-temporal states, the proposed algorithm decouples planning such that constraints are resolved separately by solving reasonable subproblems. We evaluate this approach in simulation and show how the uncertainty of our world model is maintained below an acceptable threshold while the algorithm retains a tractable computation time.

Keywords: Networked Robots, Deep Learning in Robotics and Automation, Learning and Adaptive Systems
Abstract: The growing demand of industrial, automotive and service robots presents a challenge to the centralized Cloud Robotics model in terms of privacy, security, latency, bandwidth, and reliability. In this paper, we present a `Fog Robotics' approach to deep robot learning that distributes compute, storage and networking resources between the Cloud and the Edge in a federated manner. Deep models are trained on non-private (public) synthetic images in the Cloud; the models are adapted to the private real images of the environment at the Edge within a trusted network and subsequently, deployed as a service for low-latency and secure inference/prediction for other robots in the network. We apply this approach to surface decluttering, where a mobile robot picks and sorts objects from a cluttered floor by learning a deep object recognition and a grasp planning model. Experiments suggest that Fog Robotics can improve performance by sim-to-real domain adaptation in comparison to exclusively using Cloud or Edge resources, while reducing the inference cycle time by 4x in decluttering with 185 objects over 213 grasp attempts.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Networked Robots
Abstract: In several multirobot applications in which communication is limited, the mission could require the robots to iteratively take coordinated joint decisions on how to spread out in the environment and on how to reconnect with each other to share data and compute plans. Exploration and surveillance are examples of these applications. In this paper, we consider the problem of computing robots' paths on a graph-represented environment for restoring connections at minimum traveling cost. We call it the multirobot reconnection problem, we show its NP-hardness and hardness of approximation on some important classes of graphs, and we provide optimal and heuristic algorithms to solve it in practical settings. The techniques we propose are then exploited to derive a new efficient planning algorithm for a relevant connectivity-constrained multirobot planning problem addressed in the literature, the multirobot informative path planning with periodic connectivity problem.

Keywords: Formal Methods in Robotics and Automation, Manipulation Planning
Abstract: When humans and robots perform complex tasks together, the robot must have a strategy to choose its actions based on observed human behavior. One well-studied approach for finding such strategies is reactive synthesis. Existing approaches for finite-horizon tasks have used an explicit state approach, which incurs high runtime. In this work, we present a compositional approach to perform synthesis for finite-horizon tasks based on binary decision diagrams. We show that for pick-and-place tasks, the compositional approach achieves exponential speed-ups compared to previous approaches. We demonstrate the synthesized strategy on a UR5 robot.

Keywords: Task Planning, Motion and Path Planning, Manipulation Planning
Abstract: We propose a novel approach to Combined Task and Motion Planning (TAMP) under partial observability. Previous optimization-based TAMP methods compute optimal plans and paths assuming full observability. However, partial observability requires the solution to be a policy that reacts to the observations that the agent receives. We consider a formulation where observations introduce additional branching in the symbolic decision tree. The solution is now given by a reactive policy on the symbolic level together with a path tree that describes the branchings of optimal motion depending on the observations. Our method works in two stages: First, the symbolic policy is optimized using approximate path costs estimated from independent optimizations of trajectory pieces. Second, we fix the best symbolic policy and optimize a joint trajectory tree. We test our approach on object manipulation and autonomous driving examples. We also compare the algorithm’s performance to a state-of-the-art TAMP planner in fully observable cases.

Keywords: Deep Learning in Robotics and Automation, Perception for Grasping and Manipulation, Sensor-based Control
Abstract: Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. Results for simulated and real robot experiments are presented.

Keywords: Force and Tactile Sensing, Deep Learning in Robotics and Automation
Abstract: Estimation of tactile properties from vision, such as slipperiness or roughness, is important to effectively interact with the environment. These tactile properties help us decide which actions we should choose and how to perform them. E.g., we can drive slower if we see that we have bad traction or grasp tighter if an item looks slippery. We believe that this ability also helps robots to enhance their understanding of the environment, and thus enables them to tailor their actions to the situation at hand. We therefore propose a model to estimate the degree of tactile properties from visual perception alone (e.g., the level of slipperiness or roughness). Our method extends a encoder-decoder network, in which the latent variables are visual and tactile features. In contrast to previous works, our method does not require manual labeling, but only RGB images and the corresponding tactile sensor data. All our data is collected with a webcam and uSkin tactile sensor mounted on the end-effector of a Sawyer robot, which strokes the surfaces of 25 different materials. We show that our model generalizes to materials not included in the training data by evaluating the feature space, indicating that it has learned to associate important tactile properties with images.

Keywords: Deep Learning in Robotics and Automation, Computer Vision for Transportation, Autonomous Vehicle Navigation
Abstract: Deep learning has revolutionized the ability to learn "end-to-end" autonomous vehicle control directly from raw sensory data. While there have been recent advances on extensions to handle forms of navigation instruction, these works are unable to capture the full distribution of possible actions that could be taken and to reason about localization of the robot within the environment. In this paper, we extend end-to-end driving networks with the ability to understand maps. We define a novel variational network capable of learning from raw camera data of the environment as well as higher level roadmaps to predict (1) a full probability distribution over the possible control commands; and (2) a deterministic control command capable of navigating on the route specified within the map. Additionally, we formulate how our model can be used to localize the robot according to correspondences between the map and the observed visual road topology, inspired by the rough localization that human drivers can perform. We evaluate our algorithms on real-world driving data, and reason about the robustness of the inferred steering commands under various types of rich driving scenarios. In addition, we evaluate our localization algorithm over a new set of roads and intersections which the model has never driven through and demonstrate rough localization in situations without any GPS prior.

Keywords: Marine Robotics, Motion and Path Planning, Field Robots
Abstract: Motion planning for underwater vehicles must consider the effect of ocean currents. We present an efficient method to compute reachability and cost between sample points in sampling-based motion planning that supports long-range planning over hundreds of kilometres in complicated flows. The idea is to search a reduced space of control inputs that consists of stream functions whose level sets, or streamlines, optimally connect two given points. Such stream functions are generated by superimposing a control input onto the underlying current flow. A streamline represents the resulting path that a vehicle would follow as it is carried along by the current given that control input. We provide rigorous analysis that shows how our method avoids exhaustive search of the control space, and demonstrate simulated examples in complicated flows including a traversal along the east coast of Australia, using actual current predictions, between Sydney and Brisbane.

Keywords: Marine Robotics, Cooperative Manipulators
Abstract: This paper addresses the problem of cooperative object transportation for multiple Underwater Vehicle Manipulator Systems (UVMSs) in a constrained workspace involving static obstacles. We propose a Nonlinear Model Predictive Control (NMPC) approach for a team of UVMSs in order to transport an object while avoiding significant constraints and limitations such as: kinematic and representation singularities, obstacles within the workspace, joint limits and control input saturations. More precisely, by exploiting the coupled dynamics between the robots and the object, and using certain load sharing coefficients, we design a distributed NMPC for each UVMS in order to cooperatively transport the object within the workspace’s feasible region. Moreover, the control scheme adopts load sharing among the UVMSs according to their specific payload capabilities. Additionally, the feedback relies on each UVMS’s locally measurements and no explicit data is exchanged online among the robots, thus reducing the required communication bandwidth. Finally, real-time simulation results conducted in UwSim dynamic simulator running in ROS environment verify the efficiency of the theoretical finding.

Keywords: Robust/Adaptive Control of Robotic Systems, Multi-Robot Systems, Marine Robotics
Abstract: We propose a feedback control system for a reconfigurable multi-vessel platform. The platform consists of N propeller-driven vessels each of which is capable of latching to another vessel to form a rigid body of connected vessels. The main technical challenges are that i) depending on configurations of the platform the dynamic model would be different, and ii) the number of control variables in control system design increases as does the total number of vessels in the platform. To address these challenges, we develop a coordinated robust control scheme. Through experiments we assess trajectory tracking and disturbance attenuation performance of the control scheme in various configurations of the platform. Experiment results yield that average position and orientation tracking error are approximately 0.09 m and 3 degree, and the maximum tracking error-to-disturbance ratio is 1.12.

Keywords: Marine Robotics, Sensor-based Control, Multi-Robot Systems
Abstract: In this paper, we present a method for depth control of one degree of freedom (1DOF) underwater robotic platform aMussel, based on the measurements from the ambient light sensor. Since ambient light values change during the day and depend on the weather conditions, references for the controller are acquired from other aMussel holding depth using pressure sensor based controller. Control inputs are transmitted using acoustic communication.

Keywords: Marine Robotics, Motion and Path Planning, Mobile Manipulation
Abstract: This paper presents an unified motion planning approach for an Intervention Autonomous Underwater Vehicle (I-AUV) in a cluttered environment with localization uncertainty. With the uncertainty being propagated by an information filter, a trajectory optimization problem closed by a Linear-Quadratic-Gaussian controller is formulated for a coupled design of optimal trajectory, localization, and control. Due to the presence of obstacles or complexity of the cluttered environment, a set of feasible initial I-AUV trajectories covering multiple homotopy classes are required by optimization solvers. Parameterized through polynomials, the initial base trajectories are from solving quasi-quadratic optimization problems that are linearly constrained by waypoints from RRTconnect, while the initial trajectories of the manipulator are generated by a null space saturation controller. Simulations on an I-AUV with a 3 DOF manipulator in cluttered underwater environments demonstrated that initial trajectories are generated efficiently and that optimal and collision-free I-AUV trajectories with low state uncertainty are obtained.

Keywords: Marine Robotics, Soft Material Robotics, Biologically-Inspired Robots
Abstract: Remoras employ their adhesive discs to rapidly attach to and detach from a wide range of marine surfaces. By analyzing high-speed images of remoras’ (Echeneis naucrates) hitchhiking behavior, we describe the fish’s detachment mechanism as a lip curling up to break the seal between the disc and substrate. By mimicking the kinematic and morphological properties of the biological disc, we fabricated a multi-material biomimetic disc (whose stiffness spans four orders of magnitude) that is capable of both attachment and detachment. Detachment is realized by a flexible cable-driven mechanism that curls the anterior region of the silicone soft lip, allows leakage under the disc, and equalizes the internal pressure to the external pressure. The disc lamellae with attached carbon fiber spinules can be rotated by hydraulic soft actuators whose internal pressure is precisely tuned to the ambient underwater pressure. During attachment, increasing the rotational angle of the lamellae and the preload of the disc significantly enhanced the adhesive forces. We found that curling up the soft lip and folding down the lamellae rapidly reduced the pulling force of the disc by a factor of 254 compared to that under the attached state, which lead to detachment. Based on these mechanisms, underwater maneuvers involving repeated attachment and detachment were demonstrated with an integrated ROV unit that had a self-contained actuation and control system for the disc.

Keywords: Marine Robotics, Social Human-Robot Interaction, Cognitive Human-Robot Interaction
Abstract: In this paper, we propose a novel method for underwater robot-to-human communication using the motion of the robot as "body language". To evaluate this system, we develop simulated examples of the system's body language gestures, called kinemes, and compare them to a baseline system using flashing colored lights through a user study. Our work shows evidence that motion can be used as a successful communication vector which is accurate, easy to learn, and quick enough to be used, all without requiring any additional hardware to be added to our platform. We thus contribute to "closing the loop" for human-robot interaction underwater by proposing and testing this system, suggesting a library of possible body language gestures for underwater robots, and offering insight on the design of nonverbal robot-to-human communication methods.

Keywords: Marine Robotics, Biologically-Inspired Robots, Dynamics
Abstract: Three-dimensionally (3D) maneuverable robotic fish are highly desirable due to their abilities to explore and survey the underwater environment. Existing depth control mechanism is focused on using compressed air or piston to generate volume change, which makes the system bulky and impractical in a small size underwater robot. In this paper, a small and compact 3D maneuverable robotic fish is developed. Instead of using a compressed air tank, the robot is equipped with an on-board water electrolyzer to generate the gases for depth change. The fabricated robotic fish shows fast diving and rising performance. A servo motor is used to generate asymmetric flapping motion on the caudal fin, which leads to a two-dimensionally (2D) planar motion. A 3D dynamic model is then derived for the fabricated robotic fish. Several open-loop control experiments have been conducted to validate the model as well as the design. It has been demonstrated in the experimental results that the robot is capable of generating 3D motion. The robot can achieve 0.13 m/s forward velocity, 30.6 degree/s turning rate, and it takes about 5.5 s to dive to 0.55 m and 10 s to rise.

Keywords: Marine Robotics
Abstract: A new solution to improving the poor endurance of the existing hybrid aerial underwater vehicle (HAUV) is proposed in this paper. The proposed multimodal hybrid aerial underwater vehicle (MHAUV) merges the design concept of the fixed-wing unmanned aerial vehicle (UAV), the multirotor, and the underwater glider (UG) and has a novel lightweight pneumatic buoyancy adjustment system. MHAUV is well suited for moving in distinct medium and can achieve extended endurance for long distance travel in both air and water. The mathematical model is given based on Newton-Euler formalism. Necessary design principles of the vehicle’s physical parameters are obtained through different gliding equilibrium points. Then, a control scheme composed of two separate proportional-integral-derivative (PID) is employed for the vehicle's motion control in multi-domain simulation. The simulation results are presented to verify the multi-domain mobility and the mode switch ability of the proposed vehicle intuitively. Finally, a prototype, NEZHA, is introduced to be the experimental platform. The success of the flight test, the hovering test, the underwater glide test, and the medium transition test all contribute to prove the feasibility of the proposed concept of the novel MHAUV.

Keywords: Biologically-Inspired Robots, Soft Material Robotics, Marine Robotics
Abstract: Aquatic-aerial multimodal vehicle is a new concept aircraft that can freely shuttle between water and air. Some of the natural organisms provide the inspiration to realize this multi-locomotion. Most of current prototypes use rigid link mechanisms or hinges to morph the structure thus to adapt to the aquatic-aerial environment, which is commonly complicated and bulky. In this paper, we present a novel prototype with pneumatically-driven soft fins and arms that can fold and spread just like the flying squid. The fins and arms can augment the lift force during flying by spreading and reduce drag force during swimming by folding. The performance of the morphable structures was investigated in wind and water tunnel. The results explain the tradeoff strategies of multimodal-locomotion between water and air, and verify the feasibility of the novel aquatic-aerial vehicle with soft morphable structures.

Keywords: Marine Robotics
Abstract: Compliant fish-like robots are being developed as efficient and dependable underwater observation platforms with low impact on the observed environment. Orientation control is an essential building block to achieve autonomy for those vehicles. So far, the major focus has been on rigid tails or on flexible tails with a high degree of actuation. We present a novel control strategy for an underactuated robotic fish with a flexible tail optimized for cruising. The basis for our approach is the generation asymmetric velocity profiles of the robot's tail beats. To achieve such velocity profiles, the usual sinusoidal tail actuation is replaced with skewed triangle waves. We provide a simple formulation for such waves, where their skew is dependent on only one variable which we define as skew factor. Furthermore, a nonlinear control law is derived to achieve the desired turning motions. We implement the controller on a compliant fish-like robot with a simple actuation mechanism. The control scheme is experimentally validated, and its robustness is tested in field trials.

Keywords: Field Robots, Computer Vision for Transportation, Autonomous Vehicle Navigation
Abstract: Project AutoVision aims to develop localization and 3D scene perception capabilities for a self-driving vehicle. Such capabilities will enable autonomous navigation in urban and rural environments, in day and night, and with cameras as the only exteroceptive sensors. The sensor suite employs many cameras for both 360-degree coverage and accurate multi-view stereo; the use of low-cost cameras keeps the cost of this sensor suite to a minimum. In addition, the project seeks to extend the operating envelope to include GNSS-less conditions which are typical for environments with tall buildings, foliage, and tunnels. Emphasis is placed on leveraging multi-view geometry and deep learning to enable the vehicle to localize and perceive in 3D space. This paper presents an overview of the project, and describes the sensor suite and current progress in the areas of calibration, localization, and perception.

Keywords: Visual-Based Navigation, Aerial Systems: Perception and Autonomy, Robust/Adaptive Control of Robotic Systems
Abstract: Visual-inertial navigation methods have been shown to be an effective, low-cost way to operate autonomously without GPS or other global measurements, however most filtering approaches to VI suffer from observability and consistency problems.To increase robustness of the state-of-the-art methods, we propose a three-fold improvement. First, we propose the addition of a linear drag term in the velocity dynamics which improves estimation accuracy. Second, we propose the use of a partial-update formulation which limits the effect of linearization errors in partially-observable states, such as sensor biases. Finally, we propose the use of a keyframe reset step to enforce observability and consistency of the normally unobservable position and heading states.

In this paper, we derive the proposed filter and use a Monte Carlo simulation experiment to analyze the response of visual-inertial Kalman filters with the above described additions. The results of this study show that the combination of all of these features significantly improves estimation accuracy and consistency.

Keywords: Sensor Fusion, Visual-Based Navigation, Localization
Abstract: We propose a fully dense direct filter-based visual-inertial odometry method estimating both pixel depth for all pixels and robot state simultaneously, having all uncertainties in the same state vector. Due to the fully dense method, our approach works even in low-textured areas with very low, smooth gradients (i.e. scenes where feature based or semi-dense approaches fail). Our algorithm performs in real-time on a CPU with a time complexity linearly dependent on the amount of pixels in the provided image. To achieve this, we propose complexity reduction methods for fast matrix inversion, exploiting specific structures of the covariance matrix. We provide both simulated and real-world results in low-textured areas with a smooth gradient.