Keywords: Robotics and Automation in Agriculture and Forestry, Agricultural Automation
Abstract: We introduce FruitNeRF, a unified novel fruit counting framework that leverages state-of-the-art view synthesis methods to count any fruit type directly in 3D. Our framework takes an unordered set of posed images captured by a monocular camera and segments fruit in each image. To make our system independent of the fruit type, we employ a foundation model that generates binary segmentation masks for any fruit. Utilizing both modalities, RGB and semantic, we train a semantic neural radiance field. Through uniform volume sampling of the implicit Fruit Field, we obtain fruit-only point clouds. By applying cascaded clustering on the extracted point cloud, our approach achieves precise fruit count. The use of neural radiance fields provides significant advantages over conventional methods such as object tracking or optical flow, as the counting itself is lifted into 3D. Our method prevents double counting fruit and avoids counting irrelevant fruit. We evaluate our methodology using both real-world and synthetic datasets. The real-world dataset consists of three apple trees with manually counted ground truths, a benchmark apple dataset with one row and ground truth fruit location, while the synthetic dataset comprises various fruit types including apple, plum, lemon, pear, peach, and mangoes. Additionally, we assess the performance of fruit counting using the foundation model compared to a U-Net.

Keywords: Robotics and Automation in Agriculture and Forestry, Localization, Deep Learning Methods
Abstract: 3D LiDAR-based place recognition has been extensively researched in urban environments, yet it remains underexplored in agricultural settings. Unlike urban contexts, horticultural environments, characterized by their permeability to laser beams, result in sparse and overlapping LiDAR scans with suboptimal geometries. This phenomenon leads to intra- and inter-row descriptor ambiguity. In this work, we address this challenge by introducing SPVSoAP3D, a novel modeling approach that combines a voxel-based feature extraction network with an aggregation technique based on a second-order average pooling operator, complemented by a descriptor enhancement stage. Furthermore, we augment the existing HORTO-3DLM dataset by introducing two new sequences derived from horticultural environments. We evaluate the performance of SPVSoAP3D against state-of-the-art (SOTA) models, including OverlapTransformer, PointNetVLAD, and LOGG3D-Net, utilizing a cross-validation protocol on both the newly introduced sequences and the existing HORTO-3DLM dataset. The findings indicate that the average operator is more suitable for horticultural environments compared to the max operator and other first-order pooling techniques. Additionally, the results highlight the improvements brought by the descriptor enhancement stage.

Keywords: Robotics and Automation in Agriculture and Forestry, Field Robots, Localization
Abstract: Accurate long-term place recognition is crucial for autonomous mobile robots operating in unstructured environments. However, in the challenging scene of orchard with high-frequency repetitive features, traditional LiDAR-based localization methods relying on geometric features prove to be inadequate. To address this challenge, we propose TriLoc-NetVLAD, a novel LiDAR-based long-term place recognition approach designed to handle the repetitive and ambiguous features of orchards. This approach initially fuses the point cloud density, height and spatial information to transform unordered 3D point clouds into a channel selection strategy based on descriptor’s sublayer similarity between query and its corresponding positive and negative samples to amplify the differences in environmental features. Finally, a Triplet Network is employed to extract local features, encompassing both high-dimensional and low-dimensional information. These local features are then cascaded through NetVLAD layer to form a global descriptor. Furthermore, we have built a cross-seasonal orchard dataset to evaluate the performance of our place recognition method. The experiment results demonstrate the advantageous localization performances of the proposed localization system over the existing methods.

Keywords: Robotics and Automation in Agriculture and Forestry, Robotics and Automation in Life Sciences, Deep Learning for Visual Perception
Abstract: Robotic branch pruning, a rapidly growing field addressing labor shortages in agriculture, requires detailed perception of branch geometry and topology. However, point clouds obtained in agricultural settings often lack completeness, limiting pruning accuracy. This work addressed point cloud quality via a closed-loop approach, (Real2Sim)^-1. Leveraging a Real-to-Simulation (Real2Sim) data generation pipeline, we generated simulated 3D apple trees based on realistically characterized apple tree information without manual parameterization. These 3D trees were used to train a simulation-based deep model that jointly performs point cloud completion and skeletonization on real-world partial branches, without extra real-world training. The Sim2Real qualitative results showed the model’s remarkable capability for geometry reconstruction and topology prediction. Additionally, we quantitatively evaluated the Sim2Real performance by comparing branch-level trait characterization errors using raw incomplete data and the best complete data. The Mean Absolute Error (MAE) reduced by 75% and 8% for branch diameter and branch angle estimation, respectively, which indicates the effectiveness of the Real2Sim data in a zero-shot generalization setting. The characterization improvements contributed to the precision and efficacy of robotic branch pruning.

Keywords: Robotics and Automation in Agriculture and Forestry, Dual Arm Manipulation, Perception for Grasping and Manipulation
Abstract: Horticultural tasks such as pruning and selective harvesting are labor intensive and horticultural staff are hard to find. Automating these tasks is challenging due to the semi- structured greenhouse workspaces, changing environmental conditions such as lighting, dense plant growth with many occlusions, and the need for gentle manipulation of non-rigid plant organs. In this work, we present the three-armed system HortiBot, with two arms for manipulation and a third arm as an articulated head for active perception using stereo cameras. Its perception system detects not only peppers, but also peduncles and stems in real time, and performs online data association to build a world model of pepper plants. Collision-aware online trajectory generation allows all three arms to safely track their respective targets for observation, grasping, and cutting. We integrated perception and manipulation to perform selective harvesting of peppers and evaluated the system in lab experiments. Using active perception coupled with end-effector force torque sensing for compliant manipulation, HortiBot achieves high success rates in our indoor pepper plant mock-up.

Keywords: Robotics and Automation in Agriculture and Forestry, Mapping, Aerial Systems: Applications
Abstract: For biodiversity and forestry applications, end-users desire maps of forests that are fully detailed--from the forest floor to the canopy. Terrestrial laser scanning and aerial laser scanning are accurate and increasingly mature methods to scan the forest. However, individually they are not able to estimate attributes such as tree height, trunk diameter and canopy density due to the inherent differences in their field-of-view and mapping processes. In this work, we present a pipeline that can automatically generate a single joint terrestrial and aerial forest reconstruction. The novelty of the approach is a marker-free registration pipeline, which estimates a set of relative transformation constraints between the aerial cloud and terrestrial sub-clouds without requiring any co-registration reflective markers to be physically placed in the scene. Our method then uses these constraints in a pose graph formulation, which enables us to finely align the respective clouds while respecting spatial constraints introduced by the terrestrial SLAM scanning process. We demonstrate that our approach can produce a fine-grained and complete reconstruction of large-scale natural environments, enabling multi-platform data capture for forestry applications without requiring external infrastructure.

Keywords: Robotics and Automation in Agriculture and Forestry, Reactive and Sensor-Based Planning
Abstract: In this work, a methodology to find the best view of a grape stem and approach angle in order to crop it is proposed. The control scheme is based only on a classified point cloud obtained by the in-hand camera attached to the robot’s end effector without continuous stem tracking. It is shown that the proposed controller finds and reaches the optimal view point and subsequently the stem fast and efficiently, accelerating the overall harvesting procedure. The proposed control scheme is evaluated through experiments in the lab with a UR5e robot with an in-hand RealSense camera on a mock-up vine.

Keywords: Robotics and Automation in Agriculture and Forestry, Semantic Scene Understanding, Transfer Learning
Abstract: Semantic segmentation plays a pivotal role in many robotic applications requiring high-level scene understanding, such as smart farming, where the precise identification of trees or plants can aid navigation and crop monitoring tasks. While deep-learning-based semantic segmentation approaches have reached outstanding performance in recent years, they demand large amounts of labeled data for training. Inspired by modern Unsupervised Domain Adaptation (UDA) techniques, in this paper, we introduce a two-step training pipeline specifically tailored to challenging natural scenes, where the availability of annotated data is often quite limited. Our strategy involves the initial training of a powerful domain adaptive architecture, followed by a refinement stage, where segmentation masks predicted by the Segment Anything Model (SAM) are used to improve the accuracy of the predictions on the target dataset. These refined predictions serve as pseudo-labels to supervise the training of a final distilled architecture for real-time deployment. Extensive experiments conducted in two real-world scenes demonstrate the effectiveness of the proposed method. Specifically, we show that our pipeline enables the training of a MobileNetV3 that achieves significant mIoU gains of 3.60% and 11.40% on our two datasets compared to the DAFormer while only demanding 1/15 of the latter's inference time. Code and datasets are available at https://github.com/VIS4ROB-lab/nature_uda_rt_segmentation .

Keywords: Robotics and Automation in Agriculture and Forestry, Localization, Deep Learning for Visual Perception
Abstract: Conducting visual assessments under the canopy using mobile robots is an emerging task in smart farming and forestry. However, it is challenging to register images across different data-collection days, especially across seasons, due to the self-occluding geometry and temporal dynamics in forests and orchards. This paper proposes a new approach for registering under-canopy image sequences in general and in these situations. Our methodology leverages standard GPS data and deep-learning-based perspective to bird's-eye view conversion to provide an initial estimation of the positions of the trees in images and their association across datasets. Furthermore, it introduces an innovative strategy for extracting tree trunks and clean ground surfaces from noisy and sparse 3D reconstructions created from the image sequences, utilizing these features to achieve precise alignment. Our robust alignment method effectively mitigates position and scale drift, which may arise from GPS inaccuracies and Sparse Structure from Motion (SfM) limitations. We evaluate our approach on three challenging real-world datasets, demonstrating that our method outperforms ICP-based methods on average by 50%, and surpasses FGR and TEASER++ by over 90% in alignment accuracy. These results highlight our method's cost efficiency and robustness, even in the presence of severe outliers and sparsity. https://github.com/VIS4ROB-lab/bev_undercanopy_registration

Keywords: Robotics and Automation in Agriculture and Forestry, Agricultural Automation, Multi-Robot Systems
Abstract: This work presents the design of Stickbug, a six-armed, multi-agent, precision pollination robot that combines the accuracy of single-agent systems with swarm parallelization in greenhouses. Precision pollination robots have often been proposed to offset the effects of a decreasing population of natural pollinators, but they frequently lack the required parallelization and scalability. Stickbug achieves this by allowing each arm and drive base to act as an individual agent, significantly reducing planning complexity. Stickbug uses a compact holonomic Kiwi drive to navigate narrow greenhouse rows, a tall mast to support multiple manipulators and reach plant heights, a detection model and classifier to identify Bramble flowers, and a felt-tipped end-effector for contact-based pollination. Initial experimental validation demonstrates that Stickbug can attempt over 1.5 pollinations per minute with a 49% success rate. Additionally, a Bramble flower perception dataset was created and is publicly available alongside Stickbug's software and design files.

Keywords: Robotics and Automation in Agriculture and Forestry, Computer Vision for Automation, RGB-D Perception
Abstract: In agricultural robotics, effective observation and localization of fruits present challenges due to occlusions caused by other parts of the tree, such as branches and leaves. These occlusions can result in false fruit localization or impede the robot from picking the fruit. The objective of this work is to push away branches that block the fruit's view to increase their visibility. Our setup consists of an RGB-D camera and a robot arm. First, we detect the occluded fruit in the RGB image and estimate its occluded part via a deep learning generative model in the depth space. The direction to push to clear the occlusions is determined using classic image processing techniques. We then introduce a 3D extension of the 2D Hough transform to detect straight line segments in the point cloud. This extension helps detect tree branches and identify the one mainly responsible for the occlusion. Finally, we clear the occlusion by pushing the branch with the robot arm. Our method uses a combination of deep learning for fruit appearance estimation, classic image processing for push direction determination, and 3D Hough transform for branch detection. We validate our perception methods through real data under different lighting conditions and various types of fruits (i.e. apple, lemon, orange), achieving improved visibility and successful occlusion clearance. We demonstrate the practical application of our approach through a real robot branch pushing demonstration.

Keywords: Mobile Manipulation, Perception for Grasping and Manipulation, Robotics and Automation in Agriculture and Forestry
Abstract: Robotic weed flaming is a new and environmentally friendly approach to weed removal in the agricultural field. Using a mobile manipulator equipped with a flamethrower, we design a new system and algorithm to enable effective weed flaming, which requires robotic manipulation with a soft and deformable end effector, as the thermal coverage of the flame is affected by dynamic or unknown environmental factors such as gravity, wind, atmospheric pressure, fuel tank pressure, and pose of the nozzle. System development includes overall design, hardware integration, and software pipeline. To enable precise weed removal, the greatest challenge is to detect and predict dynamic flame coverage in real time before motion planning, which is quite different from a conventional rigid gripper in grasping or a spray gun in painting. Based on the input from two onboard infrared cameras and the pose information of the mobile manipulator and the nozzle of the flamethrower, we propose a new dynamic flame coverage model. The flame model uses a center-arc curve with a Gaussian cross-section model to describe the flame coverage in real time. The experiments have demonstrated the working system and shown that our model and algorithm can achieve a mean average precision (mAP) of more than 76% in the reprojected images during online prediction.

Keywords: Robotics and Automation in Construction, Reinforcement Learning, Human-Centered Robotics
Abstract: In the dynamic construction industry, traditional robotic integration has primarily focused on automating specific tasks, often overlooking the complexity and variability of human aspects in construction workflows. This paper introduces a human-centered approach with a "work companion rover" designed to assist construction workers within their existing practices, aiming to enhance safety and workflow fluency while respecting construction labor's skilled nature. We conduct an in-depth study on deploying a robotic system in carpentry formwork, showcasing a prototype that emphasizes mobility, safety, and comfortable worker-robot collaboration in dynamic environments through a contextual Reinforcement Learning (RL)-driven modular framework. Our research advances robotic applications in construction, advocating for collaborative models where adaptive robots support rather than replace humans, underscoring the potential for an interactive and collaborative human-robot workforce.

Keywords: Robotics and Automation in Construction, Underactuated Robots, Robotics in Hazardous Fields
Abstract: Automation of hydraulic material handling machinery is currently limited to semi-static pick-and-place cycles. Dynamic throwing motions which utilize the passive joints, can greatly improve time efficiency as well as increase the dumping workspace. In this work, we use RL to design dynamic controllers for material handlers with underactuated arms as commonly used in logistics. The controllers are tested both in simulation and in real-world experiments on a 12-ton test platform. The method is able to exploit the passive joints of the gripper to perform dynamic throwing motions. With the proposed controllers, the machine is able to throw individual objects to targets outside the static reachability zone with good accuracy for its practical applications. The work demonstrates the possibility of using RL to perform highly dynamic tasks with heavy machinery, suggesting a potential for improving the efficiency and precision of autonomous material handling tasks.

Keywords: Robotics and Automation in Construction, Task and Motion Planning, Optimization and Optimal Control
Abstract: We propose a hierarchical task and motion planning (TAMP) for autonomous construction that manipulates deformable objects, such as terrain excavation. The TAMP is required to generate an efficient task plan to meet high-level construction goals at sites with environmental diversity while ensuring motion feasibility. The difficulty, however, is to manipulate deformable objects containing nonlinear dynamics with a target given by a continuous value as the task specification. Optimization-based TAMP with signal temporal logic specifications in robotics is promising because of its continuous task specification and formulation as a nonlinear programming problem. The key to its application to extensive, long-term planning at real construction sites is a computationally efficient and stable formulation. We introduce a new expression for deformable objects with a simple differentiable function and a system model that can represent mode transitions based on machine action on the objects. This allows TAMP to be formulated as simultaneously selecting an action for objects and planning the motion to execute it. Furthermore, a hierarchical method that gives appropriate initial values is combined to improve optimality for large-scale nonlinear problems. From the verification by numerical experiments, the proposed method can generate a plan that minimizes the time to meet the task goal, even when the area is expanded.

Keywords: Robotics and Automation in Construction, Software-Hardware Integration for Robot Systems, Industrial Robots
Abstract: In this paper, we present a novel study on a BEV image-based lane tracking control system for an autonomous lane repainting robot. Our research introduces a cutting-edge lane detection method based on BEV images, leveraging row-anchor techniques to enhance precision and provide detailed error information for lane tracking algorithms. By utilizing real-time sensor data and advanced deep learning processes, we have successfully implemented a high-performance lane repainting system that minimizes errors and ensures accuracy. Our proposed position-based visual pure pursuit algorithm (PV-PP) plays a crucial role in guiding the lane repainting process with precision and efficiency, ultimately improving the functionality and feasibility of the linear actuator responsible for paint spraying in the real indusrial fields. Through our contributions, including innovative lane detection methods, real-time sensor utilization, and robot control algorithm design, we aim to advance the field of autonomous lane repainting robots and enhance the safety and effectiveness of road maintenance operations.

Keywords: Physically Assistive Devices, Soft Robot Applications, Grippers and Other End-Effectors
Abstract: Assistive robot arms have the potential to help disabled or elderly adults eat everyday meals without relying on a caregiver. To provide meaningful assistance, these robots must reach for food items, pick them up, and then carry them to the human's mouth. Current work equips robot arms with standard utensils (e.g., forks and spoons). But --- although these utensils are intuitive for humans --- they are not easy for robots to control. If the robot arm does not carefully and precisely orchestrate its motion, food items may fall out of a spoon or slide off of the fork. Accordingly, in this paper we design, model, and test Kiri-Spoon, a novel utensil specifically intended for robot-assisted feeding. Kiri-Spoon combines the familiar shape of traditional utensils with the capabilities of soft grippers. By actuating a kirigami structure the robot can rapidly adjust the curvature of Kiri-Spoon: at one extreme the utensil wraps around food items to make them easier for the robot to pick up and carry, and at the other extreme the utensil returns to a typical spoon shape so that human users can easily take a bite of food. Our studies with able-bodied human operators suggest that robot arms equipped with Kiri-Spoon carry foods more robustly than when leveraging traditional utensils.

Keywords: Physically Assistive Devices, Wearable Robotics
Abstract: Spatial cognition refers to the ability to understand and navigate the spatial relationship between objects in the environment. People with blindness and low vision (pBLV) face significant challenges with spatial cognition due to the reliance on visual input. Without the full range of visual cues, pBLV individuals often find it difficult to grasp a comprehensive understanding of their environment, leading to obstacles in scene recognition and precise object localization, especially in unfamiliar environments. This limitation extends to their ability to independently detect and avoid potential tripping hazards, making navigation and interaction with their environment more challenging. In this paper, we presents a pioneering wearable platform tailored to enhance the spatial cognition of pBLV through the integration of multi-modal foundation model. The proposed platform integrates a wearable camera with audio module and leverages the advanced capabilities of vision language foundation model (i.e., GPT-4 and GPT-4V), for the nuanced processing of visual and textual data. Then we employ vision language models to bridge the gap between visual information and the proprioception of visually impaired users, offering more intelligible guidance by aligning visual data with the natural perception of space and movement. We also apply prompt engineering to guide the large language model to act as an assistant tailored specifically for pBLV users to produce accurate answers. Another notable innovation in our model is the incorporation of a chain of thought reasoning process, which enhances the accuracy and interpretability of the model, facilitating the generation of more precise responses to complex user inquiries across diverse environmental contexts. To assess the practical impact of our proposed wearable platform, we carried out a series of real-world experiments across three tasks that are commonly challenging for people with blindness and low vision: risk assessment, object localization, and scene recognition. Additionally, through an ablation study conducted on the VizWiz dataset, we rigorously assess the contribution of each individual module, substantiating the integral role in the model's overall performance.

Keywords: Physically Assistive Devices, Prosthetics and Exoskeletons
Abstract: Assistive devices are to be designed with the objective of use in daily-life as well as broad adoption by end users. In this context, it is necessary to tackle usability challenges by properly detecting and acting in accordance to user intents while minimizing the device installation complexity as well. In the case of physical assistive devices, using force/torque sensors is advantageous to detect user intent compared to EMG interfaces, but it remains difficult to correctly translate the detected intent into actuator motions. Focusing on upper-limb assistive robots, the user voluntary force is commonly used with a controller based on an admittance approach which leads to relatively poor reactivity and requires the user to develop force throughout the movement which can lead to fatigue, particularly for people with upper-limb impairments. This work proposes a Force-Triggered (FT) controller which can initiate and maintain movement only from short force impulses. The user voluntary forces are retrieved from total interaction forces by subtracting the passive component measured beforehand during a calibration phase. This paper presents the design of the proposed FT controller and its preliminary testing on pick-and-place tasks compared to an admittance strategy. This experiment was performed with one participant without impairment, equipped with an upper-limb exoskeleton prototype designed from recommendations of physical medicine therapists. This preliminary work highlights the potential of the proposed FT controller. Also, it provides directions for future work and clinical trials with end-users to assess the proposed FT approach usability while used alone or in the form of an hybrid controller between FT and admittance strategies.

Keywords: Physically Assistive Devices, Human-Centered Robotics, Haptics and Haptic Interfaces
Abstract: This study investigates the efficacy of a haptic interface, aiming to offer the walking frame users accurate, intuitive, and easily understandable directional cues. The research introduces a novel haptic feedback interface incorporated into a walking frame to enhance navigation assistance. The haptic handle encompasses three distinct haptic feedback modalities: vibration, skin stretch, and combined feedback. Ten participants, all in good health, engaged with the haptic handle for navigation. Across all three haptic feedback methods, 60% of participants found the combined feedback to be the most effective, while 40% favoured the vibration feedback; none selected the skin-stretch feedback. Comparative analysis revealed significant disparities between vibration and combined input regarding velocity (p-value: 0.04). These findings emphasize the haptic handle's capacity to give users an instinctive perception of directional cues, thus offering a promising avenue for assistive navigation.

Keywords: Prosthetics and Exoskeletons, Soft Robot Applications, Physically Assistive Devices
Abstract: The affordability and functionality of hand prosthetics in developing countries are still very limited. This work aims to present and evaluate the new version of the PrHand affordable robotic prosthesis (PrHand V3), built with soft robotics and compliant mechanisms. PrHand V3 implements a new frictionless tendon unification system, the degree of freedom of thumb opposition was removed, and the finger flexion was improved to the previous version, PrHand V2. The study contributes by evaluating these mechanical changes and conducting the first functional assessment of PrHand V3 with an amputee user. The Anthropomorphic Hand Assessment Protocol (AHAP) dexterity test was the first evaluation in this work; it evaluated how the prosthesis performs eight different grips. PrHand V3 was compared with a PrHand V2 and a commercial robotic prosthesis A3D from Pr´otesis Avanzadas SAS. PrHand V3’s score on the AHAP test was 80%. This result is higher than the 69% obtained by the PrHand V2 and the 79% obtained by A3D. The Activities Measure for Upper Limb Amputees (AM-ULA) test was the second evaluation in this work; An A3D amputee user performed 23 Activities of Daily Living with PrHand V3 and an A3D. PrHand V3 obtained an average of 2.86/4, and A3D obtained an average of 2.96/4 without significant differences between the two tests. The soft actuation of PrHand V3 as an affordable prosthesis performs similarly to a commercial robotic prosthesis with the advantage of being more flexible to assist a trans-radial hand amputee.

Keywords: Prosthetics and Exoskeletons, Human-Robot Collaboration, Virtual Reality and Interfaces
Abstract: Powered upper limb prostheses offer a particularly interesting case of human-machine interaction, where the user and the robot are physically coupled in an open chain manipulator. The biological and mechanical degrees of freedom (DOF) must collaborate for the user to manipulate objects in the environment. Current state-of-the-art systems use machine learning models to classify electromyogram (EMG) signals into motion intent primitives, allowing users to move the prosthetic joints sequentially at a fixed velocity. Current commercially available systems are limited to 1 or 2 powered DOF. This interface is intended to work for simple systems but does not extend well into higher DOF. In this paper, we present a semi-autonomous (SA) hybrid gaze-EMG interface that allows users to command the device in task-space instead of joint-space. Target end-effector poses are selected by tracking the user’s gaze vector, and then EMG signals guide the prosthetic along a calculated trajectory towards that pose. To examine how prosthesis interface performance scales with available mechanical DOF, we had 4 subjects complete virtual pick and place tasks with SA and traditional controller interfaces, varying the available DOF in the prosthetic wrist. Our results show that with the SA interface, increased DOF leads to a significant (p ≤ 0.05) reduction in compensatory motion of the upper arm, more effective (p ≤ 0.01) utilization of the increased configuration space, and overall more efficient motion (p ≤ 0.01) than traditional classification based interfaces. These findings indicate that when given SA interfaces, subjects can benefit from fully articulated prosthetic devices, which motivates more clinical research into SA systems and commercial development of higher-DOF devices.

Keywords: Prosthetics and Exoskeletons, Humanoid and Bipedal Locomotion, Model Learning for Control
Abstract: Exoskeleton locomotion must be robust while being adaptive to different users with and without payloads. To address these challenges, this work introduces a data-driven predictive control (DDPC) framework to synthesize walking gaits for lower-body exoskeletons, employing Hankel matrices and a state transition matrix for its data-driven model. The proposed approach leverages DDPC through a multi-layer architecture. At the top layer, DDPC serves as a planner employing Hankel matrices and a state transition matrix to generate a data-driven model that can learn and adapt to varying users and payloads. At the lower layer, our method incorporates inverse kinematics and passivity-based control to map the planned trajectory from DDPC into the full-order states of the lower-body exoskeleton. We validate the effectiveness of this approach through numerical simulations and hardware experiments conducted on the Atalante lower-body exoskeleton with different payloads. Moreover, we conducted a comparative analysis against the model predictive control (MPC) framework based on the reduced-order linear inverted pendulum (LIP) model. Through this comparison, the paper demonstrates that DDPC enables robust bipedal walking at various velocities while accounting for model uncertainties and unknown perturbations.

Keywords: Prosthetics and Exoskeletons, Rehabilitation Robotics, Wearable Robotics
Abstract: This study presents the development and validation of an innovative hand exoskeleton designed for the rehabilitation of patients with Complex Regional Pain Syndrome (CRPS), a condition frequently arising post-injury or surgeries. The prototype is tailored for the hand, a region commonly affected by CRPS, and is notable for its adaptability and a comprehensive sensor system for monitoring individual joint movements. These features allow for personalized rehabilitation and objective progress tracking, addressing limitations in traditional physiotherapy such as availability, cost, and time constraints. Reliable sensor performance was defined through precise force measurements and stability over time, showing minimal drift. These features enable personalized rehabilitation and objective progress tracking, addressing limitations in traditional physiotherapy such as availability, cost, and time constraints.The contributions of this work lie in its innovative design and the potential for robotic systems to improve therapeutic outcomes in CRPS rehabilitation.

Keywords: Prosthetics and Exoskeletons, Neural and Fuzzy Control, Manipulation Planning
Abstract: In this work, an upgraded exoskeleton design is presented with enhanced trajectory tracking and mechanical transparency. Compared to the first version, the design features a 3-DoF actuated shoulder joint and a mechanism to regulate the pretension of Bowden cables. Force/torque sensors are installed to directly measure the interaction forces between the human arm and the exoskeleton at the connecting points. Three control strategies were evaluated to follow a desired trajectory; A PD controller, a PD controller with friction observer, and an adaptive controller based on Radial Basis Function (RBF). These strategies also form the basis for an admittance control, aimed at improving the exoskeleton's mechanical transparency during interaction with the human arm. Simulations and experimental results demonstrate that the PD control, supported by friction estimation via a momentum observer, achieves superior tracking performance. Moreover, the system's mechanical transparency is enhanced using the admittance RBF-based controller, showing marginally superior results.

Keywords: Prosthetics and Exoskeletons, Mechanism Design, Humanoid Robot Systems
Abstract: Exoskeletons for robot operation necessitate shoulders with high range of motions and high degrees of freedom to fit the operator's shoulder girdle.These shoulder joints need high torque for force feedback on the operator. Existing exoskeletons struggle to simultaneously meet these requirements of high DOFs, wide ROM, and high torque due to spatial constraints. This study introduces an exoskeleton with a distal branching link mechanism that addresses this issue by concentrating on each link's absolute and relative degrees of freedom. In the proposed exoskeleton, the end-effector's absolute DOF, the forearm's absolute DOF, and the end-effector and forearm's relative DOF are matched between the operator and the exoskeleton. This is achieved while reducing the overall DOF by sharing the root link system's DOF. Furthermore, by avoiding direct attachment of the operator to the exoskeleton's shoulder, the design can accommodate the human shoulder's high torque and high ROM. The study demonstrates that the branching exoskeleton outperforms existing fixed-linkage exoskeletons in terms of tracking the operator's arms and the torque required by the exoskeleton's joints. Utilizing this exoskeleton, we successfully maneuvered an actual humanoid robot to perform daily activities where the forearm posture is crucial.

Keywords: Prosthetics and Exoskeletons, Datasets for Human Motion, Modeling and Simulating Humans
Abstract: Prosthetic limb abandonment remains an unsolved challenge as amputees consistently reject their devices. Current prosthetic designs often fail to balance human-like perfor- mance with acceptable device weight, highlighting the need for optimised designs tailored to modern tasks. This study aims to provide a comprehensive dataset of joint kinematics and kinetics essential for performing activities of daily living (ADL), thereby informing the design of more functional and user-friendly prosthetic devices. Functionally required Ranges of Motion (ROM), velocities, and torques for the Glenohumeral (rotation), elbow, Radioulnar, and wrist joints were computed using motion capture data from 12 subjects performing 24 ADLs. Our approach included the computation of joint torques for varying mass and inertia properties of the upper limb, while torques induced by the manipulation of experimental objects were considered by their interaction wrench with the subject’s hand. Joint torques pertaining to individual ADL scaled linearly with limb and object mass and mass distribution, permitting their generalisation to not explicitly simulated limb and object dynamics with linear regressors (LRM), exhibiting coefficients of determination R = 0.99 ± 0.01. Exemplifying an application of data-driven prosthesis design, we optimise wrist axes orientations for two serial and two differential joint con- figurations. Optimised axes reduced peak power requirements, between 22% to 38% compared to anatomical configurations, by exploiting high torque correlations (r = −0.84, p < 0.05) between Ulnar deviation and wrist flexion/extension joints. This study offers critical insights into the functional requirements of upper limb prostheses, providing a valuable foundation for data-driven prosthetic design that addresses key user concerns and enhances device adoption.

Keywords: Prosthetics and Exoskeletons, Rehabilitation Robotics, Optimization and Optimal Control
Abstract: This paper introduces a novel hybrid torque allocation method for improving wearability and mobility in integrated functional electrical stimulation (FES) of the quadriceps muscles and powered exoskeleton systems. Our proposed approach leverages a hierarchical closed-loop controller for knee joint position tracking while addressing limitations of powered exoskeletons and FES systems by reducing power consumption and battery size and by mitigating FES-induced muscle fatigue, respectively. The core component is a model-free optimization algorithm that dynamically distributes torque between FES and the exoskeleton by considering tracking error, effort, and the prediction of muscle fatigue in the cost function, computing allocation gain in an online manner. The online optimization approach interactively changes the optimal allocation gain by taking into account the instantaneous value of error and effort and also penalizing FES-induced fatigue, a common challenge in long-duration experiments. The results demonstrate that this dynamic allocation significantly improves system wearability by reducing power consumption without increasing muscle fatigue during the extension phase of walking. This hybrid control approach contributes to improving exoskeleton wearability and rehabilitation outcomes for individuals with SCI and mobility impairments, enhancing assistive technology and quality of life.

Keywords: Prosthetics and Exoskeletons, Visual Tracking, Sensor Fusion
Abstract: Environment awareness is crucial for enhancing walking safety and stability of amputee wearing powered prosthesis when crossing uneven terrains such as stairs and obstacles. However, existing environmental perception systems for prosthesis only provide terrain types and corresponding parameters, which fail to prevent potential collisions when crossing uneven terrains and may lead to falls and other severe consequences. In this paper, a visual-inertial motion estimation approach is proposed for prosthesis to perceive its movement and the changes of spatial relationship between the prosthesis and uneven terrain when traversing them. To achieve this, we estimate the knee motion by utilizing a depth camera to perceive the environment and align feature points extracted from uneven terrains. Subsequently, an error-state Kalman filter is incorporated to fuse the inertial data into visual estimations to obtain a more robust and accurate estimation, which is then utilized to derive the motion of the whole prosthesis for our prosthetic control scheme. Experiments conducted on our collected dataset and stair walking trials with powered prosthesis show that the proposed method can accurately track the motion of human leg and the prosthesis with the average root-mean-square error of toe trajectory less than 5 cm. The proposed method is expected to enable the environmental adaptive control for prosthesis, thereby enhancing amputee’s safety and mobility in uneven terrains.

Keywords: Prosthetics and Exoskeletons, Wearable Robotics, Humanoid Robot Systems
Abstract: Research has indicated that exoskeletons can assist human movement, but due to the influence of additional weight and challenges in control strategy design, only a few exoskeletons effectively reduce the wearers' metabolic costs during running. This paper proposes an innovative and efficient hip-assisted running exoskeleton (HARE) designed to facilitate the flexion and extension movements of the joint along the sagittal plane. In the field of structural engineering, we propose implementing an active-passive combination constant force suspension system, hereinafter referred to as CFS, to effectively mitigate the impact of inertial forces during running. The decoupled transmission mechanism allows the CFS and assist mechanisms to operate independently, ensuring the tension of the cables. The flexible structural design can reduce the locomotion limitation on human bodies and reduce the additional energy burden on the body. In control strategy designing, the joint torque-generating strategy provides personalized assistance strategies for wearers to actively optimize the control parameters. Meanwhile, the safety control strategy based on abnormal gait recognition can ensure human safety. Experiments have shown that compared to not wearing exoskeletons, this device can reduce the energy consumption of the human body by 5.33% at a speed of 9 km/h. This demonstrates its potential in human motion assistance processes.

Keywords: Wearable Robotics, Physically Assistive Devices
Abstract: It is a challenging task for visually impaired people to perceive their surrounding environment due to the complexity of the natural scenes. Their personal and social activities are thus highly limited. This paper introduces a Large Vision-Language Model(LVLM) based environment perception system which helps them to better understand the surrounding environment, by capturing the current scene they face with a wearable device, and then letting them retrieve the analysis results through the device. The visually impaired people could acquire a global description of the scene by long pressing the screen to activate the LVLM output, retrieve the categories of the objects in the scene resulting from a segmentation model by tapping or swiping the screen, and get a detailed description of the objects they are interested in by double-tapping the screen. To help visually impaired people more accurately perceive the world, this paper proposes incorporating the segmentation result of the RGB image as external knowledge into the input of LVLM to reduce the LVLM’s hallucination. Technical experiments on POPE, MME and LLaVA-QA90 show that the system could provide a more accurate description of the scene compared to Qwen-VL-Chat, exploratory experiments show that the system helps visually impaired people to perceive the surrounding environment effectively.

Keywords: Modeling, Control, and Learning for Soft Robots, Hydraulic/Pneumatic Actuators, Grippers and Other End-Effectors
Abstract: In situations where robots work alongside humans, they must be capable of responding flexibly to unexpected external forces. To address this challenge, researchers have conducted numerous studies on soft robotics. However, most of the soft hands studied so far are powered by pneumatic pressure and can only exert pressure up to a several hundred kPa, resulting in low output. To solve this problem, we have developed a hydraulic soft hand in our previous research. In this paper, we derive the relationship between driving pressure, bending angle, and grasping force of a soft hand with a reservoir to evaluate the effect of a rubber sheet reservoir. Additionally, we experimentally show that the soft hand provides grasping flexibility when angle control is applied using the model proposed in this paper.

Keywords: Soft Robot Applications, Soft Robot Materials and Design, Biologically-Inspired Robots
Abstract: Manta rays exhibit complex motion behavior through their flexible fins. This study proposes a novel manta ray-inspired soft robotic swimmer with bistable flapping wings for high-speed and multi-modal swimming. The wings are created with prestressed bistable composite and actuated by small McKibben artificial muscles. Pressurizing and depressurizing the McKibben actuators integrated into the flapping wings generates alternating snap-throughs between two stable states, yielding swimming. Experiments are set up and conducted to analyze how the robot responses vary as a function of input pressures and actuation frequencies for both bistable and monostable modes. Experimental results show that the highest swimming velocity is 0.58 body lengths (BL) per second (equivalent to 12.23 cm/s), and the maximum turning angle speed is 22.5°per second with a smaller turning radius by holding the fins in asymmetric positions for bistable modes. Multimodal swimming motions are achieved including forward and backward translating, turning, and flip-turning.

Keywords: Soft Robot Applications, Actuation and Joint Mechanisms, Soft Robot Materials and Design
Abstract: Soft robotics, an emerging domain in modern robotics, introduces innovative possibilities alongside challenges in controllability, particularly with multi-degree inflatable actuators. We present a novel manipulation method using underactuated soft fingers that addresses these challenges by harnessing symmetry breaking. Central to our approach is the mechanism of self-organization within a ring actuator equipped with five fingers. Typically considered a drawback, we exploit the actuator's buckling behavior to facilitate in-hand manipulation. This strategic utilization enables object motion in both clockwise and counterclockwise directions via system perturbations and adjustments in frequency and duty cycle parameters. Employing the self-organizing properties of our actuator, our method is empirically validated through simulations and real actuator experiments, demonstrating the system's ability in manipulating objects by leveraging the inherent flexibility and morphological advantages. The design enables two degrees of freedom with minimal input, allowing objects to rotate due to the actuator's self-organizing actions. This simplification of control mechanisms is essential for soft robotics manipulation. Our findings indicate that control systems in soft robotics can be significantly simplified, harnessing the adaptable behavior inherent in its morphology.

Keywords: Soft Robot Materials and Design, Modeling, Control, and Learning for Soft Robots, Soft Robot Applications
Abstract: Modelling complex deformation for soft robotics provides a guideline to understand their behaviour, leading to safe interaction with the environment. However, building a surrogate model with high accuracy and fast inference speed can be challenging for soft robotics due to the nonlinearity from complex geometry, large deformation, material nonlinearity etc. The reality gap from surrogate models also prevents their further deployment in the soft robotics domain. In this study, we proposed a Physics Informed Neural Networks (PINNs) named PINN-Ray to model complex deformation for a fin-ray soft robotic gripper, which embeds the minimum potential energy principle from elastic mechanics and additional high-fidelity experimental data into the loss function of neural network for training. This method is significant in terms of its generalisation to complex geometry and robust to data scarcity as compared to other data-driven neural networks. Furthermore, it has been extensively evaluated to model the deformation of the fin-ray finger under external actuation. PINN-Ray demonstrates improved accuracy as compared with Finite element modelling (FEM) after applying the data assimilation scheme to treat the sim-to-real gap. Additionally, we introduced our automated framework to design, fabricate soft robotic fingers, and characterise their deformation by visual tracking, which provides a guideline for the fast prototype of soft robotics.

Keywords: Soft Robot Materials and Design, Mechanism Design, Biologically-Inspired Robots
Abstract: Soft robots - due to their intrinsic flexibility of the body - can adaptively navigate unstructured environments. One of the most popular locomotion gaits that has been implemented in soft robots is undulation. The undulation motion in soft robots resembles the locomotion gait of stringy creatures such as snakes, eels, and C.~Elegans. Typically, the implementation of undulation locomotion on a soft robot requires many actuators to control each segment of the stringy body. The added weight of multiple actuators limits the navigating performance of soft-bodied robots. In this paper, we propose a simple tendon-driven flexible beam with only one actuator (a DC motor) that can generate a mechanical traveling wave along the beam to support the undulation locomotion of soft robots. The beam will be precompressed along its vertical axis to form an S-shape, thus pretensioning the tendons. The motor will wind and unwind the tendons to deform the flexible beam and generate traveling waves along the body of the robot. We experiment with different pre-tension to characterize the relationship between tendon pre-tension forces and the DC-motor winding/unwinding. Our proposal enables a simple implementation of undulation motion to support the locomotion of soft-bodied robots.

Keywords: Soft Sensors and Actuators
Abstract: As automation technologies advance, the need for compact and multi-modal sensors in robotic applications is growing. To address this demand, we introduce CompdVision, a novel sensor that employs a compound-eye imaging system to combine near-field 3D visual and tactile sensing within a compact form factor. CompdVision utilizes two types of vision units to address diverse sensing needs, eliminating the need for complex modality conversion. Stereo units with far-focus lenses can see through the transparent elastomer for depth estimation beyond the contact surface. Simultaneously, tactile units with near-focus lenses track the movement of markers embedded in the elastomer to obtain contact deformation. Experimental results validate the sensor's superior performance in 3D visual and tactile sensing, proving its capability for reliable external object depth estimation and precise measurement of tangential and normal contact forces. The dual modalities and compact design make the sensor a versatile tool for robotic manipulation.

Keywords: Soft Sensors and Actuators, Biologically-Inspired Robots
Abstract: In the last decades, soft robotics has been growing rapidly as an emerging research topic, bringing new paradigms for robotic manipulation, locomotion, and human‒machine interactions. Pneumatic artificial muscle is a powerful, lightweight, rapid response with great design flexibility, making it promising for developing biological muscle-like robotic systems. The PPAM is made of a silicone tube body with double helix coil fiber reinforcement. The double helix coil fiber restricts the radial expansion of the cylinder tube to achieve extension in actuation, and monitors the muscle length change in real time by measuring its inductance. A finite element model was built to simulate the actuation characteristics of the PPAM. A theoretical formula was derived to analyze the inductive length sensing response of the double-helix coil on the PPAM. It is verified that the PPAM can sense its length change regardless of whether it is caused by active driving or external manipulation. Rigorous testing reveals that PPAM has an ultrahigh length sensing resolution of 5.9 µm in relaxed state, with a short response time of 50 ms. The self-length sensing of PPAM is hysteresis free, and highly repeatable, showing no degradation in 1000 operation cycles. In summary, the PPAM shows promising features for developing the next-generation perceptive and responsive soft robots, intelligent hybrid robots, or safer biomedical instruments.

Keywords: Biologically-Inspired Robots, Redundant Robots, Climbing Robots
Abstract: This paper introduces climbing gait for the snake robot by adapting to a flexible net. A net deforms in various ways due to external forces. Therefore, we tilt snake robot's body to adapt to the deformation of the net. In addition, we can adjust the position of the head of a snake robot passing through the mesh. A snake robot can move not only vertically but also horizontally. We demonstrated the validity of the proposed method through experiments in vertical movement and movement in diagonal direction.

Keywords: Biologically-Inspired Robots, Biomimetics, Soft Robot Applications
Abstract: Net-winged midge larvae (genus Liponeura) can achieve robust attachment and crawl on the slippery surface in the fast stream with their powerful abdominal suckers. The rigid spine-like structures distributed in the sucker cavity called microtrichia, have been proven to be crucial in the adhesion process. In this work, we carry out a design of the biomimetic sucker with the spine-like structures and then implement various tests using biomimetic suckers to verify the adhesion capacity enhancement brought by spine-like structures. Finally, we assemble the suckers in a quadruped crawling robot capable of locomotion in both aerial and aquatic environments with a speed of 56.4 mm/s (0.225 BL/s) and can crawl upstream against a turbulent flow with a speed of 38.2 mm/s (0.152 BL/s). This study will inspire biomimetic design in future robotics, and pave the way for future robots to realize long-term observation and monitoring in complex environments.

Keywords: Biologically-Inspired Robots, Legged Robots, Modeling, Control, and Learning for Soft Robots
Abstract: Musculoskeletal quadruped robots driven by pneumatic artificial muscles (PAMs) have great softness. Due to the softness, the proprioceptive information of PAMs (e.g. tension) reflects the environmental information. However, how to utilize this information for stable quadrupedal gait has been rarely explored. In this work, we utilized PAM tension for stable locomotion over uneven terrain. We newly developed a durable tension sensor and proposed tension feedback control for quadruped locomotion over uneven terrain. Our proposed controller stabilizes the trunk posture by modulating the phase of the leg. To verify the effectiveness of the proposed controller, we implement it in a simple quadrupedal model and a musculoskeletal quadruped robot driven by PAMs. The model and robot ran over uneven terrain with and without tension feedback. Through experiments, with tension feedback, the trunk posture oscillated more stably than that without the feedback. Furthermore, the running velocity with tension feedback was higher and had a smaller variance than that without the feedback in the robot experiment. This successful result will lead to more robust musculoskeletal quadruped robots that can be employed in the real-world environment.

Keywords: Biologically-Inspired Robots, Biomimetics, Legged Robots
Abstract: The torso of quadruped mammals serves as a robust foundation for their bodies, enabling a diverse array of agile and intricate movements. However, current quadruped robots cannot match the extensive range of motion exhibited by biological torsos while also serving a load-bearing role. Therefore, this paper presents an active bionic torso that emulates the quadrupedal animal's spinal column and associated muscular structure. This innovative torso can mimic animal torsos movements, including flexing, extending, lateral bending, and axial rotation. A thorough analysis of the torso's kinetic, workspace, and structural dynamics has been conducted. This proposed torso boasts a considerable load-bearing capacity and can support a load that exceeds its weight tenfold. The passive spring incorporated into the bionic torso emulates the intervertebral discs' shock-absorbing and load-bearing functions. Additionally, this paper documents the development of a quadrupedal robot fitted with the proposed bionic torso, demonstrating the torso's mobility in a real-world application.

Keywords: Biologically-Inspired Robots, Marine Robotics, Mechanism Design
Abstract: This study introduces an agile robotic penguin featuring a pair of 2-degrees of freedom (DoF) wing mechanisms. Each wing can independently control flapping and feathering motions with two in-house submersible geared servomotors through a differential gear mechanism. Notably, our mechanism allows unrestricted feathering beyond 360°. Since feathering directly changes the wing’s angle of attack (AoA), the hydrodynamic forces can be significantly adjusted to achieve agile maneuvers. This paper demonstrates various maneuvers, including rapid acceleration, hard braking, rolling, pitching, and yawing, achieved solely by changing the feathering motion of both wings. Our robotic penguin reached a maximum forward speed of 1.8 m/s, comparable to the foraging speed of real penguins. The average roll, pitch, and yaw rates were 363°/s, 75°/s, and 92°/s, respectively. This robot serves as a model for the biological study of maneuverability in real penguins and the engineering exploration of bioinspired agile underwater robots.

Keywords: Biologically-Inspired Robots, Optimization and Optimal Control, Motion and Path Planning
Abstract: Object manipulation has been extensively studied in the context of fixed base and mobile manipulators. However, the overactuated locomotion modality employed by snake robots allows for a unique blend of object manipulation through locomotion, referred to as loco-manipulation. The following work presents an optimization approach to solving the loco-manipulation problem based on non-impulsive implicit contact path planning for our snake robot COBRA. We present the mathematical framework and show high-fidelity simulation results and experiments to demonstrate the effectiveness of our approach.

Keywords: Biologically-Inspired Robots, Field Robots, Mechanism Design
Abstract: Autonomous takeoff of flapping-wing robots (FWRs) is crucial for accelerating response speed and reducing costs in executing tasks. Jumping-aided takeoff is an effective method adopted by birds. However, the limited power density of motors poses challenges in achieving this type of takeoff for FWRs. In this study, we introduce a ground-based FWR ejecting system that utilizes a symmetric slider-crank mechanism (S-SCM) to store energy in spring. This stored energy is then converted into the takeoff speed of the FWR. The dynamic model of each working stage is established, and the design parameters are optimized according to simulation results. The prototype of the FWR ejecting system is fabricated for experimental validations. The results indicate that the system can provide a takeoff speed of 4 m/s for the 270 g FWR. Notably, the system is deployable on rough terrains and only adds a 3.2 g payload to the robot. Our work advances the autonomous takeoff of FWRs, promoting the application of such robots.

Keywords: Biologically-Inspired Robots, Visual-Inertial SLAM, Localization
Abstract: This paper presents a robust visual SLAM system designed for small-scale quadruped robots (ViQu-SLAM) for accurate localization, especially to mitigate the issue of erroneous data association caused by moving objects in dynamic environments. The proposed approach leverages a self-adaptive framework that integrates semantic segmentation with alterations in the spatial location of categorized map points. Besides, combination of leg odometry derived from forward kinematics with IMU provides scale information for positional transformations between keyframes, thus optimizing the overall localization accuracy of quadruped robots. At last, we performed evaluation across various stages and the results demonstrate competitive performance, with 53.16% reduction in average absolute trajectory error compared to that of ORB-SLAM3 in dynamic benchmark datasets. As a result, ViQu-SLAM, including visual and IMU-fused leg odometry, exhibits promising results on a small quadruped robot, reducing positioning errors in dynamic scenes by an average of 29.36% compared to existing state-of-the-art methods.

Keywords: Biomimetics, Modeling and Simulating Humans, Human and Humanoid Motion Analysis and Synthesis
Abstract: The complex ways in which humans utilize their bodies in sports and martial arts are remarkable, and human motion analysis is one of the most effective tools for robot body design and control. On the other hand, motion analysis is not easy, and it is difficult to measure complex body motions in detail due to the influence of numerous muscles and soft tissues, mainly ligaments. In response, various musculoskeletal simulators have been developed and applied to motion analysis and robotics. However, none of them reproduce the ligaments but only the muscles, nor do they focus on the shoulder complex, including the clavicle and scapula, which is one of the most complex parts of the body. Therefore, in this study, a detailed simulation model of the shoulder complex including ligaments is constructed. The model will mimic not only the skeletal structure and muscle arrangement but also the ligament arrangement and maximum muscle strength. Through model predictive control based on the constructed simulation, we confirmed that the ligaments contribute to joint stabilization in the first movement and that the proper distribution of maximum muscle force contributes to the equalization of the load on each muscle, demonstrating the effectiveness of this simulation.

Keywords: Computer Vision for Transportation, Deep Learning for Visual Perception
Abstract: The proliferation of smartphones and other mobile devices provides a unique opportunity to make Advanced Driver Assistance Systems (ADAS) accessible to everyone in the form of an application empowered by low-cost Machine/Deep Learning (ML/DL) models to enhance road safety. For the critical feature of Collision Avoidance in Mobile ADAS, lightweight Deep Neural Networks (DNN) for object detection exist, but conventional pixel-wise depth/distance estimation DNNs are vastly more computationally expensive making them unsuitable for a real-time application on resource-constrained devices. In this paper, we present a distance estimation model, DECADE, that processes each detector output instead of constructing pixel-wise depth/disparity maps. In it, we propose a pose estimation DNN to estimate allocentric orientation of detections to supplement the distance estimation DNN in its prediction of distance using bounding box features. We demonstrate that these modules can be attached to any detector to extend object detection with fast distance estimation. Evaluation of the proposed modules with attachment to and fine-tuning on the outputs of the YOLO object detector on the KITTI 3D Object Detection dataset achieves state-of-the-art performance with 1.38 meters in Mean Absolute Error and 7.3% in Mean Relative Error in the distance range of 0-150 meters. Our extensive evaluation scheme not only evaluates class-wise performance, but also evaluates range-wise accuracy especially in the critical range of 0-70m.

Keywords: Visual Tracking, Deep Learning for Visual Perception, Computer Vision for Transportation
Abstract: Visual mask learning has received increasing attention in the field of visual object tracking. However, most existing studies merely utilize visual mask learning works as pre-training models without fully exploiting their potential for visual representation. In this paper, we present a novel approach for learning tracking target features, leveraging an encoderdecoder architecture with a masked mutual guidance tracking(MMG). Initially, we perform joint visual feature extraction on both the template and search areas. Subsequently, these features undergo separate self-decoding processes, followed by mutual guidance decoding to reconstruct the original search and template images. This process fosters mutual understanding between the images, facilitating improved learning of object states and shapes across different frames. During the inference phase, we offload the decoder and implement a simple and effective tracker. Experimental results indicate that our proposed method is effective that the mutual guidance strategy can achieve state-of-the-art performance on five tracking datasets.

Keywords: Visual Tracking, Localization, SLAM
Abstract: Monocular visual odometry (MVO) is vital in autonomous navigation and robotics, providing a cost-effective and flexible motion tracking solution, but the inherent scale ambiguity in monocular setups often leads to cumulative errors over time. In this paper, we present BEV-ODOM, a novel MVO framework leveraging the Bird's Eye View (BEV) Representation to address scale drift. Unlike existing approaches, BEV-ODOM integrates a depth-based perspective-view (PV) to BEV encoder, a correlation feature extraction neck, and a CNN-MLP-based decoder, enabling it to estimate motion across three degrees of freedom without the need for depth supervision or complex optimization techniques. Our framework reduces scale drift in long-term sequences and achieves accurate motion estimation across various datasets, including NCLT, Oxford, and KITTI. The results indicate that BEV-ODOM outperforms current MVO methods, demonstrating reduced scale drift and higher accuracy.

Keywords: Computer Vision for Automation, Visual Learning, Deep Learning for Visual Perception
Abstract: Recognizing daily human activities is essential for domestic robots to assist humans effectively in indoor environments. These activities typically involve sequences of interactions between humans and objects across different locations and times within a household. Identifying these events and understanding their temporal and spatial relationships is crucial for accurately modeling human behavior patterns. However, most current methods and datasets for human activity recognition focus on identifying singular events at specific moments and locations, neglecting the complexity of activities that span multiple times and places. To address this gap, we collected data on human activity patterns over a single day through crowdsourcing. Based on this, we introduce a novel synthetic video question-answering dataset. Our proposed dataset includes videos of daily activities accompanied by question-answer pairs that require models to reason about sequences of activities in both time and space. We evaluated state-of-the-art methods against our dataset, highlighting their limitations in handling the intricate spatio-temporal dynamics of human activity sequences. To improve upon these methods, we propose a two-stage model. The proposed model initially decodes the detailed content of individual videos using a transformer-based approach, then employs LLMs for advanced spatio-temporal reasoning across multiple videos. We hope our research provides valuable benchmarks and insights, paving the way for advancements in the recognition of daily human activity patterns.

Keywords: Deep Learning in Grasping and Manipulation, Imitation Learning, Perception for Grasping and Manipulation
Abstract: Task-oriented object grasping and rearrangement are key skills for robots, which have to perform versatile real-world manipulation tasks. However, they remain challenging due to partial observations of the objects and shape variations in categorical objects. In this paper, we present the Multi-feature Implicit Model (MIMO), a novel object representation that encodes multiple spatial features between a point and an object in an implicit neural field. Training such a model on multiple features ensures that it embeds the object shapes consistently in different aspects, thus improving its performance in object shape reconstruction from partial observation, shape similarity measure, and modeling spatial relations between objects. Based on MIMO, we propose a framework to learn task-oriented object grasping and rearrangement from single or multiple human demonstration videos. The evaluations in simulation show that our approach outperforms the state-of-the-art methods for multi- and single-view observations. Real-world experiments demonstrate the efficacy of our approach in one- and few-shot imitation learning of manipulation tasks.

Keywords: Visual Learning, RGB-D Perception, Deep Learning for Visual Perception
Abstract: The reliance on accurate camera poses is a significant barrier to the widespread deployment of Neural Radiance Fields (NeRF) models for 3D reconstruction and SLAM tasks. The existing method introduces monocular depth priors to jointly optimize the camera poses and NeRF, which fails to fully exploit the depth priors and neglects the impact of their inherent noise. In this paper, we propose Truncated Depth NeRF (TD-NeRF), a novel approach that enables training NeRF from unknown camera poses - by jointly optimizing learnable parameters of the radiance field and camera poses. Our approach explicitly utilizes monocular depth priors through three key advancements: 1) we propose a novel depth-based ray sampling strategy based on the truncated normal distribution, which improves the convergence speed and accuracy of pose estimation; 2) to circumvent local minima and refine depth geometry, we introduce a coarse-to-fine training strategy that progressively improves the depth precision; 3) we propose a more robust inter-frame point constraint that enhances robustness against depth noise during training. The experimental results on three datasets demonstrate that TD-NeRF achieves superior performance in the joint optimization of camera pose and NeRF, surpassing prior works, and generates more accurate depth geometry. The implementation of our method has been released at https://github.com/nubot-nudt/TD-NeRF.

Keywords: Visual Learning, Learning Categories and Concepts, Task Planning
Abstract: Visual planning simulates how humans make decisions to achieve desired goals in the form of searching for visual causal transitions between an initial visual state and a final visual goal state. It has become increasingly important in egocentric vision with its advantages in guiding agents to perform daily tasks in complex environments. In this paper, we propose an interpretable and generalizable visual planning framework consisting of i) a novel Substitution-based Concept Learner (SCL) that abstracts visual inputs into disentangled concept representations, ii) symbol abstraction and reasoning that performs task planning via the learned symbols, and iii) a Visual Causal Transition model (ViCT) that grounds visual causal transitions to semantically similar real-world actions. Given an initial state, we perform goal-conditioned visual planning with a symbolic reasoning method fueled by the learned representations and causal transitions to reach the goal state. To verify the effectiveness of the proposed model, we collect a large-scale visual planning dataset based on AI2-THOR, dubbed as CCTP. Extensive experiments on this challenging dataset demonstrate the superior performance of our method in visual planning. Empirically, we show that our framework can generalize to unseen task trajectories, unseen object categories, and real-world data. Further details of this work are provided at https://fqyqc.github.io/ConTranPlan/.

Keywords: Visual Learning, Datasets for Human Motion, Simulation and Animation
Abstract: Human reasoning comprises the ability to understand and reason about the current action solely based on past information. To provide effective assistance in an eldercare or household environment an assistive robot or intelligent assistive system has to assess human actions correctly. Based on this presumption, the task of online action detection determines the current action solely based on the past without access to future information. During inference, the performance of the model is largely impacted by the attributes of the underlying training dataset. However, as high costs and ethical concerns are associated with the real-world data collection process, synthetically created data provides a way to mitigate these problems while providing additional data for the training process of the underlying action detection model to improve performance.

Due to the inherent domain shift between the synthetic and real data, we introduce a new egocentric dataset called Human Kitchen Interactions (HKI) to investigate the sim-to-real gap. Our dataset contains in total 100 synthetic and real videos in which 21 different actions are executed in a kitchen environment. The synthetic data is acquired in an egocentric virtual reality (VR) setup while capturing the virtual environment in a game engine. We evaluate state-of-the-art online action detection models on our dataset and provide insights into sim-to-real domain shift. Upon acceptance, we will release our dataset and the corresponding features at https://c-patsch.github.io/HKI/.

Keywords: Visual Learning, Perception-Action Coupling, Motion and Path Planning
Abstract: Many autonomous robotic applications require object-level understanding when deployed. Actively reconstructing objects of interest, i.e. objects with specific semantic meanings, is therefore relevant for a robot to perform downstream tasks in an initially unknown environment. In this work, we propose a novel framework for semantic-targeted active reconstruction using posed RGB-D measurements and 2D semantic labels as input. The key components of our framework are a semantic implicit neural representation and a compatible planning utility function based on semantic rendering and uncertainty estimation, enabling adaptive view planning to target objects of interest. Our planning approach achieves better reconstruction performance in terms of mesh and novel view rendering quality compared to implicit reconstruction baselines that do not consider semantics for view planning. Our framework further outperforms a state-of-the-art semantic-targeted active reconstruction pipeline based on explicit maps, justifying our choice of utilising implicit neural representations to tackle semantic-targeted active reconstruction problems.

Keywords: Visual Learning, Deep Learning in Grasping and Manipulation, Imitation Learning
Abstract: In this work, we introduce the Virtual In-Hand Eye Transformer (VIHE), a novel method designed to enhance 3D manipulation capabilities through action-aware view rendering. VIHE autoregressively refines actions in multiple stages by conditioning on rendered views posed from action predictions in the earlier stages. These virtual in-hand views provide a strong inductive bias for effectively recognizing the correct pose for the hand, especially for challenging high-precision tasks such as peg insertion. On 18 manipulation tasks in RLBench simulated environments, VIHE achieves a new state-of-the-art, with a 12% absolute improvement, increasing from 65% to 77% over the existing state-of-the-art model using 100 demonstrations per task. In real-world scenarios, VIHE can learn manipulation tasks with just a handful of demonstrations, highlighting its practical utility. Videos and code can be found at our project site: https://vihe-3d.github.io.

Keywords: Visual Learning, RGB-D Perception
Abstract: Satellite images provide an effective way to observe the earth surface on a large scale. 3D landscape models can provide critical structural information, such as forestry and crop growth. However, there has been very limited research to estimate the depth and the 3D models of the earth based on satellite images. LiDAR measurements on satellites are usually quite sparse. RGB images have higher resolution than LiDAR, but there has been little research on 3D surface measurements based on satellite RGB images. In comparison with in-situ sensing, satellite RGB images are usually low resolution. In this research, we explore the method that can enhance the satellite image resolution to generate super-resolution images and then conduct depth estimation and 3D reconstruction based on higher-resolution satellite images. Leveraging the strong generation capability of diffusion models, we developed a simultaneous diffusion model learning framework that can train diffusion models for both super-resolution images and depth estimation. With the super-resolution images and the corresponding depth maps, 3D surface reconstruction models with detailed landscape information can be generated. We evaluated the proposed methodology on multiple satellite datasets for both super-resolution and depth estimation tasks, which have demonstrated the effectiveness of our methodology.

Keywords: Visual Learning
Abstract: In transformer-based methods for point cloud instance segmentation, bipartite matching is used to establish one-to-one correspondences between predictions and ground truths. However, in early training stages, matches can be unstable and inconsistent between epochs, requiring the model to frequently adjust its learning path, thus reducing the quality of model convergence. To address this challenge, we propose the contrastive mask denoising transformer for 3D instance segmentation, which utilizes a mask denoising module to guide the model towards a more stable optimization path in early training stages. Furthermore, we introduce a multi-patternaware query selection module to assist the model learn multiple patterns at one position such that clustered objects can be discerned. In addition, the proposed modules are “plug and play”, which can easily be integrated into transformer-based architectures. Experimental results on ScanNetv2 dataset show that the proposed modules improve the performance of multiple pipelines, notably achieving +1.0 mAP on the main pipeline.

Keywords: Visual Learning, Deep Learning Methods, Deep Learning for Visual Perception
Abstract: Traditional motion-based approaches achieve target tracking by estimating the relative movement of target between two consecutive frames. However, they usually overlook local motion information of the target and fail to exploit historical frame information effectively. To overcome the above limitations, we propose a point-level flow method with multi-frame information for 3D SOT task, called FlowTrack. Specifically, by estimating the flow for each point in the target, our method could capture the local motion details of target, thereby improving the localization accuracy of a specific object. At the same time, to handle scenes with sparse points, we present a learnable target feature as the bridge to efficiently integrate target information from past frames. Moreover, we design a novel Instance Flow Head to transform dense point-level flow into instance-level motion, effectively aggregating local motion information to obtain global target motion. Finally, our method achieves competitive performance with improvements of 5.9% on KITTI and 2.9% on NuScenes, compared to the next best method.

Keywords: Visual Learning, Deep Learning Methods, Reinforcement Learning
Abstract: An excellent representation is crucial for reinforcement learning (RL) performance, especially in vision-based reinforcement learning tasks. The quality of the environment representation directly influences the achievement of the learning task. Previous vision-based RL typically uses explicit or implicit ways to represent environments, such as images, points, voxels, and neural radiance fields. However, these representations contain several drawbacks. They cannot either describe complex local geometries or generalize well to unseen scenes, or require precise foreground masks. Moreover, these implicit neural representations are akin to a ``black box", significantly hindering interpretability. 3D Gaussian Splatting (3DGS), with its explicit scene representation and differentiable rendering nature, is considered a revolutionary change for reconstruction and representation methods. In this paper, we propose a novel Generalizable Gaussian Splatting framework to be the representation of RL tasks, called GSRL. Through validation in the RoboMimic environment, our method achieves better results than other baselines in multiple tasks, improving the performance by 10%, 44%, and 15% compared with baselines on the hardest task. This work is the first attempt to leverage generalizable 3DGS as a representation for RL.

Keywords: Visual Learning, Recognition, Bioinspired Robot Learning
Abstract: Deploying energy-efficient deep learning algorithms on computational-limited devices, such as robots, is still a pressing issue for real-world applications. Spiking Neural Networks (SNNs), a novel brain-inspired algorithm, offer a promising solution due to their low-latency and low-energy properties over traditional Artificial Neural Networks (ANNs). Despite their advantages, the dense structure of deep SNNs can still result in extra energy consumption. The Lottery Ticket Hypothesis (LTH) posits that within dense neural networks, there exist winning Lottery Tickets (LTs), namely sub-networks, that can be obtained without compromising performance. Inspired by this, this paper delves into the spiking-based LTs (SLTs), examining their unique properties and potential for extreme efficiency. Then, two significant sparse rewards are gained through comprehensive explorations and meticulous experiments on SLTs across various dense structures. Moreover, a sparse algorithm tailored for spiking transformer structure, which incorporates convolution operations into the Patch Embedding Projection (ConvPEP) module, has been proposed to achieve Multi-level Sparsity (MultiSp). MultiSp refers to (1) Patch number sparsity; (2) ConvPEP weights sparsity and binarization; and (3) ConvPEP activation layer binarization. Extensive experiments demonstrate that our method achieves extreme sparsity with only a sight performance decrease, paving the way for deploying energy-efficient neural networks in robotics and beyond.

Keywords: Visual Learning, Object Detection, Segmentation and Categorization, Recognition
Abstract: Visual relative pose estimation is widely used in multi-robot systems. While semantic key points offer a promising solution for 6DoF pose estimation, manual data labeling for network training remains unavoidable. In this paper, we introduce a novel method that jointly estimates the semantic key point detection model and 6DoF camera pose. Our key idea is to leverage the 3D-2D projection to produce pseudo labels for detection model training while taking the key point predictions as landmarks for 6DoF camera pose estimation. Compared with state-of-the-art works, our method eliminates the need for calibration and time synchronization of multicamera systems, requiring only a handful of manually labeled data, which significantly improves the training efficiency. The experiment validates the effectiveness and practicality of our method in public datasets and real-world robotic applications. Code and data are made available.

Keywords: AI-Based Methods, Reinforcement Learning, Human Factors and Human-in-the-Loop
Abstract: Despite the impressive performance of Reinforcement Learning (RL), the black-box neural network backbone hinders users from trusting and deploying trained agents in real-world applications where safety is crucial. In order to make agents more trustworthy and controllable, for a given RL-trained policy, we propose to enhance its interpretability and make it human-controllable without retraining. We accomplish this goal by following a 3-step pipeline: 1) We interpret neurons by analyzing the causal effect of neurons on the kinematic attributes; To help agents unlock novel skills and enable human to assist agents in accomplishing tasks, 2) we locate the X-neuron, the optimal neuron that is capable of evoking the desired behavior; 3) and edit its activation values to achieve the precise control. We evaluate our method on various RL tasks ranging from autonomous driving to robot locomotion, and the results display that our approach outperforms previous work regarding almost all evaluation metrics. Through enhancing interpretability and introducing human control, the agents can improve safety and performance, even in unseen environments and novel tasks. For locomotion robots simply trained to walk forward, our method unlocks diverse controllable behaviors ranging from jump to backflip.

Keywords: Deep Learning Methods, Grippers and Other End-Effectors, Micro/Nano Robots
Abstract: Acoustic holography has emerged as a cutting-edge technique for constructing a micro-robot acoustic end-effector for non-contact manipulation. With favorable features of sufficient penetration depth, high spatial resolution, non-injury, and unlimited work environment, it exhibits promising potential in various applications encompassing biomedicine, in-vivo surgery, and lab-on-a-chip applications. Binary Amplitude-Only Hologram (BAOH) with simple structure is highly compatible with acoustic metasurface, which supports to modulate acoustic field exquisitely with straightforward production at relatively low cost. However, existing researches mainly focus on hardware implementation instead of arithmetical design. In the present study, we propose a deep learning based BAOH generation algorithm for constructing precise, reconfigurable and high-resolution end-effector based on acoustic field. Specifically, we incorporated an acoustic wave propagation model into the deep neural network, in favor of bypassing the laborious collection of labeled data and facilitating the model to learn the inverse mapping. Additionally, differentiable binarization with adaptive threshold is embedded into the framework, with the purpose of circumventing the gradient invalidation and alleviating the information loss caused by binarization process. The simulation experiments show that the proposed algorithm is capable to predict BAOH that supports precise, robust, versatile and real-time construction of acoustic end-effector, enjoying broad prospects in various applications related to micro-robotic manipulation.

Keywords: Deep Learning Methods, Machine Learning for Robot Control, Kinematics
Abstract: Multi-rotor aerial autonomous vehicles (MAVs) primarily rely on vision for navigation purposes. However, visual localization and odometry techniques suffer from poor performance in low or direct sunlight, a limited field of view, and vulnerability to occlusions. Acoustic sensing can serve as a complementary or even alternative modality for vision in many situations, and it also has the added benefits of lower system cost and energy footprint, which is especially important for micro aircraft. This paper proposes actively controlling and shaping the aircraft propulsion noise generated by the rotors to benefit localization tasks, rather than considering it a harmful nuisance. We present a neural network architecture for selfnoise-based localization in a known environment. We show that training it simultaneously with learning time-varying rotor phase modulation achieves accurate and robust localization. The proposed methods are evaluated using a computationally affordable simulation of MAV rotor noise in 2D acoustic environments that is fitted to real recordings of rotor pressure fields. Code and data are accompanied.

Keywords: Deep Learning Methods, Representation Learning, Object Detection, Segmentation and Categorization
Abstract: The annotation process for 3D point cloud data is more complex than for image data, and training with a small amount of annotated data significantly reduces the performance of deep learning models. Unsupervised learning can better utilize large amounts of unlabeled point cloud data for model pretraining, thereby achieving excellent performance on small-scale datasets. However, many existing 3D point cloud unsupervised learning methods are primarily focused on single-object CAD point clouds and may not be suitable for larger-scale autonomous driving LiDAR point clouds. To address this challenging problem, we propose a voxel contrast-based unsupervised learning method (VoxelContrast), which adapts well to different types of point cloud data through voxelization and can be seamlessly integrated with existing model frameworks. Specifically, we utilize voxelization methods to preprocess point cloud data. Then, we incorporate voxel information into contrastive learning, facilitating the creation of more meaningful positive and negative sample pairs. Finally, we conduct unsupervised training of the model using instance discrimination as the proxy task. Our method was validated in two downstream tasks: point cloud shape classification and 3D object detection. Experimental results demonstrated that models pretrained using a substantial amount of unlabeled data can further enhance the effectiveness of existing supervised learning methods.

Keywords: Deep Learning Methods, Performance Evaluation and Benchmarking, Autonomous Vehicle Navigation
Abstract: Robustness against Out-of-Distribution (OoD) samples is a key performance indicator of a trajectory prediction model. However, the development and ranking of state-of-the-art (SotA) models are driven by their In-Distribution (ID) performance on individual competition datasets. We present an OoD testing protocol that homogenizes datasets and prediction tasks across two large-scale motion datasets. We introduce a novel prediction algorithm based on polynomial representations for agent trajectory and road geometry on both the input and output sides of the model. With a much smaller model size, training effort, and inference time, we reach near SotA performance for ID testing and significantly improve robustness in OoD testing. Within our OoD testing protocol, we further study two augmentation strategies of SotA models and their effects on model generalization. Highlighting the contrast between ID and OoD performance, we suggest adding OoD testing to the evaluation criteria of trajectory prediction models.

Keywords: Deep Learning Methods, Bimanual Manipulation, Sensorimotor Learning
Abstract: Robots that autonomously operate in human living environments require the ability to adapt to unpredictable changes and flexibly handle a variety of tasks. Particularly, coordinated bimanual motions are essential for enabling tasks that are difficult with just one hand, such as grasping bulky objects, transporting heavy loads, and precision work. Traditional methods of generating robot motions typically involve executing pre-programmed motions, making it challenging to adapt to complex and unpredictable environmental changes. To address this issue, our research focuses on generating diverse motions that can flexibly adapt to environmental changes based on Deep Predictive Learning from a small amount of real-world data. Previous Deep Predictive Learning models have generated the motions of a robot's left and right arms by a single LSTM, making it difficult to operate them independently. Therefore, we propose a new hierarchical Deep Predictive Learning model specialized for generating coordinated bimanual motions. This model comprises three components: a Left-LSTM, which learns the body and visual information on the robot's left side, a Right-LSTM that performs a similar function for the right side, and a Union-LSTM which integrates this information at a higher level. To verify the effectiveness of the proposed model, we conducted bimanual grasping experiments with multiple different objects using two different robots. The experimental results showed that independent of hardware, our model demonstrated a higher success rate compared to the traditional approach, indicating its enhanced capability in coordinating bimanual motions.

Keywords: Deep Learning Methods
Abstract: To ensure safe cooperative driving in mixed traffic with both manned and unmanned vehicles, it is crucial to understand and model the driving styles of human drivers. This paper explores how to develop accurate recognition of driving style and use that for the prediction of vehicle motion, which enables better performance in cooperative driving. A simulation testbed that consists of a driving simulator and a copilot is first introduced for the purpose of data collection and testing. A Long Short-Term Memory (LSTM)-based network that models human driving styles and predicts driving acceleration is developed. Standalone tests are conducted to examine the model performance in the simulation testbed. Finally, the model is evaluated in a series of merging experiments that involves 5 vehicles.

Keywords: Deep Learning Methods, Recognition
Abstract: 3D point clouds enhanced the robot's ability to perceive the geometrical information of the environments, making it possible for many downstream tasks such as grasp pose detection and scene understanding. The performance of these tasks, though, heavily relies on the quality of data input, as incomplete can lead to poor results and failure cases. Recent training loss functions designed for deep learning-based point cloud completion, such as Chamfer distance (CD) and its variants (e.g., HyperCD), imply a good gradient weighting scheme can significantly boost performance. However, these CD-based loss functions usually require data-related parameter tuning, which can be time-consuming for data-extensive tasks. To address this issue, we aim to find a family of weighted training losses (weighted CD) that requires no parameter tuning. To this end, we propose a search scheme, Loss Distillation via Gradient Matching, to find good candidate loss functions by mimicking the learning behavior in backpropagation between HyperCD and weighted CD. Once this is done, we propose a novel bilevel optimization formula to train the backbone network based on the weighted CD loss. We observe that: (1) with proper weighted functions, the weighted CD can always achieve similar performance to HyperCD, and (2) the Landau weighted CD, namely Landau CD, can outperform HyperCD for point cloud completion and lead to new state-of-the-art results on several benchmark datasets. Our demo code is available at https://github.com/Zhang-VISLab/IROS2024-LossDistillationWeightedCD.

Keywords: Deep Learning Methods
Abstract: Few-shot fine-grained human (FGH) action recognition is crucial in the context of human-robot interaction within open-set real-world environments. Existing works mainly focus on features extracted from RGB frames. However, their performances are drastically impacted in challenging scenarios, such as high-dynamic or low lighting conditions. Event cameras can independently and sparsely capture brightness changes in a scene at microsecond resolution and high dynamic range, which offer a promising solution. However, the modality differences between events and RGB frames, and the lack of paired fine-grained data hinder the development of event-based FGH action recognition. Therefore, in this paper, we introduce the first Event Camera Fine-grained Human Action (E-FAction) dataset. This dataset comprises 3304 paired `event stream and RGB sequence', covering 15 coarse action classes and 128 fine-grained actions. Then, we develop a versatile event feature extractor. Considering the spatial sparsity of event stream, we design two modules to mine the temporal motion and semantic features under the guidance of paired RGB frames, facilitating robust weight initialization for the feature extractor in few-shot FGH action recognition. We conduct extensive experiments on both published and our built synthetic and real datasets, and consistently achieve state-of-the-art performance compared to existing baselines. Code and dataset will be available at link.

Keywords: Deep Learning Methods, Semantic Scene Understanding, SLAM
Abstract: Abstract— Recent advancements in neural implicit fields for Simultaneous Localization and Mapping (SLAM) have provided breakthroughs. However, the benefits of reconstruction results to the perception ability of robot are minimal. Therefore, we propose FI-SLAM, a dense semantic instance SLAM system based on neural implicit representation, which significantly aids robots in better understanding the scene. FI-SLAM employs a coordinate and plane joint encoding method, which reduces the difficulty of feature storage by flattening the feature space. Furthermore, to improve representation efficiency, we use the method of adjacent feature level linear interpolation to describe features. We propose a feature fusion (FF) method to merge the object features with the scene features. The fused feature vector enhances the reconstruction accuracy of the local scene while ensuring the global reconstruction effect. It has improved the global reconstruction effect of the scene and the accuracy of camera tracking. Numerous experiments on synthetic and realworld datasets demonstrate that our method can assure accurate tracking precision, high-fidelity reconstruction results, and complete semantic instance maps. In summary, the algorithm we proposed heavily augments the scene perception capabilities of robot.

Keywords: Deep Learning Methods, Deep Learning in Grasping and Manipulation, Compliant Assembly
Abstract: The study addresses the foundational and challenging task of peg-in-hole assembly in robotics, where misalignments caused by sensor inaccuracies and mechanical errors often result in insertion failures or jamming. This research introduces PolyFit, representing a paradigm shift by transitioning from a reinforcement learning approach to a supervised learning methodology. PolyFit is a Force/Torque (F/T)-based supervised learning framework designed for 5-DoF peg-in-hole assembly. It utilizes F/T data for accurate extrinsic pose estimation and adjusts the peg pose to rectify misalignments. Extensive training in a simulated environment involves a dataset encompassing a diverse range of peg-hole shapes, extrinsic poses, and their corresponding contact F/T readings. The study proposes a sim-to-real adaptation method for real-world application, using a sim-real paired dataset to enable effective generalization to complex and unseen polygon shapes. Real-world evaluations demonstrate substantial success rates of 96.7% and 91.3%, highlighting the robustness and adaptability of the proposed method. Videos of data generation and experiments are available online at https://sites.google.com/view/polyfit-peginhole.

Keywords: Reinforcement Learning, Deep Learning in Grasping and Manipulation, Probabilistic Inference
Abstract: Robot arms should be able to learn new tasks. One framework here is reinforcement learning, where the robot is given a reward function that encodes the task, and the robot autonomously learns actions to maximize its reward. Existing approaches to reinforcement learning often frame this problem as a Markov decision process, and learn a policy (or a hierarchy of policies) to complete the task. These policies reason over hundreds of fine-grained actions that the robot arm needs to take: e.g., moving slightly to the right or rotating the end-effector a few degrees. But the manipulation tasks that we want robots to perform can often be broken down into a small number of high-level motions: e.g., reaching an object or turning a handle. In this paper we therefore propose a waypoint-based approach for model-free reinforcement learning. Instead of learning a low-level policy, the robot now learns a trajectory of waypoints, and then interpolates between those waypoints using existing controllers. Our key novelty is framing this waypoint-based setting as a sequence of multi-armed bandits: each bandit problem corresponds to one waypoint along the robot's motion. We theoretically show that an ideal solution to this reformulation has lower regret bounds than standard frameworks. We also introduce an approximate posterior sampling solution that builds the robot's motion one waypoint at a time. Results across benchmark simulations and two real-world experiments suggest that this proposed approach learns new tasks more quickly than state-of-the-art baselines. See our website here: https://collab.me.vt.edu/rl-waypoints/

Keywords: Reinforcement Learning, Art and Entertainment Robotics, Machine Learning for Robot Control
Abstract: Free-roaming dollies enhance filmmaking with dynamic movement, but challenges in automated camera control remain unresolved. Our study advances this field by applying Reinforcement Learning (RL) to automate dolly-in shots using free-roaming ground-based filming robots, overcoming traditional control hurdles. We demonstrate the effectiveness of combined control for precise film tasks by comparing it to independent control strategies. Our robust RL pipeline surpasses traditional Proportional-Derivative controller performance in simulation and proves its efficacy in real-world tests on a modified ROSBot 2.0 platform equipped with a camera turret. This validates our approach's practicality and sets the stage for further research in complex filming scenarios, contributing significantly to the fusion of technology with cinematic creativity. This work presents a leap forward in the field and opens new avenues for research and development, effectively bridging the gap between technological advancement and creative filmmaking.

Keywords: Representation Learning, Semantic Scene Understanding, AI-Based Methods
Abstract: We study the problem of multimodal physical scene understanding, where an embodied agent needs to find fallen objects by inferring object properties, direction, and distance of an impact sound source. Previous works adopt feed-forward neural networks to directly regress the variables from sound, leading to poor generalization and domain adaptation issues. In this paper, we illustrate that learning a disentangled model of acoustic formation, referred to as disentangled acoustic field (DAF), to capture the sound generation and propagation process, enables the embodied agent to construct a spatial uncertainty map over where the objects may have fallen. We demonstrate that our analysis-by-synthesis framework can jointly infer sound properties by explicitly decomposing and factorizing the latent space of the disentangled model. We further show that the spatial uncertainty map can significantly improve the success rate for the localization of fallen objects by proposing multiple plausible exploration locations.

Keywords: Representation Learning, Autonomous Agents, AI-Enabled Robotics
Abstract: Trajectory generation and trajectory prediction are two critical tasks in autonomous driving, which generate various trajectories for testing during development and predict the trajectories of surrounding vehicles during operation, respectively. In recent years, emerging data-driven deep learning-based methods have shown great promise for these two tasks in learning various traffic scenarios and improving average performance without assuming physical models. However, it remains a challenging problem for these methods to ensure that the generated/predicted trajectories are physically realistic and controllable. This challenge arises because learning-based approaches often function as opaque black boxes and do not adhere to physical laws. Conversely, existing model-based methods provide physically feasible results but are constrained by predefined model structures, limiting their capabilities to address complex scenarios. To address the limitations of these two types of approaches, we propose a new method that integrates kinematic knowledge into neural stochastic differential equations (SDE) and designs a variational autoencoder based on this latent kinematics-aware SDE (LK-SDE) to generate vehicle motions. Experimental results demonstrate that our method significantly outperforms both model-based and learning-based baselines in producing physically realistic and precisely controllable vehicle trajectories. Additionally, it performs well in predicting unobserved physical variables in the latent space.

Keywords: Representation Learning, Deep Learning in Grasping and Manipulation
Abstract: Autonomous robotic systems capable of learning novel manipulation tasks are poised to transform industries from manufacturing to service automation. However, current methods (e.g., VIP and R3M) still face significant hurdles, notably the domain gap among robotic embodiments and the sparsity of successful task executions within specific action spaces, resulting in misaligned and ambiguous task representations. We introduce Ag2Manip (Agent Agnostic representations for Manipulation), a framework aimed at addressing these challenges through two key innovations: (1) an agent-agnostic visual representation derived from human manipulation videos, with the specifics of embodiments obscured to enhance generalizability; and (2) an agent-agnostic action representation abstracting a robot's kinematics to a universal agent proxy, emphasizing crucial interactions between end-effector and object. Ag2Manip has been empirically validated across simulated benchmarks, showing a 325% performance increase without relying on domain-specific demonstrations. Ablation studies further underline the essential contributions of the agent-agnostic visual and action representations to this success. Extending our evaluations to the real world, Ag2Manip significantly improves imitation learning success rates from 50% to 77.5%, demonstrating its effectiveness and generalizability across both simulated and real environments.

Keywords: Planning, Scheduling and Coordination, Multi-Robot Systems, Path Planning for Multiple Mobile Robots or Agents
Abstract: In multi-robot systems, achieving coordinated missions remains a significant challenge due to the coupled nature of coordination behaviors and the lack of global information for individual robots. To mitigate these challenges, this paper introduces a novel approach, Bi-level Coordination Learning (Bi-CL), that leverages a bi-level optimization structure within a CTDE paradigm. Our bi-level reformulation decomposes the original problem into a reinforcement learning level with reduced action space, and an imitation learning level that gains demonstrations from a global optimizer. Bi-CL further integrates an alignment penalty mechanism, aiming to minimize the discrepancy between the two levels without degrading their training efficiency. We introduce a running example to conceptualize the problem formulation. Simulation results demonstrate that Bi-CL can learn more efficiently and achieve comparable performance with traditional multi-agent reinforcement learning baselines for multi-robot coordination.

Keywords: Reinforcement Learning
Abstract: Reinforcement Learning (RL) has been widely used to solve tasks where the environment consistently provides a dense reward value. However, in real-world scenarios, rewards can often be poorly defined or sparse. Auxiliary signals are indispensable for discovering efficient exploration strategies and aiding the learning process. In this work, inspired by intrinsic motivation theory, we postulate that the intrinsic stimuli of novelty and surprise can assist in improving exploration in complex, sparsely rewarded environments. We introduce a novel sample-efficient method able to learn directly from pixels, an image-based extension of TD3 with an autoencoder called NaSA-TD3. The experiments demonstrate that NaSA-TD3 is easy to train and an efficient method for tackling complex continuous-control robotic tasks, both in simulated environments and real-world settings. NaSA-TD3 outperforms existing state-of-the-art RL image-based methods in terms of final performance without requiring pre-trained models or human demonstrations.

Keywords: Reinforcement Learning, Robot Safety, Robust/Adaptive Control
Abstract: Recent studies reveal that well-performing reinforcement learning (RL) agents in training often lack resilience against adversarial perturbations during deployment. This highlights the importance of building a robust agent before deploying it in the real world. Most prior works focus on developing robust training-based procedures to tackle this problem, including enhancing the robustness of the deep neural network component itself or adversarially training the agent on strong attacks. In this work, we instead study an input transformation-based defense for RL. Specifically, we propose using a variant of vector quantization (VQ) as a transformation for input observations, which is then used to reduce the space of adversarial attacks during testing, resulting in the transformed observations being less affected by attacks. Our method is computationally efficient and seamlessly integrates with adversarial training, further enhancing the robustness of RL agents against adversarial attacks. Through extensive experiments in multiple environments, we demonstrate that using VQ as the input transformation effectively defends against adversarial attacks on the agent's observations.

Keywords: Reinforcement Learning, Robust/Adaptive Control
Abstract: Deep Reinforcement Learning (DRL) has achieved remarkable success, ranging from complex computer games to real-world applications, showing the potential for intelligent agents capable of learning in dynamic environments. However, its application in real-world scenarios presents challenges, including the jerky problem, in which jerky trajectories not only compromise system safety but also increase power consumption and shorten the service life of robotic and autonomous systems. To address jerky actions, a method called conditioning for action policy smoothness (CAPS) was proposed by adding regularization terms to reduce the action changes. This paper further proposes a novel method, named Gradient-based CAPS (Grad-CAPS), that modifies CAPS by reducing the difference in the gradient of action and then uses displacement normalization to enable the agent to adapt to invariant action scales. Consequently, our method effectively reduces zigzagging action sequences while enhancing policy expressiveness and the adaptability of our method across diverse scenarios and environments. In the experiments, we integrated Grad-CAPS with different reinforcement learning algorithms and evaluated its performance on various robotic-related tasks in DeepMind Control Suite and OpenAI Gym environments. The results demonstrate that Grad-CAPS effectively improves performance while maintaining a comparable level of smoothness compared to CAPS and Vanilla agents.

Keywords: Reinforcement Learning, Learning from Demonstration, Task and Motion Planning
Abstract: This paper enhances model-based offline inverse reinforcement learning (IRL) by refining conservative Markov decision process (MDP) frameworks, traditionally employing uncertainty penalties to deter exploitation in uncertain areas. Existing methods, dependent on neural network ensembles to model MDP dynamics and quantify uncertainty through ensemble prediction heuristics, face limitations: they presume Gaussian-distributed state transitions, leading to simplified environmental representations. Additionally, ensemble modeling often results in high variance, indicating potential overfitting and a lack of generalizability. Moreover, the heuristic reliance for uncertainty quantification struggles to fully grasp environmental complexities, offering an incomplete foundation for informed decisions. Maintaining multiple models also demands substantial computational resources. Addressing these shortcomings, we propose leveraging score-based diffusion generative models for dynamic modeling. This method significantly broadens the scope of representable target distributions, surpassing Gaussian constraints. It not only improves the accuracy of transition modeling but also roots uncertainty quantification in diffusion models' theoretical underpinnings, enabling more precise and dependable reward regularization. We further innovate by incorporating a transition stability regularizer (textit{TSR}) into the reward estimation. This novel element embeds stability into the reward learning process, diminishing the influence of transition variability and promoting more consistent policy optimization. Our empirical studies on diverse Mujoco robotic control tasks demonstrate that our diffusion-based methodology not only furnishes more accurate transition estimations but also surpasses conventional ensemble approaches in policy effectiveness. The addition of the textit{TSR} marks a distinctive advancement in offline IRL by enhancing the reward and policy learning efficacy.

Keywords: Reinforcement Learning, Machine Learning for Robot Control, Robot Safety
Abstract: Safe Reinforcement Learning (Safe RL) is one of the prevalently studied subcategories of trial-and-error-based methods with the intention to be deployed on real-world systems. In safe RL, the goal is to maximize reward performance while minimizing constraints, often achieved by setting bounds on constraint functions and utilizing the Lagrangian method. However, deploying Lagrangian-based safe RL in real-world scenarios is challenging due to the necessity of threshold fine-tuning, as imprecise adjustments may lead to suboptimal policy convergence. To mitigate this challenge, we propose a unified Lagrangian-based model-free architecture called Meta Soft Actor-Critic Lagrangian (Meta SAC-Lag). Meta SAC-Lag uses meta-gradient optimization to automatically update the safety-related hyperparameters. The proposed method is designed to address safe exploration and threshold adjustment with minimal hyperparameter tuning requirement. In our pipeline, the inner parameters are updated through the conventional formulation and the hyperparameters are adjusted using the meta-objectives which are defined based on the updated parameters. Our results show that the agent can reliably adjust the safety performance due to the relatively fast convergence rate of the safety threshold. We evaluate the performance of Meta SAC-Lag in five simulated environments against Lagrangian baselines, and the results demonstrate its capability to create synergy between parameters, yielding better or competitive results. Furthermore, we conduct a real-world experiment involving a robotic arm tasked with pouring coffee into a cup without spillage. Meta SAC-Lag is successfully trained to execute the task, while minimizing effort constraints. The success of Meta SAC-Lag in performing the experiment is intended to be a step toward practical deployment of safe RL algorithms to learn the control process of safety-critical real-world systems without explicit engineering.

Keywords: Reinforcement Learning, Machine Learning for Robot Control
Abstract: Reinforcement learning (RL) policies are prone to high frequency oscillations, specially undesirable when deploying to hardware in the real-world. In this paper, we identify, categorize, and compare methods from the literature that aim to mitigate high frequency oscillations in deep RL. We define two broad classes: loss regularization and architectural methods. At their core, these methods incentivize learning a smooth mapping, such that nearby states in the input space produce nearby actions in the output space. We present benchmarks in terms of policy performance and control smoothness on traditional RL environments from Gymnasium and a complex manipulation task, as well as three robotics locomotion tasks that include deployment and evaluation with real-world hardware. Finally, we also propose hybrid methods that combine elements from both loss regularization and architectural methods. We find that the best-performing hybrid outperforms other methods, and improves control smoothness by 26.8% over the baseline, with a worst case performance degradation of just 2.8%.

Keywords: Reinforcement Learning, SLAM, Visual Tracking
Abstract: Large scale active exploration has recently revealed limitations of visual SLAM’s tracking ability. Active view planning methods, based on reinforcement learning, have been proposed to improve visual tracking robustness.

In this work, we expose the limitations of deep reinforcement learning-based visual SLAM over longer routes. We demonstrate that additional modalities (depth, scene layout) offer little improvement. Furthermore, reward shaping is not the main reason behind the short-sightedness of the state-of-the-art visual SLAM tracker. We propose a novel video vision transformer-based architecture that improves the farsightedness of the visual tracker, which results in the completion of longer routes with efficient paths.

Out of 60 challenging routes, our approach manages to complete 56 routes, which is a three-fold improvement over the state-of-the-art active view mapping (DI-SLAM) baseline. Interestingly, ORBSLAM3 was unable to complete a single route without tracking failure.

Keywords: Reinforcement Learning, Multi-Robot Systems, Distributed Robot Systems
Abstract: In multi-agent reinforcement learning (MARL), the Centralized Training with Decentralized Execution (CTDE) framework is pivotal but struggles due to a gap: global state guidance in training versus reliance on local observations in execution, lacking global signals. Inspired by human societal consensus mechanisms, we introduce the Hierarchical Consensus-based Multi-Agent Reinforcement Learning (HC-MARL) framework to address this limitation. HC-MARL employs contrastive learning to foster a global consensus among agents, enabling cooperative behavior without direct communication. This approach enables agents to form a global consensus from local observations, using it as an additional piece of information to guide collaborative actions during execution. To cater to the dynamic requirements of various tasks, consensus is divided into multiple layers, encompassing both short-term and long-term considerations. Short-term observations prompt the creation of an immediate, low-layer consensus, while long-term observations contribute to the formation of a strategic, high-layer consensus. This process is further refined through an adaptive attention mechanism that dynamically adjusts the influence of each consensus layer. This mechanism optimizes the balance between immediate reactions and strategic planning, tailoring it to the specific demands of the task at hand. Extensive experiments and real-world applications in multi-robot systems showcase our framework's superior performance, marking significant advancements over baselines.

Keywords: Reinforcement Learning, Representation Learning, Visual Learning
Abstract: Reinforcement Learning (RL) algorithms can learn robotic control tasks from visual observations, but they often require a large amount of data, especially when the visual scene is complex and unstructured. In this paper, we explore how the agent’s knowledge of its shape can improve the sample efficiency of visual RL methods. We propose a novel method, Disentangled Environment and Agent Representations (DEAR), that uses the segmentation mask of the agent as supervision to learn disentangled representations of the environment and the agent through feature separation constraints. Unlike previous approaches, DEAR does not require reconstruction of visual observations. These representations are then used as an auxiliary loss to the RL objective, encouraging the agent to focus on the relevant features of the environment. We evaluate DEAR on two challenging benchmarks: Distracting DeepMind control suite and Franka Kitchen manipulation tasks. Our findings demonstrate that DEAR surpasses state-of-the-art methods in sample efficiency, achieving comparable or superior performance with reduced parameters. Our results indicate that integrating agent knowledge into visual RL methods has the potential to enhance their learning efficiency and robustness.

Keywords: Reinforcement Learning, Transfer Learning
Abstract: Deep reinforcement learning (DRL) has shown remarkable success in simulation domains, yet its application in designing robot controllers remains limited, due to its single-task orientation and insufficient adaptability to environmental changes. To overcome these limitations, we present a novel adaptive agent that leverages transfer learning techniques to dynamically adapt policy in response to different tasks and environmental conditions. The approach is validated through the blimp control challenge, where multitasking capabilities and environmental adaptability are essential. The agent is trained using a custom, highly parallelized simulator built on IsaacGym. We perform zero-shot transfer to fly the blimp in the real world to solve various tasks. We share our code at: https://github.com/robot-perception-group/adaptive_agent.

Keywords: Reinforcement Learning, Machine Learning for Robot Control, Model Learning for Control
Abstract: The application of learning-based control methods in robotics presents significant challenges. One is that model-free reinforcement learning algorithms use observation data with low sample efficiency. To address this challenge, a prevalent approach is model-based reinforcement learning, which involves employing an environment dynamics model. We suggest approximating transition dynamics with symbolic expressions, which are generated via symbolic regression. Approximation of a mechanical system with a symbolic model has fewer parameters than approximation with neural networks, which can potentially lead to higher accuracy and quality of extrapolation. We use a symbolic dynamics model to generate trajectories in model-based policy optimization to improve the sample efficiency of the learning algorithm. We evaluate our approach across various tasks within simulated environments. Our method demonstrates superior sample efficiency in these tasks compared to model-free and model-based baseline methods.

Keywords: Reinforcement Learning, Model Learning for Control, Transfer Learning
Abstract: Domain randomization (DR), which entails training a policy with randomized dynamics, has proven to be a simple yet effective algorithm for reducing the gap between simulation and the real world. However, DR often requires careful tuning of randomization parameters. Methods like Bayesian Domain Randomization (Bayesian DR) and Active Domain Randomization (Adaptive DR) address this issue by automating parameter range selection using real-world experience. While effective, these algorithms often require long computation time, as a new policy is trained from scratch every iteration. In this work, we propose Adaptive Bayesian Domain Randomization via Strategic Fine-tuning (BayRnTune), which inherits the spirit of BayRn but aims to significantly accelerate the learning processes by fine-tuning from previously learned policy. This idea leads to a critical question: which previous policy should we use as a prior during fine-tuning? We investigated four different fine-tuning strategies and compared them against baseline algorithms in five simulated environments, ranging from simple benchmark tasks to more complex legged robot environments. Our analysis demonstrates that our method yields better rewards in the same amount of timesteps compared to vanilla domain randomization or Bayesian DR.

Keywords: Reinforcement Learning, Planning, Scheduling and Coordination, Logistics
Abstract: We consider a warehouse in which dozens of mobile robots and human pickers work together to collect and deliver items within the warehouse. The fundamental problem we tackle, called the order-picking problem, is how these worker agents must coordinate their movement and actions in the warehouse to maximise performance in this task. Established industry methods using heuristic approaches require large engineering efforts to optimise for innately variable warehouse configurations. In contrast, multi-agent reinforcement learning (MARL) can be flexibly applied to diverse warehouse configurations (e.g. size, layout, number/types of workers, item replenishment frequency), and different types of order-picking paradigms (e.g. Goods-to-Person and Person-to-Goods), as the agents can learn how to cooperate optimally through experience. We develop hierarchical MARL algorithms in which a manager agent assigns goals to worker agents, and the policies of the manager and workers are co-trained toward maximising a global objective (e.g. pick rate). Our hierarchical algorithms achieve significant gains in sample efficiency over baseline MARL algorithms and overall pick rates over multiple established industry heuristics in a diverse set of warehouse configurations and different order-picking paradigms.

Keywords: Reinforcement Learning, Haptics and Haptic Interfaces
Abstract: Actively guiding attention is an important mechanism to employ limited processing resources efficiently. The Recurrent Visual Attention Model (RAM) has been successfully applied to process large input images by sequentially attending to smaller image regions with an RL framework. In tactile perception, sequential attention methods are required naturally due to the limited size of the tactile receptive field. The concept of RAM was transferred to the haptic domain by the Haptic Attention Model (HAM) to iteratively generate a fixed number of informative haptic glances for tactile object classification. We extend HAM to a system capable of actively determining when sufficient haptic data is available for reliable classification. To this end, we introduce a hybrid action space, augmenting the continuous glance location with the discrete decision of when to classify. This allows balancing the cost of obtaining new samples against the cost of misclassification, resulting in an optimized number of glances while maintaining reasonable accuracy. We evaluate the efficiency of our approach on a hand-crafted dataset, which allows us to compute the most efficient glance locations.

Keywords: Reinforcement Learning, Autonomous Vehicle Navigation, Motion and Path Planning
Abstract: Autonomous robots exploring unknown environ- ments face a significant challenge: navigating effectively without prior maps and with limited external feedback. This challenge intensifies in sparse reward environments, where traditional exploration techniques often fail. In this paper, we present TopoNav, a novel topological navigation framework that inte- grates active mapping, hierarchical reinforcement learning, and intrinsic motivation to enable efficient goal-oriented exploration and navigation in sparse-reward settings. TopoNav dynamically constructs a topological map of the environment, capturing key locations and pathways. A two-level hierarchical policy architecture, comprising a high-level graph traversal policy and low-level motion control policies, enables effective navigation and obstacle avoidance while maintaining focus on the overall goal. Additionally, TopoNav incorporates intrinsic motivation to guide exploration towards relevant regions and frontier nodes in the topological map, addressing the challenges of sparse extrinsic rewards. We evaluate TopoNav both in the simulated and real- world off-road environments using a Clearpath Jackal robot, across three challenging navigation scenarios: goal-reaching, feature-based navigation, and navigation in complex terrains. We observe an increase in exploration coverage by 7-20%, in success rates by 9-19%, and reductions in navigation times by 15-36% across various scenarios, compared to state-of-the-art methods.

Keywords: Reinforcement Learning, Legged Robots
Abstract: A quadruped robot faces balancing challenges on a six-degrees-of-freedom moving platform, like subways, buses, airplanes, and yachts, due to independent platform motions and resultant diverse inertia forces on the robot. To alleviate these challenges, we present the Learning-based Active Stabilization on Moving Platforms (LAS-MP), featuring a self-balancing policy and system state estimators. The policy adaptively adjusts the robot's posture in response to the platform's motion. The estimators infer robot and platform states based on proprioceptive sensor data. For a systematic training scheme across various platform motions, we introduce platform trajectory generation and scheduling methods. Our evaluation demonstrates superior balancing performance across multiple metrics compared to three baselines. Furthermore, we conduct a detailed analysis of the LAS-MP, including ablation studies and evaluation of the estimators, to validate the effectiveness of each component.

Keywords: Contact Modeling, Dynamics
Abstract: A global modeling methodology based on Koopman operator theory for the dynamics of rigid bodies that make and break contact is presented. Traditionally, robotic systems that contact with their environment are represented as a system comprised of multiple dynamic equations that are switched depending on the contact state. This switching of governing dynamics has been a challenge in both task planning and control. Here, a Koopman lifting linearization approach is presented to subsume multiple dynamics such that no explicit switching is required for examining the dynamic behaviors across diverse contact states. First, it is shown that contact/non-contact transitions are continuous at a microscopic level. This allows for the application of Koopman operator theory to the class of robotic systems that repeat contact/non-contact transitions. Second, an effective method for finding Koopman operator observables for capturing rapid changes to contact forces is presented. The method is applied to the modeling of dynamic peg insertion where a peg collides against and bounces on the chamfer of the hole. Furthermore, the method is applied to the dynamic modeling of a sliding object subject to complex friction and damping properties. Segmented dynamic equations are unified with the Koopman modeling method.

Keywords: Neurorobotics, Biologically-Inspired Robots, Force Control
Abstract: As robots become smarter and more ubiquitous, optimizing the power consumption of intelligent compute becomes imperative towards ensuring the sustainability of technological advancements. Neuromorphic computing hardware makes use of biologically inspired neural architectures to achieve energy and latency improvements compared to conventional von Neumann computing architecture. Applying these benefits to robots has been demonstrated in several works in the field of neurorobotics, typically on relatively simple control tasks. Here, we introduce an example of neuromorphic computing applied to the real-world industrial task of object insertion. We trained a spiking neural network (SNN) to perform force-torque feedback control using a reinforcement learning approach in simulation. We then ported the SNN to the Intel neuromorphic research chip Loihi interfaced with a KUKA robotic arm. At inference time we show latency competitive with current CPU/GPU architectures, and one order of magnitude less energy usage in comparison to traditional low-energy edge-hardware. We offer this example as a proof of concept implementation of a neuromoprhic controller in real-world robotic setting, highlighting the benefits of neuromorphic hardware for the development of intelligent controllers for robots.

Keywords: Force and Tactile Sensing, Deep Learning in Grasping and Manipulation, Force Control
Abstract: The advent of tactile sensors in robotics has sparked many ideas on how robots can leverage direct contact measurements of their environment interactions to improve manipulation tasks. An important line of research in this regard is grasp force control, which aims to manipulate objects safely by limiting the amount of force exerted on the object. While prior works have either hand-modeled their force controllers, employed model-based approaches, or not shown sim-to-real transfer, we propose a model-free deep reinforcement learning approach trained in simulation and then transferred to the robot without further fine-tuning. We, therefore, present a simulation environment that produces realistic normal forces, which we use to train continuous force control policies. A detailed evaluation shows that the learned policy performs similarly or better than a hand-crafted baseline. Ablation studies prove that the proposed inductive bias and domain randomization facilitate sim-to-real transfer. Code, models, and supplementary videos are available on https://sites.google.com/view/rl-force-ctrl

Keywords: Force and Tactile Sensing, Sensor-based Control
Abstract: Robots performing tasks in challenging environments must be supported by control or planning algorithms that exploit sensor feedback to effectively plan the robot’s actions. In this paper, we propose a reactive control law that simultaneously utilizes proximity and tactile feedback to perform a pick-and-place task in an unknown and cluttered environment. Specifically, the presented solution leverages proximity sensing obtained from distributed Time of Flight (ToF) sensors to avoid collision when this does not interfere with the pick-and-place task. Safety is guaranteed by a higher-priority task using tactile feedback that reduces contact forces when a collision occurs. Additionally, we compare the effectiveness of this control scheme with a collision detection and reaction scheme based solely on tactile sensing. Our results demonstrate that the proposed approach reduces the collisions with the environment and the task execution time of the pick-and-place operation.

Keywords: Semiconductor Manufacturing, Motion Control
Abstract: This paper introduces a position control system for a motion stage driven by a low-energy C-core reluctance actuator. The central concept explored here is the utilization of a variable air gap to enable energy-efficient operation of the motion stage. The paper begins by presenting the design and mathematical model of the reluctance-actuated motion system (RAMS). Subsequently, open-loop responses of the RAMS are analyzed under various conditions, including variable air gaps and different applied voltages, highlighting the advantages of a variable air gap, particularly in reducing the required current. Finally, the paper formulates a control approach that combines a feedforward controller to linearize the RAMS’s dynamic behavior and a state feedback controller to achieve tracking performance. Experimental results demonstrate the effectiveness of this control approach in achieving tracking objectives with errors that are less than 2% for constant desired displacement and less than 10% for tracking a sinusoidal reference signal

Keywords: Compliance and Impedance Control, Human-Robot Collaboration, Motion Control
Abstract: Industrial robots have increased payload, repeatability, and reach compared to collaborative robots, however, they have a fixed position controller and low intrinsic admittance. This makes realizing safe contact challenging due to large contact force overshoots in contact transitions and contact instability when the environment and robot dynamics are coupled. To improve safe contact on industrial robots, we propose an admittance controller with inverse model compensation, designed and implemented outside the position controller. By including both the inner loop and outer loop dynamics in its design, the proposed method achieves expanded admittance in terms of increasing both gain and cutoff frequency of the desired admittance. Results from theoretical analyses and experiments on a commercial industrial robot show that the proposed method improves rendering of the desired admittance while maintaining contact stability. We further validate this by conducting actual assembly tasks of plug insertion with fine positioning, switch insertion onto the rail, and colliding the robot end effector with random objects and surfaces, as seen at https://youtu.be/8XfkdHEdWDs.

Keywords: Compliance and Impedance Control, Mobile Manipulation, Physical Human-Robot Interaction
Abstract: As robots shift from industrial to human-centered spaces, adopting mobile manipulators, which expand workspace capabilities, becomes crucial. In these settings, seamless interaction with humans necessitates compliant control. Two common methods for safe interaction, admittance, and impedance control, require force or torque sensors, often absent in lower-cost or lightweight robots. This paper presents an adaption of impedance control that can be used on current-controlled robots without the use of force or torque sensors and shows its application for compliant control of a mobile manipulator. A calibration method is designed that enables estimation of the actuators' current/torque ratios and frictions, used by the adapted impedance controller, and that can handle model errors. The calibration method and the performance of the designed controller are experimentally validated using the Kinova GEN3 Lite arm. Results show that the calibration method is consistent and that the designed controller for the arm is compliant while also being able to track targets with five-millimeter precision when no interaction is present. Additionally, this paper presents two operational modes for interacting with the mobile manipulator: one for guiding the robot around the workspace through interacting with the arm and another for executing a tracking task, both maintaining compliance to external forces. These operational modes were tested in real-world experiments, affirming their practical applicability and effectiveness.

Keywords: Force Control, Hydraulic/Pneumatic Actuators, Dynamics
Abstract: In this paper, a force servo for hydraulic robot joints is designed and applied to an inverted pendulum demo. First, the nonlinear nominal model of the hydraulic cylinder is reviewed, and the nominal integrator is introduced. Second, through a partial fusion of our modeling and control techniques, a force servo is designed based on a modification of the nominal integrator as an intermediate stage. Finally, against the non-short hydraulic pipes (1.9 m), the response is improved by the designed force servo in the presence of rotational motion effects.

Keywords: Motion Control, Mechanism Design, Body Balancing
Abstract: In this paper, we present a spherical wheel-legged mobile robot, aiming to meet the demands of adaptability to complex terrains and high maneuverability. It consists of a spherical main body and five-bar linkage parallel wheel-legged mechanisms. It can switch between legged and spherical modes by extending and retracting its legs according to the demands of the actual environment, thereby enhancing the overall mobility of the robot. By designing the Linear Quadratic Regulator (LQR) controller, we achieve the impact-resistant leg balancing motion and autonomous pitch adjustment for the robot on inclined surfaces in the legged configuration. For the rolling control in the spherical configuration, a hierarchical sliding mode control method and Proportional-Integral-Derivative controller (PID) are employed to control the rolling and turning of the robot. We finally verify The robustness of the robot in legged configuration against disturbances and the stability of its motion in spherical configuration.

Keywords: Motion Control, Robust/Adaptive Control, Underactuated Robots
Abstract: Mobile self-reconfigurable robots (MSRRs), as a novel multi-robot system with flexible configurations and task adaptability, hold promising applications in unstructured task environments. However, existing autonomous docking strategies are primarily applied in laboratory settings and face numerous challenges and limitations in actual applications, including differences in sensor characteristics, safety threats, and saturation constraints. To address these issues, this paper proposes a segmented secure docking control framework based on global localization and local perception to achieve stable and reliable reconfiguration of MSRRs in practical applications. Specific contributions include the implementation of a dual-layer constraint framework for safeness of units in the long-distance phase against velocity and acceleration nested windups, and the integration of active line-of-sight (LOS) correction and adaptive windup driving mobile units to achieve precise and rapid locking of docked positions within the LOS in the close-range phase. Finally, the validity of the proposed method is verified via physical experiments, offering an innovative approach to deploying MSRRs in complex scenarios.

Keywords: Motion Control, Sensor-based Control, Marine Robotics
Abstract: In this paper, we present an attitude control scheme for an autonomous underwater vehicle (AUV), which is based on incremental nonlinear dynamic inversion (INDI). Conventional model-based controllers depend on an exact model of the controlled system, which is difficult to find, especially for marine vehicles subject to highly nonlinear hydrodynamic effects. INDI trades off model accuracy with sensor accuracy by incorporating acceleration feedback and actuator output feedback to linearize a nonlinear system incrementally. Existing research primarily focuses on studying INDI on unmanned aerial vehicles. However, there is barely any research on controlling marine vehicles using INDI. The control task we are performing is a 90 degrees pitch-up maneuver, where the dual-arm intervention AUV Cuttlefish transitions from a horizontal traveling pose to a vertical intervention pose. We compare INDI to a classical model-based control scheme in the maritime test basin at DFKI RIC, Germany, and we find that INDI keeps the AUV much more steady both in the transitioning phase as well as in the station keeping phase.

Keywords: Motion Control, Search and Rescue Robots, Human-Aware Motion Planning
Abstract: We present a viewpoint-based non-linear Model Predictive Control (MPC) for evacuation guiding robots. Specifically, the proposed MPC algorithm enables evacuation guiding robots to track and guide cooperative human targets in emergency scenarios. Our algorithm accounts for the environment layout as well as distances between the robot and human target and distance to the goal location. A key challenge for evacuation guiding robot is the trade-off between its planned motion for leading the target toward a goal position and staying in the target's viewpoint while maintaining line-of-sight for guiding. We illustrate the effectiveness of our proposed evacuation guiding algorithm in both simulated and real-world environments with an Unmanned Aerial Vehicle (UAV) guiding a human. Our results suggest that using the contextual information from the environment for motion planning, increases the visibility of the guiding UAV to the human while achieving faster total evacuation time.

Keywords: Motion Control, Compliance and Impedance Control, Whole-Body Motion Planning and Control
Abstract: Virtual Model Control (VMC) is an approach to design a controller for force-controlled robots in complex uncertain environments. While this method was primarily investigated for legged robot locomotion in the past, it can be more generally applicable to other types of robotic systems. This paper investigates the VMC framework for reaching tasks in a force-controlled robotic arm. We propose six different approaches to designing virtual models in order to achieve reaching tasks in environments with obstacles and uncertainties. A force-controlled 8 degree-of-freedom humanoid robot was used to validate the proposed approach in the real world. We conducted three experiments to test the performance of VMC controllers in terms of predictability, sensitivity to external force, and adaptability against known and unknown obstacles. Experimental analyses show that, even though the proposed approach needs to sacrifice accuracy and trajectory optimality, it enables us to design complex reaching motions under uncertainties, in an intuitive and extendable manner.

Keywords: Continual Learning, Model Learning for Control, Field Robots
Abstract: We represent a vehicle dynamics model for autonomous driving near the limits of handling via a multi-layer neural network. Online adaptation is desirable in order to address unseen environments. However, the model needs to adapt to new environments without forgetting previously encountered ones. In this study, we apply Continual-MAML to overcome this difficulty. It enables the model to adapt to the previously encountered environments quickly and efficiently by starting updates from optimized initial parameters. We evaluate the impact of online model adaptation with respect to inference performance and impact on control performance of a model predictive path integral (MPPI) controller using the TRIKart platform. The neural network was pre-trained using driving data collected in our test environment, and experiments for online adaptation were executed on multiple different road conditions not contained in the training data. Empirical results show that the model using Continual-MAML outperforms the fixed model and the model using gradient descent in test set loss and online tracking performance of MPPI.

Keywords: Sensor Networks, Sensor Fusion, Intelligent Transportation Systems
Abstract: In indoor environments where the Global Navigation Satellite System (GNSS) isn't available, the infrastructure-based LiDAR-camera joint array can provide high-precision localization for mobile robots, such as Autonomous Valet Parking(AVP). The primary challenge in employing the infrastructure-based LiDAR-camera joint array is the extrinsic calibration between the LiDAR and the camera. Moreover, to handle interference deviation caused by vibrations or inadequate mounting stiffness during operation, the calibration’s extrinsic parameters must be automatically updated online, presenting higher demands for infrastructure-based LiDAR-camera extrinsic calibration. This paper proposes an infrastructure LiDAR-camera online automatic calibration method based on prior knowledge of cross-modal target registration. This method requires no manual targets and initial pose guesses and can achieve extrinsic calibration. The object-prior model based on a lightweight object detection algorithm can rapidly detect scenes favorable for extrinsic calibration in sub-images of camera images. This creates favorable conditions for the registration of cross-modal networks and poses optimization of the LiDAR camera. Additionally, because a lightweight algorithm is used, the process does not compromise efficiency or consume excessive computational resources. Experimental results demonstrate that the proposed calibration method is suitable for calibrating infrastructure-based LiDAR-camera, with comparable accuracy and the ability to perform online calibration. Comparative experiments also show that the object prior model can indeed select better scenes for LiDAR-camera extrinsic calibration, thus improving the accuracy and stability of extrinsic calibration to some extent.

Keywords: Calibration and Identification, Computer Vision for Automation, Recognition
Abstract: Hand-eye calibration plays a fundamental role in robotics by directly influencing the efficiency of critical operations such as manipulation and grasping. In this work, we present a novel framework, EasyHeC++, designed for fully automatic hand-eye calibration. In contrast to previous methods that necessitate manual calibration, specialized markers, or the training of arm-specific neural networks, our approach is the first system that enables accurate calibration of any robot arm in a marker-free, training-free, and fully automatic manner. Our approach employs a two-step process. First, we initialize the camera pose using a sampling or feature-matching-based method with the aid of pretrained image models. Subsequently, we perform pose optimization through differentiable rendering. Extensive experiments demonstrate the system's superior accuracy in both synthetic and real-world datasets across various robot arms and camera settings. Project page: https://ootts.github.io/easyhec_plus.

Keywords: Calibration and Identification, Sensor Networks, Visual-Inertial SLAM
Abstract: In this paper, we address the problem of estimating the rotational extrinsics, as well as the scale factors of two gyroscopes rigidly mounted on the same device. In particular, we formulate the problem as a least-squares minimization and introduce a direct algorithm that computes the estimated quantities without any iterations, hence avoiding local minima and improving efficiency. Furthermore, we show that the rotational extrinsics are always observable while the scale factors can be determined up to global scale for general configurations of the gyroscopes. To this end, we also study special placements of the gyroscopes where a pair, or all, of their axes are parallel and analyze their impact on the scale factors’ observability. Lastly, we evaluate our algorithm in simulations and real-world experiments to assess its performance as a function of key motion and sensor characteristics.

Keywords: Calibration and Identification, Sensor Fusion, Kinematics
Abstract: Visual sensor modalities dominate traditional robot calibration, but when environment contacts are relevant, the tactile modality can provide another natural, accurate, and highly relevant modality. Most existing tactile sensing methods for robot calibration are constrained to specific sensor-object pairs, limiting their applicability. This paper pioneers a general approach to exploit contacts in robot calibration, supporting self-touch throughout the entire system kinematics by generalizing touchable surfaces to any accurately represented mesh surface. The approach supports different contact sensors as long as a simple single-contact interface can be provided. Integrated into the Atomic Transformation Optimization Method (ATOM) calibration methodology, our work facilitates seamless integration of both modalities in a single approach. Our results demonstrate comparable performance to single-modality calibration but can trade off accuracy between both modalities, thus increasing overall robustness. Furthermore, we observe that utilizing a touch point at the end of a kinematic chain slightly improves calibration over touching the chain links with an external sensor but find no significant advantage of restricting touch to end-effector contacts when calibrating a dual-arm system with our method.

Keywords: Calibration and Identification, Sensor Fusion
Abstract: Mobile robots equipped with multiple light detection and ranging (LiDARs) and capable of recognizing their surroundings are increasing due to the minitualization and cost reduction of LiDAR. This paper proposes a target-less extrinsic calibration method of multiple LiDARs with non-overlapping field of view (FoV). The proposed method uses accumulated point clouds of floor plane and objects while in motion. It enables accurate calibration with challenging configuration of LiDARs that directed towards the floor plane, caused by biased feature values. Additionally, the method includes a noise removal module that considers the scanning pattern to address bleeding points, which are noises of significant source of error in point cloud alignment using high-density LiDARs. Evaluations through simulation demonstrate that the proposed method achieved higher accuracy extrinsic calibration with two and four LiDARs than conventional methods, regardless type of objects. Furthermore, the experiments using a real mobile robot has shown that our proposed noise removal module can eliminate noise more precisely than conventional methods, and the estimated extrinsic parameters have successfully created consistent 3D maps.

Keywords: Calibration and Identification, SLAM, Wheeled Robots
Abstract: This paper proposes a method for calibrating the external parameters of the LDS sensor and ToF depth camera based on three cylinders. This method obtains the scanning data of the side surfaces of the three cylinders at different postures by changing the posture of the robot. For the single-line laser plane scanned by the LDS sensor, three elliptical contours are obtained by intersecting with the side surfaces of the three cylinders respectively. The Random Sample Consensus (RANSAC) algorithm is used to obtain the coordinates of the center points of the three elliptical contours and two random points on each elliptical contour. For the three-dimensional point cloud image of the cylinder scanned by the ToF depth camera, the RANSAC algorithm is used to fit the central axis of the three cylinders. The nonlinear optimization equation is established using the three center points obtained from the three elliptical contours and the distances from the two random points on each elliptical contour to their corresponding central axes. In this paper, we propose to use a fusion method of the Powell algorithm and the BFGS algorithm to solve the nonlinear optimization equations to obtain the transformation matrix between the LDS sensor and the ToF depth camera. Finally, simulation and actual test are carried out based on the proposed method, and the influence of the initial value of the calibration parameter on the calibration result is discussed. The accuracy of the calibration algorithm in this paper is verified through comparative experiments of the calibration algorithm. The results show that the calibration accuracy of the proposed method is better than that of the traditional planar calibration method, and it has the characteristics of simple operation and high calibration accuracy.

Keywords: Calibration and Identification, Sensor Fusion, Space Robotics and Automation
Abstract: This paper addresses the problem of choosing a sparse subset of measurements for quick calibration parameter estimation. A standard solution to this is selecting a measurement only if its utility---the difference between posterior (with the measurement) and prior information (without the measurement)---exceeds some threshold. Theoretically, utility, a function of the parameter estimate, should be evaluated at the estimate obtained with all measurements selected so far, hence necessitating a recalibration with each new measurement. However, we hypothesize that utility is insensitive to changes in the parameter estimate for many systems of interest, suggesting that evaluating utility at some initial parameter guess would yield equivalent results in practice. We provide evidence supporting this hypothesis for extrinsic calibration of multiple inertial measurement units (IMUs), showing the reduction in calibration time by two orders of magnitude by forgoing recalibration for each measurement.

Keywords: Calibration and Identification, SLAM, Sensor Fusion
Abstract: This paper presents MFCalib, an innovative extrinsic calibration technique for LiDAR and RGB camera that operates automatically in targetless environments with a single data capture. At the heart of this method is using a rich set of edge information, significantly enhancing calibration accuracy and robustness. Specifically, we extract both depth-continuous and depth-discontinuous edges, along with intensity-discontinuous edges on planes. This comprehensive edge extraction strategy ensures our ability to achieve accurate calibration with just one round of data collection, even in complex and varied settings. Addressing the uncertainty of depth-discontinuous edges, we delve into the physical measurement principles of LiDAR and develop a beam model, effectively mitigating the issue of edge inflation caused by the LiDAR beam. Extensive experiment results demonstrate that MFCalib outperforms the state-of-the-art targetless calibration methods across various scenes, achieving and often surpassing the precision of multi-scene calibrations in a single-shot collection. To support community development, we make our code available open-source on GitHub.

Keywords: Calibration and Identification, Parallel Robots, Kinematics
Abstract: The efficient operation of large-scale Cable-Driven Parallel Robots (CDPRs) relies on precise calibration of kinematic parameters and the simplicity of the calibration process. This paper presents a graph-based self-calibration framework that explicitly addresses cable sag effects and facilitates the calibration procedure for large-scale CDPRs by only relying on internal sensors. A unified factor graph is proposed, incorporating a catenary cable model to capture cable sagging. The factor graph iteratively refines kinematic parameters, including anchor point locations and initial cable length, by considering jointly onboard sensor data and the robot's kineto-static model. The applicability and accuracy of the proposed technique are demonstrated through Finite Element (FE) simulations, on both large and small-scale CDPRs subjected to significant initialization perturbations.

Keywords: Calibration and Identification, Imitation Learning, Perception-Action Coupling
Abstract: Large-scale robotic policies trained on data from diverse tasks and robotic platforms hold great promise for enabling general-purpose robots; however, reliable generalization to new environment conditions remains a major challenge. To address this challenge, we propose an approach to achieve this in pre-trained language-conditioned imitation learning agents. Specifically, we use temperature scaling to calibrate these models and exploit the calibrated model to make uncertainty-aware decisions, by aggregating the local information of candidate actions. We implement our approach in simulation using three such pre-trained models, and showcase its potential to significantly enhance task completion rates. The accompanying code is accessible at the link: href{https://github.com/BobWu1998/uncertainty_quant_all.git}{https://github.com/BobWu1998/uncertainty_quant_all.git }

Keywords: Calibration and Identification, Wheeled Robots, Sensor Networks
Abstract: This paper presents MEMROC (Multi-Eye to Mobile RObot Calibration), a novel motion-based calibration method that simplifies the process of accurately calibrating multiple cameras relative to a mobile robot’s reference frame. MEMROC utilizes a known calibration pattern to facilitate accurate calibration with a lower number of images during the optimization process. Additionally, it leverages robust ground plane detection for comprehensive 6-DoF extrinsic calibration, overcoming a critical limitation of many existing methods that struggle to estimate the complete camera pose. The proposed method addresses the need for frequent recalibration in dynamic environments, where cameras may shift slightly or alter their positions due to daily usage, operational adjustments, or vibrations from mobile robot movements. MEMROC exhibits remarkable robustness to noisy odometry data, requiring minimal calibration input data. This combination makes it highly suitable for daily operations involving mobile robots. A comprehensive set of experiments on both synthetic and real data proves MEMROC’s efficiency, surpassing existing state- of-the-art methods in terms of accuracy, robustness, and ease of use. To facilitate further research, we have made our code publicly available: https://github.com/davidea97/MEMROC.git

Keywords: Calibration and Identification, Intelligent Transportation Systems, Localization
Abstract: Cooperative LiDAR systems integrating vehicles and road infrastructure, termed V2I calibration, exhibit substantial potential, yet their deployment encounters numerous challenges. A pivotal aspect of ensuring data accuracy and consistency across such systems involves the calibration of LiDAR units across heterogeneous vehicular and infrastructural endpoints. This necessitates the development of calibration methods that are both real-time and robust, particularly those that can ensure robust performance in urban canyon scenarios without relying on initial positioning values. Accordingly, this paper introduces a novel approach to V2I calibration, leveraging spatial association information among perceived objects. Central to this method is the innovative Overall Intersection over Union (oIoU) metric, which quantifies the correlation between targets identified by vehicle and infrastructure systems, thereby facilitating the real-time monitoring of calibration results. Our approach involves identifying common targets within the perception results of vehicle and infrastructure LiDAR systems through the construction of an affinity matrix. These common targets then form the basis for the calculation and optimization of extrinsic parameters. Comparative and ablation studies conducted using the DAIR-V2X dataset substantiate the superiority of our approach. For further insights and resources, our project repository is accessible at https://github.com/MassimoQu/v2i-calib.

Keywords: Calibration and Identification, Kinematics
Abstract: Industrial robots and collaborative robots are widely employed in industry and are progressively being utilized to assist individuals in their daily routines. To improve their absolute accuracy, self-calibration methods using portable local measurement devices are cost-effective solutions. However, compared with the conventional external calibration methods, self-calibration methods employing two configurations as a calibration sample introduce more non-kinematic errors to the robot. Therefore, noise reduction is significantly necessary in self-calibration. A novel Piecewise-weighted Random Sample Consensus (RANSAC) method is proposed in this paper. Instead of choosing an optimal model with all inliers, the proposed method employs a general weight considering both the sample and hypothesis model qualities to generate a new model with Weighted Least Square (WLS) method. Besides, the proposed method turns the target of finding an uncontaminated set of inliers into the training of the proper weight coefficient for WLS, which not only improves the accuracy but also greatly enhances the speed. The self-calibration experiment on a 6 degree-of-freedom(DOF) robot CR10 shows that the accuracy of the proposed Piecewise-weighted RANSAC method makes a 27.7% accuracy improvement from that employing Least Square method, a 20.0% accuracy improvement from that employing standard RANSAC method, and a 5.5% accuracy improvement from that employing LO-RANSAC method. Besides, the proposed method is also over 10.9 times faster than the standard RANSAC method and 18.6 times faster than the LO-RANSAC method.

Keywords: Calibration and Identification, Surgical Robotics: Laparoscopy, Model Learning for Control
Abstract: Haptic feedback to the surgeon during robotic surgery would enable safer and more immersive surgeries but estimating tissue interaction forces at the tips of robotically controlled surgical instruments has proven challenging. Few existing surgical robots can measure interaction forces directly and the additional sensor may limit the life of instruments. We present a hybrid model and learning-based framework for force estimation for the Patient Side Manipulators (PSM) of a da Vinci Research Kit (dVRK). The model-based component identifies the dynamic parameters of the robot and estimates free-space joint torque, while the learning-based component compensates for environmental factors, such as the additional torque caused by trocar interaction between the PSM instrument and the patient's body wall. We evaluate our method in an abdominal phantom and achieve an error in force estimation of under 10% normalized root-mean-squared error. We show that by using a model-based method to perform dynamics identification, we reduce reliance on the training data covering the entire workspace. Although originally developed for the dVRK, the proposed method is a generalizable framework for other compliant surgical robots. The code is available at https://github.com/vu-maple-lab/dvrk_force_estimation.

Keywords: Robot Audition, Calibration and Identification
Abstract: Asynchronous microphone array calibration is a prerequisite for many audition robot applications. A popular solution to the above calibration problem is the batch form of Simultaneous Localisation and Mapping (SLAM), using the time difference of arrival measurements between two microphones (TDOA-M), and the robot (which serves as a moving sound source during calibration) odometry information. In this paper, we introduce a new form of measurement for microphone array calibration, i.e. the time difference of arrival between adjacent sound events (TDOA-S) with respect to the microphone channels. We propose to use TDOA-S and TDOA-M, called hybrid TDOA, together with odometry measurements for bath SLAM-based calibration of asynchronous microphone arrays. Extensive simulation and real-world experiments show that our method is more independent of microphone number, less sensitive to initial values (when using off-the-shelf algorithms such as Gauss-Newton iterations), and has better calibration accuracy and robustness under various TDOA noises. Simulation results also demonstrate that our method has a lower Cramér-Rao lower bound (CRLB) for microphone parameters. To benefit the community, we open-source our code and data at https://github.com/AISLAB-sustech/Hybrid-TDOA-Calib.

Keywords: Autonomous Vehicle Navigation, Intelligent Transportation Systems
Abstract: Recent advancements in deep learning have significantly enhanced the development of efficient models for multi-modal path prediction within urban environments, offering approaches to navigate complex environments accurately. Despite their performance, models grounded in deep learning techniques frequently encounter challenges related to interpretability. This limitation not only hampers their practical application but also complicates the process of diagnosing and rectifying errors within these systems, which is a critical factor for ensuring reliability and safety in real-world deployments. In this paper we propose NeSyMoF, a Neuro-Symbolic model for Motion Forecasting, to address this critical gap by combining the predictive power of deep neural networks with the interpretable logic inherent in symbolic reasoning. Data processing in NeSyMoF involves extracting pertinent features from the agent's environment and channeling them into a neuro-symbolic reasoning module. The neuro-symbolic reasoning module generates first-order logic rules that describe and condition the path prediction process, thereby providing clear explanations and intentions behind the forecasts of the model. We evaluate our model with the Argoverse benchmark for path forecasting, as it includes challenging driving situations, necessary to extensively evaluate our model. The results of our evaluation show that NeSyMoF outperforms state-of-the-art interpretable models for single-mode predictions while providing logic-based explanations for its forecasts, that articulate the reasoning behind predictions, making NeSyMoF more adapted for human-centric applications.

Keywords: Autonomous Vehicle Navigation, Intelligent Transportation Systems, Big Data in Robotics and Automation
Abstract: Multiple approaches have already been proposed to mimic real driver behaviors in simulation. This article proposes a new one, based solely on the exploration of undisturbed observation of intersections. From them, the behavior profiles for each macro-maneuver will be discovered. Using the macro-maneuvers already identified in previous works, a comparison method between trajectories with different lengths using an Extended Kalman Filter (EKF) is proposed, which combined with an Expectation-Maximization (EM) inspired method, defines the different clusters that represent the behaviors observed. This is also paired with a Kullback-Liebler divergent (KL) criteria to define when the clusters need to be split or merged. Finally, the behaviors for each macro-maneuver are determined by each cluster discovered, without using any map information about the environment and being dynamically consistent with vehicle motion. By observation it becomes clear that the two main factor for driver behavior are their assertiveness and interaction with other road users.

Keywords: Intelligent Transportation Systems, Collision Avoidance, Agent-Based Systems
Abstract: Dynamic scheduling of access to shared resources by autonomous systems is a challenging problem, characterized as being NP-hard. The complexity of this task leads to a combinatorial explosion of possibilities in highly dynamic systems where arriving requests must be continuously scheduled subject to strong safety and time constraints. An example of such a system is an unsignalized intersection, where automated vehicles' access to potential conflict zones must be dynamically scheduled. In this paper, we apply Neural Monte Carlo Tree Search (NMCTS) to the challenging task of scheduling platoons of vehicles crossing unsignalized intersections. Crucially, we introduce a transformation model that maps successive sequences of potentially conflicting road-space reservation requests from platoons of vehicles into a series of board-game-like problems and use NMCTS to search for solutions representing optimal road-space allocation schedules in the context of past allocations. To optimize search, we incorporate a prioritized re-sampling method with parallel NMCTS (PNMCTS) to improve the quality of training data. To optimize training, a curriculum learning strategy is used to train the agent to schedule progressively more complex boards culminating in overlapping boards that represent busy intersections. In a single four-way unsignalized intersection simulation, PNMCTS managed 95% of new high-density scenarios, reducing crossing time by 43% in light and 52% in heavy traffic versus first-in first-out control. In a 3x3 multi-intersection network, the proposed method maintained free-flow in light traffic when all intersection are under control of PNMCTS and outperformed state-of-the-art RL-based traffic-light controls in average travel time by 74.5% and total throughput by 16% in heavy traffic.

Keywords: Intelligent Transportation Systems, Agent-Based Systems, Autonomous Vehicle Navigation
Abstract: In trajectory forecasting tasks for traffic, future output trajectories can be computed by advancing the ego vehicle's state with predicted actions according to a kinematics model. By unrolling predicted trajectories via time integration and models of kinematic dynamics, predicted trajectories should not only be kinematically feasible but also relate uncertainty from one timestep to the next. While current works in probabilistic prediction do incorporate kinematic priors for mean trajectory prediction, _variance_ is often left as a learnable parameter, despite uncertainty in one time step being inextricably tied to uncertainty in the previous time step. In this paper, we show simple and differentiable analytical approximations describing the relationship between variance at one timestep and that at the next with the kinematic bicycle model. In our results, we find that encoding the relationship between variance across timesteps works especially well in unoptimal settings, such as with small or noisy datasets. We observe up to a 50% performance boost in partial dataset settings and up to an 8% performance boost in large-scale learning compared to previous kinematic prediction methods on SOTA trajectory forecasting architectures out-of-the-box, with no fine-tuning.

Keywords: Intelligent Transportation Systems, Simulation and Animation, Computer Vision for Transportation
Abstract: Data augmentation methods to enhance perception performance in adverse weather have recently attracted considerable attention. Most of the LiDAR data augmentation methods post-process the existing dataset by physics-based models or machine-learning methods. However, due to the limited environmental annotations and the fixed vehicle trajectories in existing datasets, it is challenging to edit the scene and expand the diversity of traffic flow and scenario. To this end, we propose a simulator-based physical modeling approach to augment LiDAR data in rainy weather, enhancing the performance of the perception model. We complete the modeling task of the rainy weather effect in the CARLA simulator and establish a data collection pipeline for LiDAR. Furthermore, we pay special attention to the spray generated by vehicles in rainy weather and simulate this phenomenon through the Spray Emitter method we developed. In addition, considering the influence of different weather conditions on point cloud intensity, we develop a prediction network to forecast the intensity of the LiDAR echo. This enables us to complete the rainy weather simulation of 4D point cloud data. In the experiment, we observe that the model augmented by our synthetic dataset improves the performance for 3D object detection in rainy weather. Both code and dataset are available at https://github.com/PJLab-ADG/PCSim#rainypcsim.

Keywords: Intelligent Transportation Systems, Path Planning for Multiple Mobile Robots or Agents
Abstract: In the evolving landscape of urban mobility, the prospective integration of Connected and Automated Vehicles (CAVs) with Human-Driven Vehicles (HDVs) presents a com- plex array of challenges and opportunities for autonomous driving systems. While recent advancements in robotics have yielded Multi-Agent Path Finding (MAPF) algorithms tai- lored for agent coordination task characterized by simplified kinematics and complete control over agent behaviors, these solutions are inapplicable in mixed-traffic environments where uncontrollable HDVs must coexist and interact with CAVs. Addressing this gap, we propose the Behavior Prediction Kinematic Priority Based Search (BK-PBS), which leverages an offline-trained conditional prediction model to forecast HDV responses to CAV maneuvers, integrating these insights into a Priority Based Search (PBS) where the A* search proceeds over motion primitives to accommodate kinematic constraints. We compare BK-PBS with CAV planning algorithms derived by rule-based car-following models, and reinforcement learning (RL). Through comprehensive simulation on a highway merging scenario across diverse scenarios of CAV penetration rate and traffic density, BK-PBS outperforms these baselines in reducing collision rates and enhancing system-level travel delay. Our work is directly applicable to many scenarios of multi-human multi-robot coordination.

Keywords: Intelligent Transportation Systems, Embodied Cognitive Science, Agent-Based Systems
Abstract: Leveraging advanced reasoning capabilities and extensive world knowledge of large language models (LLMs) to construct generative agents for solving complex real-world problems is a major trend. However, LLMs inherently lack embodiment as humans, resulting in suboptimal performance in many embodied decision-making tasks. In this paper, we introduce a framework for building human-like generative driving agents using post-driving self-report driving-thinking data from human drivers as both demonstration and feedback. To capture high-quality, natural language data from drivers, we conducted urban driving experiments, recording drivers' verbalized thoughts under various conditions to serve as chain-of-thought prompts and demonstration examples for the LLM-Agent. The framework's effectiveness was evaluated through simulations and human assessments. Results indicate that incorporating expert demonstration data significantly reduced collision rates by 81.04% and increased human likeness by 50% compared to a baseline LLM-based agent. Our study provides insights into using natural language-based human demonstration data for embodied tasks. The driving-thinking dataset is available at url{https://github.com/AIR-DISCOVER/Driving-Thinking-Datas et}.

Keywords: Intelligent Transportation Systems, Deep Learning Methods, Dynamics
Abstract: Velocity estimation is of great importance in autonomous racing. Still, existing solutions are characterized by limited accuracy, especially in the case of aggressive driving or poor generalization to unseen road conditions. To address these issues, we propose to utilize Unscented Kalman Filter (UKF) with a learned dynamics model that is optimized directly for the state estimation task. Moreover, we propose to aid this model with the online-estimated friction coefficient, which increases the estimation accuracy and enables zero-shot adaptation to the new road conditions. To evaluate the UKF-based velocity estimator with the proposed dynamics model, we introduced a publicly available dataset of aggressive maneuvers performed by an F1TENTH car, with sideslip angles reaching 40°. Using this dataset, we show that learning the dynamics model through UKF leads to improved estimation performance and that the proposed solution outperforms state-of-the-art learning-based state estimators by 17% in the nominal scenario. Moreover, we present unseen zero-shot adaptation abilities of the proposed method to the new road surface thanks to the proposed learning-based tire dynamics model with online friction estimation.

Keywords: Intelligent Transportation Systems, Computer Vision for Transportation, AI-Based Methods
Abstract: Motion prediction is among the most fundamental tasks in autonomous driving. Traditional methods of motion forecasting primarily encode vector information of maps and historical trajectory data of traffic participants, lacking a comprehensive understanding of overall traffic semantics, which in turn affects the performance of prediction tasks. In this paper, we utilized Large Language Models (LLMs) to enhance the global traffic context understanding for motion prediction tasks. We first conducted systematic prompt engineering, visualizing complex traffic environments and historical trajectory information of traffic participants into image prompts---Transportation Context Map (TC-Map), accompanied by corresponding text prompts. Through this approach, we obtained rich traffic context information from the LLM. By integrating this information into the motion prediction model, we demonstrate that such context can enhance the accuracy of motion predictions. Furthermore, considering the cost associated with LLMs, we propose a cost-effective deployment strategy: enhancing the accuracy of motion prediction tasks at scale with 0.7% LLM-augmented datasets. Our research offers valuable insights into enhancing the understanding of traffic scenes of LLMs and the motion prediction performance of autonomous driving.

Keywords: Intelligent Transportation Systems, Telerobotics and Teleoperation, Human Factors and Human-in-the-Loop
Abstract: Autonomous driving is a highly anticipated approach toward eliminating roadway fatalities. At the same time, the bar for safety is both high and costly to verify. This work considers the role of remotely-located human operators supervising a fleet of autonomous vehicles (AVs) for safety. Such a 'scalable supervision' concept was previously proposed to bridge the gap between still-maturing autonomy technology and the pressure to begin commercial offerings of autonomous driving. The present article proposes DISCES, a framework for data-informed safety-critical event simulation, to investigate the practicality of this concept from a dynamic network loading standpoint. With a focus on the safety-critical context of AVs merging into mixed-autonomy traffic, vehicular arrival processes at 1,097 highway merge points are modeled using microscopic traffic reconstruction with historical data from interstates across three California counties. Combined with a queuing theoretic model, these results characterize the dynamic supervision requirements and thereby scalability of the teleoperation approach. Across all scenarios we find reductions in operator requirements greater than 99% as compared to in-vehicle supervisors for the time period analyzed. The work also demonstrates two methods for reducing these empirical supervision requirements: (i) the use of cooperative connected AVs --- which are shown to produce an average 3.67 orders-of-magnitude system reliability improvement across the scenarios studied --- and (ii) aggregation across larger regions.

Keywords: Intelligent Transportation Systems, Logistics, Reinforcement Learning
Abstract: The Vehicle Routing Problem with Drones (VRPD) seeks to optimize the routing paths for both trucks and drones, where the trucks are responsible for delivering parcels to customer locations, and the drones are dispatched from these trucks for parcel delivery, subsequently being retrieved by the trucks. Given the NP-Hard complexity of VRPD, numerous heuristic approaches have been introduced. However, improving solution quality and reducing computation time remain significant challenges. In this paper, we conduct a comprehensive examination of heuristic methods designed for solving VRPD, distilling and standardizing them into core elements. We then develop a novel reinforcement learning (RL) framework that is seamlessly integrated with the heuristic solution components, establishing a set of universal principles for incorporating the RL framework with heuristic strategies in an aim to improve both the solution quality and computational speed. This integration has been applied to a state-of-the-art heuristic solution for VRPD, showcasing the substantial benefits of incorporating the RL framework. Our evaluation results demonstrated that the heuristic solution incorporated with our RL framework not only elevated the quality of solutions but also achieved rapid computation speeds, especially when dealing with extensive customer locations.

Keywords: Intelligent Transportation Systems, Reinforcement Learning, Deep Learning Methods
Abstract: Active traffic management with autonomous vehicles offers the potential for reduced congestion and improved traffic flow. However, developing effective algorithms for real-world scenarios requires overcoming challenges related to infinite-horizon traffic flow and partial observability. To address these issues and further decentralize traffic management, we propose an asymmetric actor-critic model that learns decentralized cooperative driving policies for autonomous vehicles using single-agent reinforcement learning. By employing attention neural networks with masking, our approach efficiently manages real-world traffic dynamics and partial observability, eliminating the need for predefined agents or agent-specific experience buffers in multi-agent reinforcement learning. Extensive evaluations across various traffic scenarios demonstrate our method's significant potential in improving traffic flow at critical bottleneck points. Moreover, we address the challenges posed by conservative autonomous vehicle driving behaviors that adhere strictly to traffic rules, showing that our cooperative policy effectively alleviates potential slowdowns without compromising safety.

Keywords: Intelligent Transportation Systems, Multi-Robot Systems, Reinforcement Learning
Abstract: Efficient and scalable adaptive traffic signal control is crucial in reducing congestion, maximizing throughput, and improving mobility experience in ever-expanding cities. Recent advances in multi-agent reinforcement learning (MARL) with parameter sharing have significantly improved the adaptive optimization of large-scale, complex, and dynamic traffic flows. However, the limited model representation capability due to shared parameters impedes the learning of diverse control strategies for intersections with different flows/topologies, posing significant challenges to achieving effective signal control in complex and varied real-world traffic scenarios. To address these challenges, we present a novel MARL-based general traffic signal control framework, called HeteroLight. Specifically, we first introduce a General Feature Extraction (GFE) module, crafted in a decoder-only fashion, where we employ an attention mechanism to facilitate efficient and flexible extraction of traffic dynamics at intersections with varied topologies. Additionally, we incorporate an Intersection Specifics Extraction (ISE) module, designed to identify key latent vectors that represent the unique intersection's topology and traffic dynamics through variational inference techniques. By integrating the learned intersection-specific information into policy learning, we enhance the parameter-sharing mechanism, improving the model's representation diversity among different agents and enabling the learning of a more efficient shared control strategy. Through comprehensive evaluations against other state-of-the-art traffic signal control methods on the real-world Monaco traffic network, our empirical findings reveal that HeteroLight consistently outperforms other methods across various evaluation metrics, highlighting its superiority in optimizing traffic flows in heterogeneous traffic networks.

Keywords: Intelligent Transportation Systems, Multi-Robot Systems
Abstract: Autonomous cooperative planning (ACP) is a promising technique to improve the efficiency and safety of multi-vehicle interactions for future intelligent transportation systems. However, realizing robust ACP is a challenge due to the aggregation of perception, motion, and communication uncertainties. This paper proposes a novel multi-uncertainty aware ACP (MUACP) framework that simultaneously accounts for multiple types of uncertainties via regularized cooperative model predictive control (RC-MPC). The regularizers and constraints for perception, motion, and communication are constructed according to the confidence levels, weather conditions, and outage probabilities, respectively. The effectiveness of the proposed method is evaluated in the Car Learning to Act (CARLA) simulation platform. Results demonstrate that the proposed MUACP efficiently performs cooperative formation in real time and outperforms other benchmark approaches in various scenarios under imperfect knowledge of the environment.

Keywords: Intelligent Transportation Systems, Deep Learning Methods, Robot Safety
Abstract: Accurate trajectory prediction is crucial for autonomous vehicles to realize safe driving. Current trajectory prediction approaches generally rely on deep neural networks, which are susceptible to adversarial attacks. To evaluate the adversarial robustness and security of deep-learning-based trajectory prediction models, this paper proposes an adversarial attack method on trajectory prediction using generative adversarial networks (GANs). First, a novel LSTM-based attack trajectory model named Adv-GAN is proposed considering both the temporal and spatial driving features. The networks in Adv-GAN are trained through game learning between the generator and the discriminator to obtain the adversarial trajectories with real driving feature distribution. Furthermore, the generated trajectory is optimized with the vehicle kinematics model for driving feasibility on roads. The derived adversarial attack can lead to considerable deviations in trajectory prediction which affects driving safety for autonomous vehicles. We evaluate the proposed Adv-GAN on three public datasets, and experimental results show the effectiveness with better attack performance compared to a state-of-the-art adversarial attack model.

Keywords: Intelligent Transportation Systems, Localization, Field Robots
Abstract: The installation of a multi-layer 3D LiDAR atop the vehicle is a widely adopted hardware configuration for mapmatching-based localization in intelligent driving. By offering a comprehensive 360◦ horizontal Field of View (FoV), this setup aims to achieve precise matching outcomes through the imposition of substantial geometric constraints against dynamic interferences and structural degradation. However, several factors limit its environmental adaptability, such as sparse point cloud density at distances, insufficient maximum sensing range, and notably, the restricted beam elevation angle, limiting the perception of the environment beyond obstacles. The rapid advancement of non-repetitive scanning LiDARs shows promise in mitigating such limitations. Nevertheless, their narrow FoV remains a challenge to overcome. In this study, we propose a solution by mounting such one single LiDAR on a two-axis rotating gimbal, enabling the vehicle to surpass the ranges and vertical FoV limitations of traditional setups actively. The corresponding gimbal motion strategy has been designed to automatically focus on the environment component with the most robust geometric constraints. Experimental results validate that the proposed method achieves superior robustness under high dynamic interference while delivering sufficient performance under standard conditions

Keywords: Mapping, Marine Robotics, SLAM
Abstract: In recent years, the interest in underwater exploration with Autonomous Underwater Vehicles (AUVs) equipped with side-scan sonars (SSS) has grown considerably. However, state-of-the-art SSS Simultaneous Localization and Mapping (SLAM) systems encounter challenges in data association across large viewpoint changes. Additionally, these systems assume that the seabed is a flat surface, leading to significant mapping error in uneven underwater terrains. To address these challenges, we propose a framework that leverages the side-scan sonar geometry to facilitate data association and improve mapping accuracy. The framework begins with a preprocessing module that extracts feature points and provides initial estimates of the elevation angles of the landmarks. Then, a non-consecutive data association module applies epipolar line searching to establish correspondences between the current and historical frames. Finally, the mapping module uses side-scan sonar bundle adjustment to recover the positions of the landmarks. The proposed method is evaluated using an underwater terraced fields dataset. Our method achieves over 90 percent matching rate and reduces the average reprojection error from 764.17 to 4.91 after three iterations.

Keywords: Mapping, Data Sets for Robotic Vision
Abstract: In recent years, Neural Radiance Fields (NeRFs) have demonstrated significant potential in encoding highlydetailed 3D geometry and environmental appearance, positioning themselves as a promising alternative to traditional explicit representation for 3D scene reconstruction. However, the predominant reliance on RGB imaging presupposes ideal lighting conditions—a premise frequently unmet in robotic applications plagued by poor lighting or visual obstructions. This limitation overlooks the capabilities of infrared (IR) cameras, which excel in low-light detection and present a robust alternative under such adverse scenarios. To tackle these issues, we introduce Thermal-NeRF, the first method that estimates a volumetric scene representation in the form of a NeRF solely from IR imaging. By leveraging a thermal mapping and structural thermal constraint derived from the thermal characteristics of IR imaging, our method showcases unparalleled proficiency in recovering NeRFs in visually degraded scenes where RGB-based methods fall short. We conduct extensive experiments to demonstrate that Thermal-NeRF can achieve superior quality compared to existing methods. Furthermore, we contribute a dataset for IR-based NeRF applications, paving the way for future research in IR NeRF reconstruction, see https: //github.com/Cerf-Volant425/Thermal-NeRF.

Keywords: Mapping, Autonomous Vehicle Navigation, Semantic Scene Understanding
Abstract: Highly accurate and robust vectorized reconstruction of road structures is crucial for autonomous vehicles. Traditional LiDAR-based methods require multiple processes and are often expensive, time-consuming, labor-intensive, and cumbersome. In this paper, we propose a lightweight crowdsourced road structure reconstruction system (termed CSR) that relies solely on online perceived semantic elements. Ambiguities and perceptual errors of semantic features and Global Navigation Satellite System (GNSS) global pose errors constitute the predominant challenge in achieving alignment across multi-trip data. To this end, a robust two-phased coarse-to-fine multi-trip alignment method is performed considering local geometric consistency, global topology consistency, intra-trip temporal consistency, and inter-trip consistency. Further, we introduce an incremental pose graph optimization framework with adaptive weight tuning ability to integrate pre-built maps, currently perceived multi-trip semantic features, odometry, and GNSS, enabling accurate and robust incremental road structure reconstruction. CSR is highly automated, efficient, and scalable for large-scale autonomous driving scenarios, significantly expediting road structure production. We quantitatively and qualitatively validate the reconstruction performance of CSR in real-world scenes. CSR achieves centimeter-level accuracy commensurate with established LiDAR-based methods, concurrently boosting efficiency and reducing resource expenditure.

Keywords: Mapping, Multi-Robot SLAM, Computer Vision for Transportation
Abstract: Autonomous vehicles demand detailed maps to maneuver reliably through traffic, which need to be kept up-to-date to ensure a safe operation. A promising way to adapt the maps to the ever-changing road-network is to use crowd-sourced data from a fleet of vehicles. In this work, we present a mapping system that fuses local submaps gathered from a fleet of vehicles at a central instance to produce a coherent map of the road environment including drivable area, lane markings, poles, obstacles and more as a 3D mesh. Each vehicle contributes locally reconstructed submaps as lightweight meshes, making our method applicable to a wide range of reconstruction methods and sensor modalities. Our method jointly aligns and merges the noisy and incomplete local submaps using a scene-specific Neural Signed Distance Field, which is supervised using the submap meshes to predict a fused environment representation. We leverage memory-efficient sparse feature-grids to scale to large areas and introduce a confidence score to model uncertainty in scene reconstruction. Our approach is evaluated on two datasets with different local mapping methods, showing improved pose alignment and reconstruction over existing methods. Additionally, we demonstrate the benefit of multi-session mapping and examine the required amount of data to enable high-fidelity map learning for autonomous vehicles.

Keywords: Mapping, Calibration and Identification, Kinematics
Abstract: We study the motion model of a sensor rigidly mounted inside a ball. Due to the rigid placement inside the ball, the geometry of the sensor trajectory resembles a 3D curate trochoid. A new calibration method for spherical systems estimates the extrinsic parameters of the sensor with respect to the balls center of rotation. We deploy the calibration and motion model on our spherical mobile mapping platform to estimate the trajectory of a LiDAR sensor and compare it to trajectories of state-of-the-art LiDAR-Inertial odometry (LIO) methods. The motion model, which is solely based on IMU measurements, produces comparable results to the LIO methods, sometimes even outperforming them in positional accuracy. Although the LIO methods provide better rotational accuracy due to the utilization of LiDAR data, they struggle to reproduce the trochoidal nature of the trajectory and only provide pose estimations at the LiDAR frequency, whereas the motion model produces a more consistent trochoidal trajectory at the much higher IMU frequency. The results demonstrate the difficulty that current LIO methods have on spherical systems and indicate that our motion model is suitable for overcoming these issues.

Keywords: Mapping, RGB-D Perception, Probability and Statistical Methods
Abstract: The ability to construct concise scene representations from sensor input is central to the field of robotics. This paper addresses the problem of robustly creating a 3D representation of a tabletop scene from a segmented RGB-D image. These representations are then critical for a range of downstream manipulation tasks. Many previous attempts to tackle this problem do not capture accurate uncertainty, which is required to subsequently produce safe motion plans. In this paper, we cast the representation of 3D tabletop scenes as a multi-class classification problem. To tackle this, we introduce V-PRISM, a framework and method for robustly creating probabilistic 3D segmentation maps of tabletop scenes. Our maps contain both occupancy estimates, segmentation information, and principled uncertainty measures. We evaluate the robustness of our method in (1) procedurally generated scenes using open-source object datasets, and (2) real-world tabletop data collected from a depth camera. Our experiments show that our approach outperforms alternative continuous reconstruction approaches that do not explicitly reason about objects in a multi-class formulation.

Keywords: Mapping, Semantic Scene Understanding, Computer Vision for Transportation
Abstract: Autonomous driving for urban and highway driving applications often requires High Definition (HD) maps to generate a navigation plan. Nevertheless, various challenges arise when generating and maintaining HD maps at scale. While recent online mapping methods have started to emerge, their performance especially for longer ranges is limited by heavy occlusion in dynamic environments. With these considerations in mind, our work focuses on leveraging lightweight and scalable priors—Standard Definition (SD) maps—in the development of online vectorized HD map representations. We first examine the integration of prototypical rasterized SD map representations into various online mapping architectures. Furthermore, to identify lightweight strategies, we extend the OpenLane-V2 dataset with OpenStreetMaps and evaluate the benefits of graphical SD map representations. A key finding from designing SD map integration components is that SD map encoders are model agnostic and can be quickly adapted to new architectures that utilize bird's eye view (BEV) encoders. Our results show that making use of SD maps as priors for the online mapping task can significantly speed up convergence and boost the performance of the online centerline perception task by 30% (mAP). Furthermore, we show that the introduction of the SD maps leads to a reduction of the number of parameters in the perception and reasoning task by leveraging SD map graphs while improving the overall performance. Project Page: https://henryzhangzhy.github.io/sdhdmap/.

Keywords: Mapping, Learning from Demonstration, Probabilistic Inference
Abstract: This paper introduces Spatial Diagrammatic Instructions (SDIs), an approach for human operators to specify objectives and constraints that are related to spatial regions in the working environment. Human operators are enabled to sketch out regions on camera images that correspond to the objectives and constraints. These sketches are projected to 3D spatial coordinates, and continuous emph{Spatial Instruction Maps} are learned upon them. These models can then be integrated into optimization problems for tasks of robots. In particular, we demonstrate how SDIs can be applied to solve the Base Placement Problem of mobile manipulators, which concerns the best place to put the manipulator to facilitate a certain task. Human operators can specify, via sketch, spatial regions of interest for a manipulation task and permissible regions for the mobile manipulator to be at. Then, an optimization problem that maximizes the manipulator's reachability, or coverage, over the designated regions of interest while remaining in the permissible regions is solved. We provide extensive empirical evaluations, and show that our formulation of Spatial Instruction Maps provides accurate representations of user-specified Diagrammatic Instructions. Furthermore, we demonstrate that our diagrammatic approach to the Mobile Base Placement Problem enables higher quality solutions and faster runtime.

Keywords: Mapping, Semantic Scene Understanding
Abstract: Maps provide robots with crucial environmental knowledge, thereby enabling them to perform interactive tasks effectively. Easily accessing accurate abstract-to-detailed geometric and semantic concepts from maps is crucial for robots to make informed and efficient decisions. To comprehensively model the environment and effectively manage the map data structure, we propose DHP-Mapping, a dense mapping system that utilizes multiple Truncated Signed Distance Field (TSDF) submaps and panoptic labels to hierarchically model the environment. The output map is able to maintain both voxel- and submap-level metric and semantic information. Two modules are presented to enhance the mapping efficiency and label consistency: (1) an inter-submaps label fusion strategy to eliminate duplicate points across submaps and (2) a conditional random field (CRF) based approach to enhance panoptic labels. We conducted experiments with two public datasets including indoor and outdoor scenarios. Our system performs comparably to state-of-the-art (SOTA) methods across geometry and label accuracy evaluation metrics. The experiment results highlight the effectiveness and scalability of our system, as it is capable of constructing precise geometry and maintaining consistent panoptic labels. Our code is publicly available at https://github.com/hutslib/DHP-Mapping.

Keywords: Mapping, Range Sensing, Deep Learning Methods
Abstract: Millimeter Wave Radar is being adopted as a viable alternative to lidar and radar in adverse visually degraded conditions, such as in the presence of fog and dust. However, this sensor modality suffers from severe sparsity and noise under nominal conditions, which makes it difficult to use in precise applications such as mapping. This work presents a novel solution to generate accurate 3D maps from sparse radar point clouds. RMap uses a generative transformer architecture which upsamples, denoises, and fills the incomplete radar maps to resemble lidar maps. We test this method on the ColoRadar dataset to demonstrate its efficacy.

Keywords: Mapping, Semantic Scene Understanding, Computer Vision for Automation
Abstract: In the absence of a pre-built indoor map, robot navigation suffers from the limitations of sensors and environments, resulting in decreased efficiency in performing ad-hoc tasks. Given that blueprints are difficult to obtain, an intuitive method is to provide robots with prior knowledge via hand-drawn floor plans. However, due to the inability of robots to directly comprehend hand-drawn styles, the applicability of this method is limited. In this paper, we present a novel framework for hand-drawn floor plan reconstruction that can recognize abstract hand-drawn elements and standardize the reconstruction of hand-drawn floor plans, thereby providing robots with valuable global map information. Specifically, we design a new series of structure descriptors as reconstruction components and employ a deep learning-based model for recognition. Then the standardized results are obtained through the proposed floor plan reconstruction algorithm. To verify the effectiveness of the framework, we conduct experiments on electronic and paper hand-drawn floor plans. Compared with other state-of-the-art methods, our proposed method achieves superior reconstruction results. This work expands the application scenarios for indoor robots, enabling them to quickly comprehend the semantics of complex scenes, thereby enhancing the competitiveness in downstream tasks.

Keywords: Mapping, SLAM
Abstract: Accurate and consistent construction of point clouds from LiDAR scanning data is fundamental for 3D modeling applications. Current solutions, such as multiview point cloud registration and LiDAR bundle adjustment, predominantly depend on the local plane assumption, which may be inadequate in complex environments lacking of planar geometries or substantial initial pose errors. To mitigate this problem, this paper presents a LiDAR bundle adjustment with progressive spatial smoothing, which is suitable for complex environments and exhibits improved convergence capabilities. The proposed method consists of a spatial smoothing module and a pose adjustment module, which combines the benefits of local consistency and global accuracy. With the spatial smoothing module, we can obtain robust and rich surface constraints employing smoothing kernels across various scales. Then the pose adjustment module corrects all poses utilizing the novel surface constraints. Ultimately, the proposed method simultaneously achieves fine poses and parametric surfaces that can be directly employed for high-quality point cloud reconstruction. The effectiveness and robustness of our proposed approach have been validated on both simulation and real-world datasets. The experimental results demonstrate that the proposed method outperforms the existing methods and achieves better accuracy in complex environments with low planar structures.

Keywords: Mapping, SLAM
Abstract: The joint optimization of the sensor trajectory and 3D map is a crucial characteristic of bundle adjustment (BA), essential for autonomous driving. This paper presents nu-DBA, a novel framework implementing geometric dense bundle adjustment (DBA) using 3D neural implicit surfaces for map parametrization, which optimizes both the map surface and trajectory poses using geometric error guided by dense optical flow prediction. Additionally, we fine-tune the optical flow model with per-scene self-supervision to further improve the quality of the dense mapping. Our experimental results on multiple driving scene datasets demonstrate that our method achieves superior trajectory optimization and dense reconstruction accuracy. We also investigate the influences of photometric error and different neural geometric priors on the performance of surface reconstruction and novel view synthesis. Our method stands as a significant step towards leveraging neural implicit representations in dense bundle adjustment for more accurate trajectories and detailed environmental mapping.

Keywords: Mapping, SLAM, Marine Robotics
Abstract: In robotics, occupancy grids serve as required repositories of information about the environment in numerous applications. One such critical application is Simultaneous Localization and Mapping (SLAM), where robots dynamically scan and explore their surroundings while in motion. In the context of extended-duration missions, it becomes imperative to confront the complexities linked to the expansion of occupancy grids as well as handling loop closure detection. These challenges primarily revolve around two key aspects: enabling the seamless expansion of the map on multiple occasions, thus avoiding the need to map smaller regions in numerous separate missions, and ensuring real-time updates to the map to sustain the robot’s knowledge base and enhance its responsiveness. To address these challenges, we introduce an innovative map called Frustum Accelerated GPU-Based Grid Map (FRAGG-Map). This map adopts a highly parallelizable 3D grid structure and leverages the power of CUDA kernels to facilitate efficient insertion of point-clouds and enables real-time updates of the map. FRAGG- Map identifies the portions of the map that require updates and utilises the GPU to update them, significantly enhancing computational performance. Our results show that FRAGG- Map can run 31 times faster than OctoMap, significantly outperforming state-of-the-art methods.

Keywords: Mapping, Semantic Scene Understanding, Deep Learning Methods
Abstract: 3D reconstruction has been widely used in autonomous navigation fields of mobile robotics. However, the former research can only provide the basic geometry structure without the capability of open-world scene understanding, limiting advanced tasks like human interaction and visual navigation. Moreover, traditional 3D scene understanding approaches rely on expensive labeled 3D datasets to train a model for a single task with supervision. Thus, geometric reconstruction with zero-shot scene understanding i.e. Open vocabulary 3D Understanding and Reconstruction, is crucial for the future development of mobile robots. In this paper, we propose OpenOcc, a novel framework unifying the 3D scene reconstruction and open vocabulary understanding with neural radiance fields. We model the geometric structure of the scene with occupancy representation and distill the pre-trained open vocabulary model into a 3D language field via volume rendering for zero-shot inference. Furthermore, a novel semantic-aware confidence propagation (SCP) method has been proposed to relieve the issue of language field representation degeneracy caused by inconsistent measurements in distilled features. Experimental results show that our approach achieves competitive performance in 3D scene understanding tasks, especially for small and long-tail objects.

Keywords: Mapping, AI-Based Methods, Human Factors and Human-in-the-Loop
Abstract: In spatial navigation, the shift from manual cartography to digital map representations has revolutionized how we interact with and comprehend both outdoor and indoor environments. While digital mapping has substantially advanced outdoor navigation with robust techniques like satellite imagery and sophisticated data labeling, the full potential of indoor digital mapping remains untapped. Accurate indoor mapping promises to enhance the operational efficiency of mobile robots, improving their ability to interact with human environments, and bolstering emergency response capabilities. However, its realization is impeded by the complexity of current methods and the need for heavy manual labor, expertise knowledge, and specialized equipment. To address these challenges, we introduce Text2Map -- a novel methodology that harnesses natural language navigational instructions, the power of off-the-shelf Large Language Models (LLMs), and Few-shot Learning to create graph-based digital maps of indoor spaces. This approach simplifies the mapping process for widespread use, leveraging crowd-sourceable ubiquitous navigation instructions as a data source without requiring specialized map data formats or hardware. Our paper presents the Text2Map system architecture, details the creation of the first dedicated dataset, and evaluates the system's efficacy, highlighting the substantial potential and scalability of our approach. Text2Map achieves a Graph-Edit-Distance (GED) ranging from 0.5X to 2X the total number of regions in a building and an Edge Similarity score between 0.87 and 0.9. These results highlight the precision, robustness, and effectiveness of our methodology. Our work paves the way for a more accessible and streamlined approach to indoor digital mapping, setting the stage for broader adoption in human and mobile robot navigation applications.