Keywords: SLAM, RGB-D Perception
Abstract: This paper presents a simultaneous localization and mapping (SLAM) system to provide accurate pose estimation and dynamic scene reconstruction. Our approach proposes a Joint Point-Gaussian Splatting representation, which fully integrates the robustness of isotropic feature points in pose estimation and the flexibility of anisotropic 3D Gaussians in scene representation. This system does not need to suppress the anisotropic representation of Gaussian elements, which enables the mapping module to achieve finer scene representation with lower memory consumption. Additionally, in order to enhance the adaptability of the system in dynamic environments, we introduced a dynamic region recognition module and utilized 3D Gaussian Splatting and 4D Gaussian Splatting representations to represent static and dynamic regions respectively. Furthermore, we developed a local map management strategy for Gaussian Splatting mapping, effectively reducing the memory and computational resource usage in the mapping process. Experiments on public datasets demonstrate that our system achieves state-of-the-art tracking and mapping accuracy compared to existing baselines.

Keywords: Sensor Fusion, Localization, SLAM
Abstract: Conventional LiDAR-inertial odometry (LIO) or SLAM methods heavily rely on geometric features of environments, as LiDARs primarily provide range measurements instead of motion measurements. From now on, however, the situation changes thanks to the novel Frequency Modulated Continuous Wave (FMCW) LiDARs. FMCW LiDARs not only offer the point range with high resolution but also capture the instant point Doppler velocity through the Doppler effect. In the letter, we propose FMCW-LIO, a novel and robust LIO, leveraging intrinsic Doppler measurements from FMCW LiDARs. To correctly exploit Doppler velocities, a motion compensation method is designed, and a Doppler-aided observation model is applied for on-manifold state estimation. Then, dynamic points can be effectively removed by the Doppler criteria, deriving more consistent geometric observations. FMCW-LIO eventually achieves accurate state estimation and static mapping, even in structure-degenerated environments. Extensive experiments in diverse scenes are performed and FMCW-LIO outperforms other algorithms on both accuracy and robustness.

Keywords: SLAM, Optimization and Optimal Control, Field Robots
Abstract: Keyframes are LiDAR scans saved for future reference in Simultaneous Localization And Mapping (SLAM), but despite their central importance most algorithms leave choices of which scans to save and how to use them to wasteful heuristics. This work proposes two novel keyframe selection strategies for localization and map summarization, as well as a novel approach to submap generation which selects keyframes that best constrain localization. Our results show that online keyframe identification and submap generation reduce the number of saved keyframes and improve per scan computation time without compromising localization performance. We also present a map summarization feature for quickly capturing environments under strict map size constraints.

Keywords: SLAM, Range Sensing, Mapping
Abstract: Range-only Simultaneous Localisation and Mapping (RO-SLAM) is of interest in the robotics community due to its practical applications; for example, ultra-wideband (UWB) and Bluetooth Low Energy (BLE) localisation in terrestrial and aerial applications and acoustic beacon localisation in marine applications. In this work, we consider a mobile robot equipped with an inertial measurement unit (IMU) and a range sensor that measures distances to a collection of fixed landmarks. We derive an equivariant filter (EqF) for the RO-SLAM problem based on a symmetry Lie group that is compatible with the range measurements. The proposed filter does not require bootstrapping or initialisation of landmark positions, and demonstrates robustness to the no-prior situation. The filter is demonstrated on a real-world dataset, and it is shown to significantly outperform a state-of-the-art EKF alternative in terms of both accuracy and robustness.

Keywords: Optimization and Optimal Control, Localization, Robot Safety, Global Optimality
Abstract: In recent years, semidefinite relaxations of common optimization problems in robotics have attracted growing attention due to their ability to provide globally optimal solutions. In many cases, it was shown that specific handcrafted redundant constraints are required to obtain tight relaxations, and thus global optimality. These constraints are formulation-dependent and typically identified through a lengthy manual process. Instead, the present article suggests an automatic method to find a set of sufficient redundant constraints to obtain tightness, if they exist. We first propose an efficient feasibility check to determine if a given set of variables can lead to a tight formulation. Second, we show how to scale the method to problems of bigger size. At no point of the process do we have to find redundant constraints manually. We showcase the effectiveness of the approach, in simulation and on real datasets, for range-based localization and stereo-based pose estimation. We also reproduce semidefinite relaxations presented in recent literature and show that our automatic method always finds a smaller set of constraints sufficient for tightness than previously considered.

Keywords: SLAM, Autonomous Vehicle Navigation, Mapping
Abstract: Abstract—Lidar-based Simultaneous Localization and Mapping (SLAM) encounters a substantial challenge in the form of accumulating errors, which can adversely impact its reliability. Loop closing techniques have been extensively employed to counteract this issue. Nonetheless, the loop closing conundrum remains difficult to resolve, as point clouds often exhibit partial overlap due to disparities in scanning pose (viewpoints). This renders the conventional point cloud registration such as Iterative Closest Point (ICP) algorithm problematic. To overcome this challenge, this paper proposes a two-stage viewpoint-aware point cloud registration technique that assigns suitable weights to the correspondence pairs associating two point clouds from different viewpoints. The weights account for the visibility of points from their respective viewpoint as well as from the viewpoint of the counterpart point cloud, making the registration more relying on commonly visible points from the both viewpoints. Experimental results, utilizing the KITTI and Apollo-SouthBay dataset, indicate that the proposed technique delivers more precise and robust performance compared to the baseline techniques.

Keywords: Actuation and Joint Mechanisms, Force Control, Tendon/Wire Mechanism
Abstract: This study presents a comprehensive experimental analysis of Twisted String Actuators (TSA), focused on enhancing contraction modelling accuracy and establishing a baseline for TSA tension and impedance control efficacy. A novel TSA string radius function is introduced, computing effective radii for multi-strand bundles based on axial actuator tension. The proposed model was validated in physical experiments, resulting in a reduction of maximal errors between measured and simulated actuator contraction trajectories from up to 60% in established models to around 10% in our work. Additionally, the tension-dependent radius modification effectively reduced errors between the estimated and the measured bundle tension by an order of magnitude, marking an essential step towards TSA control independent of bundle tension measurements. TSA tension control was assessed based on four metrics: accuracy, precision, impact stability, and bandwidth, following ISO 9283:1998 standards. The quality of tension control was found to be dependent on bundle tension, twisting angle and strand quantity, whereas impact stability was maintained in all configurations. Joint impedance control with TSA was evaluated for perturbation stability and position control bandwidth, where the latter was enhanced with increasing joint stiffness. The presented analysis informs designers about the capabilities of TSAs in different configurations, and their respective suitability for desired applications.

Keywords: Tendon/Wire Mechanism, Mechanism Design, Actuation and Joint Mechanisms
Abstract: This paper presents a novel actuator system combining a twisted string actuator (TSA) with a winch mechanism. Relative to traditional hydraulic and pneumatic systems in robotics, TSAs are compact and lightweight but face limitations in stroke length and force-transmission ratios. Our integrated TSA-winch system overcomes these constraints by providing variable transmission ratios through dynamic adjustment. It increases actuator stroke by winching instead of overtwisting, and it improves force output by twisting. The design features a rotating turret that houses a winch, which is mounted on a bevel gear assembly driven by a through-hole drive shaft. Mathematical models are developed for the combined displacement and velocity control of this system. Experimental validation demonstrates the actuator's ability to achieve a wide range of transmission ratios and precise movement control. We present performance data on movement precision and generated forces, discussing the results in the context of existing literature. This research contributes to the development of more versatile and efficient actuation systems for advanced robotic applications and improved automation solutions.

Keywords: Tendon/Wire Mechanism, Actuation and Joint Mechanisms, Medical Robots and Systems
Abstract: Cable transmissions are commonly used in robotics for remote force transmission, offering a lightweight, compact, and efficient solution for transmitting high forces between input and output. However, cables in flexible compression housings (Bowden cables), exhibit high static friction, which increases exponentially with total bend angle. Alternatively, internally routed ball-bearing supported cable capstan transmissions are low friction, but complex and present challenges in routing multiple sets of cables. In this paper, we propose motion-decoupled cable transmission modules that address these challenges, occupying the middle ground, functioning as discrete-joint ball-bearing supported Bowden cables. Our rolling-plus-twist joint design decouples pairs of routed cables from changing significantly in tension, length, or friction during large angle motion of the linked transmission. Using sub-1 mm diameter high-strength synthetic cable, the transmission exhibits a maximum coupling motion of only 0.15 mm over the full range of motion of the cable-transmission mechanism, approximately 10% of pretension in combined hysteresis and friction, a transmission stiffness of 10 N/mm, weighing just 9 g per rolling joint and 5 g per twist joint. Two applications are demonstrated: cable routing alongside a robot arm for, say, gripper remote actuation, and remote needle advancement for an MRI-safe needle biopsy robot.

Keywords: Actuation and Joint Mechanisms
Abstract: This study examines a novel setup of a micropositioning trajectory manipulator in Xθ, energized by a reluctance actuator (RA) and two accompanying moving magnet actuators (MMA). The design is characterized by a C-core RA, which features asymmetrical air gaps between the mover and the stator elements when under angular θ rotation. When the stator coil is energized, a magnetic flux induces a force in the mover. Two MMAs can add force and torque dynamics to the system via solenoid and permanent magnet (PM) pairs to offer additional corrective actions. Facilitating control of a translational x and rotational θ two-degree-of-freedom (2DOF) actuation system. Flexure hinges aid in the retraction force of the mover element and provide needed stiffness to the system without frictional effects. This was modeled analytically and optimized to achieve outlined performance objectives. The system was validated experimentally through triangle, and sinusoidal trajectories in open loop control. The most relevant application is scanning mirror systems where specific targeted rotational and translational trajectories can benefit light beam positioning. This system allows both translation and rotation specifications of a selected trajectory to be realized in one actuation unit, opening up more design possibilities for controlling precision positioning systems.

Keywords: Machine Learning for Robot Control, Actuation and Joint Mechanisms, Legged Robots
Abstract: This paper presents a novel approach through the design and implementation of Cycloidal Quasi-Direct Drive actuators for legged robotics. The cycloidal gear mechanism, with its inherent high torque density and mechanical robustness, offers significant advantages over conventional designs. By integrating cycloidal gears into the Quasi-Direct Drive framework, we aim to enhance the performance of legged robots, particularly in tasks demanding high torque and dynamic loads, while still keeping them lightweight. Additionally, we develop a torque estimation framework for the actuator using an Actuator Network, which effectively reduces the sim-to-real gap introduced by the cycloidal drive’s complex dynamics. This integration is crucial for capturing the complex dynamics of a cycloidal drive, which contributes to improved learning efficiency, agility, and adaptability for reinforcement learning.

Keywords: Mechanism Design, Compliant Joint/Mechanism, Robotic Wrist, Grasping
Abstract: We have developed a two-degree-of-freedom robotic wrist with variable stiffness capability, designed for situations where collisions between the end-effector and the environment are inevitable. To enhance environmental adaptability and prevent physical damage, the wrist can operate in a low-stiffness mode. However, the flexibility of this mode might negatively impact stable and precise manipulation. To address this, we proposed a robotic wrist that switches between a passive low-stiffness mode for environmental adaptation and an active high-stiffness mode for precise manipulation. Initially, we developed a functional prototype that could manually switch between these modes, demonstrating the wrist's passive low-stiffness and active high-stiffness states. This prototype was designed as a lightweight, flat-type modular device, incorporating a sheet-type flexure as the motion guide and embedding all essential components, including actuators, sensors, and a control unit, into the wrist module. Based on the functional prototype, we developed an improved version to enhance durability and functionality. The resulting wrist module incorporates a three-axis F/T sensor and an impedance control system to control the stiffness. It measures 55 mm in height, weighs 200 g, and offers a 232.4-fold active stiffness variation.

Keywords: Computer Vision for Automation, Computer Vision for Manufacturing, Recognition
Abstract: Active stereo vision (ASV) computes the parallax and depth information from the coded structured light patterns. Thus, it could overcome the difficulties of measuring objects without textures and colors. However, decoding of the structured light patterns at locations of color crosstalk, specular reflection and occlusion remains challenging. In this paper, we propose a natural-neighbor-interpolant-based pattern modeling method to decode the structured light point pattern robustly. The robustness is achieved in the sense of hundred percent point segmentation completeness. Due to the hundred percent completeness, the points in the corresponding blocks are matched directly according to their indexes. Experimental results verified the effectiveness of the proposed method.

Keywords: Computer Vision for Automation
Abstract: Video object detection (VOD) of motile cells (e.g., bacteria and sperm) under microscopy is challenging due to motion blur, sporadic out-of-focus, and pose variations. Compared with VOD in generic scenes, the lower contrast and smaller color space of microscopy imaging further introduce feature overlap between the foreground objects and the background objects (e.g., impurity cells and contaminants). Transformer-based methods have achieved great success in the VOD of generic scenes by utilizing object queries to model the inner-frame objects and the inter-frame objects. However, the appearance overlap problem in microscopy video frames significantly compromises the inter-frame query aggregation by introducing background features into the object query. To tackle this challenge, this paper reports a static-dynamic query-based VOD network that treats object queries of the current video frame and reference video frames differently. Specifically, a two-stage framework is implemented that first generates high-quality object queries of reference frames with a static Transformer decoder pre-trained on a still image dataset. The network is then trained on a per-frame annotated dataset using a dynamic Transformer decoder to model the object queries of the current frame. A Reference Query Relation Module is further proposed to enhance the reference queries for more effective aggregation with the current query. Experiments on clinically collected biopsied sperm datasets validated the effectiveness of the proposed method.

Keywords: Space Robotics and Automation, Mining Robotics, Learning from Demonstration
Abstract: To support sustainable infrastructure on the Moon, NASA is developing the In-Situ Resource Utilization (ISRU) Pilot Excavator (IPEx) to extract and transport lunar regolith for processing and construction. During its mission, IPEx will execute various driving patterns, primarily cycling between excavation and unloading sites, with additional maneuvers such as circular traverses around the lander and raster scans for environmental mapping. In this work, dynamic movement primitives (DMPs) are used to represent these patterns. We augment the DMPs with a vision-based real-time obstacle avoidance system to navigate surface hazards, such as rocks, encountered during traversal. Our approach is evaluated in a high-fidelity simulation replicating the challenging environment of the lunar south pole to demonstrate IPEx’s ability to adapt to surface hazards while fulfilling its operational tasks.

Keywords: Deep Learning for Visual Perception, Aerial Systems: Perception and Autonomy, Recognition
Abstract: Fire poses significant threats to life and property, necessitating efficient inspection and accurate identification. Although aerial computer vision algorithms hold great promise, the computational limitations of onboard platforms prevent existing algorithms from meeting high standards of accuracy and real-time performance. To address this challenge, we propose an lightweight aerial fire detection model, LAFNET. This model incorporates the EffiDarknetLight backbone, optimized for lightweight design, integrates specially designed LG block components within the LG PAN neck, resulting in a model Params of only 1.3M. Experimental results demonstrate that our method attains a good trade-off between lightweight design and detection accuracy. Compared to the smallest standard YOLO series' model YOLOv5n, LAFNET improves MAP by 2.1%, while reducing Params and FLOPs by 27.8% and 29.3%, the inference speed on Nvidia Orin Nano edge computing side improves 24.8%. These experiments indicate that LAFNET offers a highly efficient solution for aerial fire detection, combining speed and accuracy.

Keywords: Omnidirectional Vision, Computer Vision for Transportation, Intelligent Transportation Systems
Abstract: In recent years, with the rapid development of Advanced Driver Assistance Systems (ADAS), the demand for the precise and efficient surround view stitching system has significantly increased. Traditional stitching methods perform well in small single-unit vehicles with stable camera poses. However, the stitching quality sharply degrades when applied to large tractor-trailers due to the continuous pose changes caused by the non-rigid connection between the tractor and trailer. In detail, first, the extended length of tractor-trailers results in low overlap between cameras, making feature extraction and matching challenging. Additionally, the stitched images often appear irregular, detracting from visual quality. Besides, even if static stitching looks natural, it causes jitter in dynamic scenarios due to random feature extraction. In this paper, we propose an unsupervised deep stitching method for tractortrailer surround view system. We introduce a feature extraction module for tractor-trailer scenarios (FMT) to enhance feature extraction in low-overlap situations. Besides, we design a spatiotemporally consistent control point constraint strategy (STCC) to achieve spatial shape preservation and temporal smoothing effects, resulting in visually consistent and stable stitched sequences. Experimental results from both public and real dataset show that our method efficiently completes tractortrailer surround view stitching, producing well-aligned and natural panoramic images compared to previous methods.

Keywords: Computer Vision for Medical Robotics, Surgical Robotics: Laparoscopy, Visual Learning
Abstract: Despite advancements in robotic systems and surgical data science, ensuring safe execution in robot-assisted minimally invasive surgery (RMIS) remains challenging. Current methods for surgical error detection typically involve two parts: identifying gestures and then detecting errors within each gesture clip. These methods often overlook the rich contextual and semantic information inherent in surgical videos, with limited performance due to reliance on accurate gesture identification. Inspired by the chain-of-thought prompting in natural language processing, this letter presents a novel and real-time end-to-end error detection framework, Chain-of-Gesture (COG) prompting, integrating contextual information from surgical videos step by step. This encompasses two reasoning modules that simulate expert surgeons' decision-making: a Gestural-Visual Reasoning module using transformer and attention architectures for gesture prompting and a Multi-Scale Temporal Reasoning module employing a multi-stage temporal convolutional network with slow and fast paths for temporal information extraction. We validate our method on the JIGSAWS dataset and show improvements over the state-of-the-art, achieving 4.6% higher F1 score, 4.6% higher Accuracy, and 5.9% higher Jaccard index, with an average frame processing time of 6.69 milliseconds. This demonstrates our approach's potential to enhance RMIS safety and surgical education efficacy. The code is available at https://github.com/jinlab-imvr/Chain-of-Gesture.

Keywords: Aerial Systems: Mechanics and Control, Physical Human-Robot Interaction, Aerial Systems: Applications
Abstract: This paper presents a new cargo transportation solution based on physical human-robot interaction utilizing a novel fully-actuated multirotor platform called Palletrone. The platform is designed with a spacious upper flat surface for easy cargo loading, complemented by a rear-mounted handle reminiscent of a shopping cart. Flight trajectory control is achieved by a human operator gripping the handle and applying three-dimensional forces and torques while maintaining a stable cargo transport with zero roll and pitch attitude throughout the

flight. To facilitate physical human-robot interaction, we employ an admittance control technique. Instead of relying on complex force estimation methods, like in most admittance control implementations, we

introduce a simple yet effective estimation technique based on a disturbance observer robust control algorithm. We conducted an analysis

of the flight stability and performance in response to changes in system mass resulting from arbitrary cargo loading. Ultimately, we demonstrate that individuals can effectively control the system trajectory by applying appropriate interactive forces and torques. Furthermore, we showcase the performance of the system through various experimental scenarios.

Keywords: Aerial Systems: Mechanics and Control, Art and Entertainment Robotics, Tendon/Wire Mechanism
Abstract: Art is one of the oldest forms of human expression, constantly evolving, taking new forms and using new techniques. With their increased accuracy and versatility, robots can be considered as a new class of tools to perform works of art. The STRAD (STReet Art Drone) project aims to perform a 10-meter- high painting on a vertical surface with sub-centimetric precision. To achieve this goal we introduce a new design for an aerial manipulator with elastic suspension capable of moving from one equilibrium position to another using only its thrusters and an elliptic pulley-counterweight system. A feedback linearization control law is implemented to perform fast and accurate winding and unwinding of an elastic cable.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications, Mobile Manipulation
Abstract: Aerial physical interaction (APhI) with a multirotor-based platform such as pushing a heavy object demands generation of a sufficiently large interaction force while maintaining the stability. Such requirement can cause rotor saturation, because the rotor thrust enlarged for interaction force may leave a reduced margin for attitude stabilization. We first design an H -shaped coaxial tiltrotor that can generate a sufficiently large interaction force than a conventional multirotor. We then propose an overall framework composed of high-level robust controller and low-level control allocation for the coaxial tiltrotor to ensure robustness against uncertain motion of the unknown interacting object and to overcome the saturation issue. To guarantee the robustness at all time, we design a controller based on a nonlinear disturbance observer (DOB). Then, we formulate a problem of computing low-level actuator inputs avoiding rotor saturation as a tractable nonlinear optimization problem, which can be solved real-time. The proposed framework is validated in extensive real-world experiments where the 3.3 kg tiltrotor successfully pushes a cart weighing up to 60 kg. An ablation study with the tiltrotor shows effectiveness of the proposed control allocation law in avoiding rotor saturation. Furthermore, a comparative experiment with a conventional multirotor shows failure in the same setting, which validates the use of the coaxial tiltrotor. An experimental video can be found at htt

Keywords: Aerial Systems: Applications, Grasping, Motion Control
Abstract: Delivery by aerial robots is an emerging topic in many scenarios, such as logistics, construction industry, and disaster response. Compared to the standard styles that deploy cage or sling, grasping style by gripper can handle objects in various shapes. A multi-limbed structure with distributed vectorable rotors called SPIDAR shows a higher potential to grasp large object in a three-dimensional manner. Therefore, in this paper, we focus on the advanced usage of the vectored thrust forces to achieve aerial grasping by this robot. First, a vectored thrust control to avoid the aerointerference on the underwind segments (e.g., grasped object) during flight is proposed. Then, an optimization-based planning method that utilizes redundant vectored thrust forces for firm grasping is developed. Finally, we demonstrate the feasibility of the proposed flight control and grasp planning by performing challenging grasping and transporting motion with a spherical object of which the diameter is 0.6m. To the best of our knowledge, this work is the first to achieve multi-finger-like grasping to carry a large object in midair.

Keywords: Aerial Systems: Applications, Motion and Path Planning, Planning under Uncertainty
Abstract: This paper investigates payload grasping from a moving platform using a hook-equipped aerial manipulator. First, a computationally efficient trajectory optimization based on complementarity constraints is proposed to determine the optimal grasping time. To enable application in complex, dynamically changing environments, the future motion of the payload is predicted using a physics simulator-based model. The success of payload grasping under model uncertainties and external disturbances is formally verified through a robustness analysis method based on integral quadratic constraints. The proposed algorithms are evaluated in a high-fidelity physical simulator, and in real flight experiments using a custom-designed aerial manipulator platform.

Keywords: Aerial Systems: Applications, Aerial Systems: Mechanics and Control, Cooperating Robots, Physical Human-Robot Interaction
Abstract: Human-robot interaction will play an essential role in various industries and daily tasks, enabling robots to effectively collaborate with humans and reduce their physical workload. This paper proposes a novel approach for physical human- robot collaborative transportation and manipulation of a cable- suspended payload with multiple aerial robots. The proposed method enables smooth and intuitive interaction between the transported objects and a human worker. In the same time, we consider distance constraints during the operations by exploiting the internal redundancy of the multi-robot transportation system. We validate the approach through extensive simulation and real-world experiments. These include scenarios where the robot team assists the human in transporting and manipulating a load, or where the human helps the robot team navigate the environment. We experimentally demonstrate for the first time, to the best of our knowledge, that our approach enables a quadrotor team to physically collaborate with a human in manipulating a payload in all 6 DoF in collaborative human- robot transportation and manipulation tasks.

Keywords: Vision-Based Navigation
Abstract: Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to navigate with low-level actions following natural language instructions in 3D environments. Most existing approaches utilize observation features from the current step to represent the viewpoint. However, these representations often conflate redundant and essential information for navigation, introducing ambiguity into the agent's action prediction. To address the problem of inadequate representation, we propose a Knowledge-and-History Aware Visual Representation for Continuous Vision-and-Language Navigation (VLN-KHVR). The proposed approach constructs enriched visual representations tailored to navigation instructions, enhancing agents’ navigation performance. Specifically, VLN-KHVR extracts image features from the current observation, retrieves relevant knowledge in the knowledge base, and obtains the history of the navigation episode. Subsequently, the knowledge and history features are filtered to eliminate the information irrelevant to navigation instruction. These refined features are integrated with the instruction for further interaction. Finally, the aggregated features are used to guide navigation. Our model outperforms previous methods on the VLN-CE benchmark, demonstrating the effectiveness of the proposed method.

Keywords: Localization, Vision-Based Navigation, SLAM
Abstract: This paper presents LiteVLoc, a hierarchical vi-sual localization framework that uses a lightweight topo-metric map to represent the environment. The method consists of three sequential modules that estimate camera poses in a coarse-to-fine manner. Unlike dense 3D mapping methods, LiteVLoc reduces storage by avoiding geometric reconstruction. It uses a learning-based feature matcher to establish dense corre-spondences between sparse keyframes and observations, and then refines poses with a geometric solver, enabling robustness to viewpoint changes. The system assumes depth sensors or stereo camera for deployment. A novel dataset for the map-free relocalization task is also introduced. Extensive experiments including localization and navigation in both simulated and real-world scenarios have validate the system’s performance and demonstrated its precision and efficiency for large-scale de-ployment. Code and data will be made publicly available at the webpage: https://rpl-cs-ucl.github.io/LiteVLoc.

Keywords: Vision-Based Navigation, Search and Rescue Robots, Probabilistic Inference
Abstract: Image-goal navigation enables a robot to reach the location where a target image was captured, using visual cues for guidance. However, current methods either rely heavily on data and computationally expensive learning-based approaches or lack efficiency in complex environments due to insufficient exploration strategies. To address these limitations, we propose Bayesian Embodied Image-goal Navigation Using Gaussian Splatting, a novel method that formulates ImageNav as an optimal control problem within a model predictive control framework. BEINGS leverages 3D Gaussian Splatting as a scene prior to predict future observations, enabling efficient, real-time navigation decisions grounded in the robot’s sensory experiences. By integrating Bayesian updates, our method dynamically refines the robot's strategy without requiring extensive prior experience or data. Our algorithm is validated through extensive simulations and physical experiments, showcasing its potential for embodied robot systems in visually complex scenarios. Project Page: www.mwg.ink/BEINGS-web.

Keywords: View Planning for SLAM, Vision-Based Navigation, SLAM
Abstract: This paper presents FLAF, a focal line and feature-constrained active view planning method for tracking failure avoidance in feature-based visual navigation of mobile robots. FLAF is built on a feature-based visual teach and repeat (VT&R) framework, which supports robotic applications by teaching robots to cruise various paths that fulfill many daily autonomous navigation requirements. However, tracking failures in feature-based Visual Simultaneous Localization and Mapping (VSLAM), particularly in textureless regions common in human-made environments, poses a significant challenge to the real-world deployment of VT&R. To address this problem, the proposed view planner is integrated into a feature-based VSLAM system, creating an active VT&R solution that mitigates tracking failures. Our system features a Pan-Tilt Unit (PTU)-based active mounted on a mobile robot. Using FLAF, the active camera-based VSLAM (AC-SLAM) operates during the teaching phase to construct a complete path map and in the repeating phase to maintain stable localization. FLAF actively directs the camera toward more map points to avoid mapping failures during path learning and toward more feature-identifiable map points while following the learned trajectory. Experimental results in real scenarios show that FLAF significantly outperforms existing methods by accounting for feature identifiability, particularly the view angle of the features. While effectively dealing with low-texture regions in active view planning, considering feature identifiability enables our active VT&R system to perform well in challenging environments.

Keywords: Deep Learning Methods, Vision-Based Navigation
Abstract: Vision-and-Language Navigation (VLN) empowers agents to associate time-sequenced visual observations with corresponding instructions to make sequential decisions. However, dealing with visually diverse scenes or transitioning from simulated environments to real-world deployment is still challenging. In this paper, we address the mismatch between human-centric instructions and quadruped robots with a low-height field of view, proposing a Ground-level Viewpoint Navigation (GVNav) approach to mitigate this issue. This work represents the first attempt to highlight the generalization gap in VLN across varying heights of visual observation in realistic robot deployments. Our approach leverages weighted historical observations as enriched spatiotemporal contexts for instruction following, effectively managing feature collisions within cells by assigning appropriate weights to identical features across different viewpoints. This enables low-height robots to overcome challenges such as visual obstructions and perceptual mismatches. Additionally, we transfer the connectivity graph from the HM3D and Gibson datasets as an extra resource to enhance spatial priors and a more comprehensive representation of real-world scenarios, leading to improved performance and generalizability of the waypoint predictor in real-world environments. Extensive experiments demonstrate that our Ground-level Viewpoint Navigation (GVnav) approach significantly improves performance in both simulated environments and real-world deployments with quadruped robots.

Keywords: Vision-Based Navigation, Representation Learning, Reinforcement Learning
Abstract: This paper introduces Navigation Transformer (NavTr), a novel framework for object-goal navigation using Transformer queries to enhance the learning and representation of environment states. By integrating semantic information, object positions, and neighborhood information, NavTr creates a unified, comprehensive, and extensible state representation for the object-goal navigating task. In the framework, the Transformer queries implicitly learn inter-object relationships, which facilitates high-level understanding of the environment. Additionally, NavTr implements target-oriented supervisory signals, such as rotation rewards and spatial loss, which improve exploration efficiency in the reinforcement learning framework. NavTr outperforms popular graph-based and Attention-based methods by a large margin in terms of success rate (SR) and success weighted by path length (SPL). Extensive experiments on the AI2-THOR dataset demonstrate the effectiveness of our approach.

Keywords: Marine Robotics, Autonomous Vehicle Navigation, SLAM
Abstract: Existing bathymetric simultaneous localization and mapping (SLAM) methods predominantly rely on odometry information for loop closure detection, which has a deteriorating performance when handling unreliable odometry data or conducting large-scale mapping missions. This letter introduces a novel generalized Bag of Words (BoW) named Shape BoW (S-BoW) for appearance-based loop closure detection in bathymetric SLAM. S-BoW is trained from the collection of the terrain gradient features extracted from existing bathymetric datasets and can be used in various bathymetric scenarios. We integrated the loop closure detection method using S-BoW into a feature-based bathymetric SLAM method called TTT SLAM, and we evaluated its performance against three existing bathymetric SLAM methods using two datasets. The results indicate that S-BoW not only serves as a generalized BoW but also enhances the efficiency of the integrated SLAM method, achieving accuracy comparable to the original TTT SLAM while offering a 37% speed improvement in a large-scale sea trial dataset. To the best of our knowledge, S-BoW is the first generalized BoW that can be used to realize effective appearance-based loop closure detection in bathymetric SLAM.

Keywords: Marine Robotics, Recognition, Data Sets for Robot Learning
Abstract: Oysters are an important keystone species in coastal ecosystems that provide several economic, environmental, and cultural benefits. Given the array of utilities derived from oysters, the application of autonomous robotic systems for oyster detection and monitoring grows increasingly relevant. However, current monitoring strategies for assessing oyster assemblages are mostly destructive. While manually identifying and monitoring oysters from video footage is nondestructive, is it tedious and requires expert input.

An alternative to human monitoring is deploying trained object detection models on edge devices, such as the Aqua2 robot, to enable real-time monitoring of oysters directly in the field. Yet training these models to maximum efficacy requires an extensive dataset that accurately represents the domain, and it is difficult to obtain such high-quality training data due to the complications inherent to underwater environments. To address these complications, we introduce a novel method leveraging stable diffusion to generate high-quality synthetic data for the marine domain. We exploit diffusion models to create photorealistic oyster imagery, using ControlNet inputs to ensure consistency with the segmentation ground-truth mask, the geometry of the scene, and the target domain of real oyster images. This large dataset is used to train a vision model, specifically based on YOLOv10. The trained model is then deployed and tested on an edge platform, the Aqua2, in an underwater robotics system. We achieve state-of-the-art (0.657 mAP@50) for oyster detection, which can pave the way for autonomous oyster habitat monitoring and increase the efficiency of on-bottom oyster aquaculture

Keywords: Marine Robotics, Data Sets for Robotic Vision, Visual Learning
Abstract: We present an image blending pipeline, IBURD, that creates realistic synthetic images to assist in the training of deep detectors for use on underwater autonomous vehicles (AUVs) for marine debris detection tasks. Specifically, IBURD generates both images of underwater debris and their pixel-level annotations, using source images of debris objects, their annotations, and target background images of marine environments. With Poisson editing and style transfer techniques, IBURD is even able to robustly blend transparent objects into arbitrary backgrounds and automatically adjust the style of blended images using the blurriness metric of target background images. These generated images of marine debris in actual underwater backgrounds address the data scarcity and data variety problems faced by deep-learned vision algorithms in challenging underwater conditions, and can enable the use of AUVs for environmental cleanup missions. Both quantitative and robotic evaluations of IBURD demonstrate the efficacy of the proposed approach for robotic detection of marine debris.

Keywords: Marine Robotics, Mapping
Abstract: Underwater autonomous robotic operations require online localization and 3D mapping. Because of the absence of absolute positioning underwater, these tasks strongly rely on embedded sensors, including proprioceptive or navigation sensors — which can be fused for an odometry, — and exteroceptive sensors. One of the most popular exteroceptive sensors for underwater is the imaging sonar, which emits a large fan-shaped acoustic signal and estimates the position of the surrounding obstacles from a measure of the reflected signal. This paper addresses underwater online localization and 3D mapping using a forward looking, wide-aperture imaging sonar and vehicle’s intrinsic navigation estimates. We introduce 3DSSDF (3D Sonar Reconstruction Using Signed Distance Functions), a new localization and 3D mapping algorithm based on signed distance functions, which is evaluated in simulation and on real data, in man-made and natural environments. Comparisons to reference trajectories and maps demonstrate that, in our tests, 3DSSDF efficiently corrects navigation drift and that trajectory and map accuracy is always below 1 m and below 1% of the distanced travelled, which can be sufficient for the safe inspection of natural or artificial underwater structures.

Keywords: Marine Robotics, Localization, Sensor Fusion
Abstract: This paper presents a novel cascade nonlinear observer framework for inertial state estimation. It tackles the problem of intermediate state estimation when external localization is unavailable or in the event of a sensor outage. The proposed observer comprises two nonlinear observers based on a recently developed iteratively preconditioned gradient descent (IPG) algorithm. It takes the inputs via an IMU preintegration model where the first observer is a quaternion-based IPG. The output for the first observer is the input for the second observer, estimating the velocity and, consequently, the position. The proposed observer is validated on a public underwater dataset and a real-world experiment using our robot platform. The estimation is compared with an extended Kalman filter (EKF) and an invariant extended Kalman filter (InEKF). Results demonstrate that our method outperforms these methods regarding better positional accuracy and lower variance.

Keywords: Marine Robotics, Planning under Uncertainty, Collision Avoidance
Abstract: Motion planning for autonomous active perception in cluttered environments remains a challenging problem, requiring real-time solutions that both maximize safety and achieve a desired behavior. In dynamic underwater environments, such as in aquaculture operations, the robots are additionally expected to deal with state and motion uncertainty and errors, dynamic and deformable obstacles, currents, and disturbances. Previous work has introduced real-time frameworks that provided safe navigation in cluttered environments, active perception in static environments, and robust navigation in uncertain dynamic environments. This paper introduces a new real-time approach called ResiVis, which leverages the best aspects of the aforementioned techniques along with a new formulation that further enhances underwater autonomy by enabling active perception of static and dynamic target objects from desired distances. The proposed method utilizes path-optimization for real-time response with constraints guaranteeing continuous collision safety, and computes paths with clearance adaptive to both the conditions of the environments and the performance of the path follower. An improved new constraint encourages observations of dynamic objects with the planner adapting to satisfy desired observation distances and their projected future positions. ResiVis is validated with challenging simulation experiments and with hardware-in-the-loop trials in real industrial-scale aquaculture facilities.

Keywords: Legged Robots, Optimization and Optimal Control, Dynamics
Abstract: Symmetrical gaits, such as trotting, are com- monly employed in quadrupedal robots for their simplicity and stability. However, the potential of asymmetrical gaits, such as bounding and galloping—which are prevalent in their natural counterparts at high speeds or over long distances—is less clear in the design of locomotion controllers for legged machines. In these asymmetrical gaits, the system dynamics are more complex because the front and rear leg pairs exhibit different motions, which are coupled by the rotational motion of the torso. This study systematically examines five distinct asymmetrical quadrupedal gaits on a legged robot, aiming to uncover the fundamental differences in footfall sequences and the consequent energetics across a broad range of speeds. Utilizing a full-body model of a quadrupedal robot (Unitree A1), we developed a hybrid system for each gait, incorporating the desired footfall sequence and rigid impacts. To identify the most energy-optimal gait, we applied optimal control methods, framing it as a trajectory optimization problem with specific constraints and a work-based cost of transport as an objective function. Our results show that, in the context of asymmetrical gaits, when minimizing cost of transport across the entire stride, the front leg pair primarily propels the system forward, while the rear leg pair acts more like an inverted pendulum, contributing significantly less to the energetic output. Addi- tionally, while bounding—characterized by two aerial phases per cycle—is the most energy-optimal gait at higher speeds, the energy expenditure of gaits at speeds below 1 m/s depend heavily on the robot’s specific design.

Keywords: Legged Robots, Compliant Joints and Mechanisms, Motion Control
Abstract: In biomechanics and robotics, elasticity plays a crucial role in enhancing locomotion efficiency and stability. Traditional approaches in legged robots often employ series elastic actuators (SEA) with discrete rigid components, which, while effective, add weight and complexity. This paper presents an innovative alternative by integrating continuously compliant structures into the lower legs of a bipedal robot, fundamentally transforming the SEA concept. Our approach replaces traditional rigid segments with lightweight, deformable materials, reducing overall mass and simplifying the actuation design. This novel design introduces unique challenges in modeling, sensing, and control, due to the infinite dimensionality of continuously compliant elements. We address these challenges through effective approximations and control strategies. The paper details the design and modeling of the compliant leg structure, presents low-level force and kinematics controllers, and introduces a high-level posture controller with a gait scheduler. Experimental results demonstrate successful bipedal walking using this new design.

Keywords: Legged Robots, Robust/Adaptive Control, Multi-Contact Whole-Body Motion Planning and Control
Abstract: As legged robots continue to evolve, new control methods are being developed to provide fast, robust, accurate and computationally efficient algorithms for traversing challenging environments. This paper presents a real-time adaptive locomotion controller for quadrupeds, designed to maintain stability and controllability on various surfaces, including highly slippery terrains. The proposed approach optimizes control effort distribution based on the probability of slippage by utilizing a surface-independent adaptation layer. By balancing the robot's redundant kinematic system through rank relaxation—similar to loosening constraints in optimization problems—this method demonstrates significant performance improvements. Unlike Reinforcement Learning (RL) approaches, which depend on pre-trained policies and may struggle to adapt velocity tracking control across different terrains, our method rapidly adjusts to changing conditions, as validated by extensive simulation experiments.

Keywords: Legged Robots, Reinforcement Learning, Natural Machine Motion
Abstract: In reinforcement learning for legged robot locomotion, crafting effective reward strategies is crucial. Predefined gait patterns and complex reward systems are widely used to stabilize policy training. Drawing from the natural locomotion behaviors of humans and animals, which adapt their gaits to minimize energy consumption, we investigate the impact of incorporating an energy-efficient reward term that prioritizes distance-averaged energy consumption into the reinforcement learning framework. Our findings demonstrate that this simple addition enables quadruped robots to autonomously select appropriate gaits—such as four-beat walking at lower speeds and trotting at higher speeds—without the need for explicit gait regularizations. Furthermore, we provide a guideline for tuning the weight of this energy-efficient reward, facilitating its application in real-world scenarios. The effectiveness of our approach is validated through simulations and on a real Unitree Go1 robot. This research highlights the potential of energy-centric reward functions to simplify and enhance the learning of adaptive and efficient locomotion in quadruped robots. Videos and more details are at https://sites.google.com/berkeley.edu/efficient-locomotion.

Keywords: Legged Robots, Reinforcement Learning, Biomimetics
Abstract: We address the challenge of effectively controlling the locomotion of legged robots by incorporating precise frequency and phase characteristics, which is often ignored in locomotion policies that do not account for the periodic nature of walking. We propose a hierarchical architecture that integrates a low-level phase tracker, oscillators, and a high-level phase modulator. This controller allows quadruped robots to walk in a natural manner that is synchronized with external musical rhythms. Our method generates diverse gaits across different frequencies and achieves real-time synchronization with music in the physical world. This research establishes a foundational framework for enabling real-time execution of accurate rhythmic motions in legged robots. The video and code are available at https://music-walker.github.io/.

Keywords: Humanoid Robot Systems, Sensorimotor Learning, Representation Learning
Abstract: Humanoid robots require both robust lower-body locomotion and precise upper-body manipulation. While recent Reinforcement Learning (RL) approaches provide whole-body loco-manipulation policies, they lack precise manipulation with high DoF arms. In this paper, we propose decoupling upper-body control from locomotion, using inverse kinematics (IK) and motion retargeting for precise manipulation, while RL focuses on robust lower-body locomotion. We introduce PMP (Predictive Motion Priors), trained with Conditional Variational Autoencoder (CVAE) to effectively represent upper-body motions. The locomotion policy is trained and conditioned on this upper-body motion representation, ensuring that the system remains robust with both manipulation and locomotion. We show that CVAE features are crucial for stability and robustness, and significantly outperforms RL-based whole-body control in precise manipulation. With precise upper-body motion and robust lower-body locomotion control, operators can remotely control the humanoid to walk around and explore different environments, while performing diverse manipulation tasks.

Keywords: Multi-Robot Systems, Collision Avoidance, Optimization and Optimal Control
Abstract: This paper proposes a distributed controller synthesis framework for safe navigation of multi-agent systems. We leverage control barrier functions to formulate collision avoidance with obstacles and teammates as constraints on the control input for a state-dependent network optimization problem that encodes team formation and the navigation task. Our algorithmic solution is valid under general assumptions for nonlinear dynamics and state-dependent network optimization problems with convex constraints and strongly convex objectives. The resulting controller is distributed, satisfies the safety constraints at all times, and asymptotically converges to the solution of the state-dependent network optimization problem. We illustrate its performance in a team of differential-drive robots in a variety of complex environments, both in simulation and in hardware.

Keywords: Multi-Robot Systems, Cooperating Robots, Constrained Motion Planning
Abstract: This paper presents a framework for multi-agent navigation in structured but dynamic environments, integrating three key components: a shared semantic map encoding metric and semantic environmental knowledge, a claim policy for coordinating access to areas within the environment, and a Model Predictive Controller for generating motion trajectories that respect environmental and coordination constraints. The main advantages of this approach include: (i) enforcing area occupancy constraints derived from specific task requirements; (ii) enhancing computational scalability by eliminating the need for collision avoidance constraints between robotic agents; and (iii) the ability to anticipate and avoid deadlocks between agents. The paper includes both simulations and physical experiments demonstrating the framework’s effectiveness in various representative scenarios

Keywords: Aerial Systems: Applications, Distributed Robot Systems, Collision Avoidance
Abstract: Multi-quadrotor systems face significant challenges in decentralized control, particularly with safety and coordination under sensing and communication limitations. State-of-the-art methods leverage Control Barrier Functions (CBFs) to provide safety guarantees but often neglect actuation constraints and limited detection range. To address these gaps, we propose a novel decentralized Nonlinear Model Predictive Control (NMPC) that integrates Exponential CBFs (ECBFs) to enhance safety and optimality in multi-quadrotor systems. We provide both conservative and practical minimum bounds of the range that preserve the safety guarantees of the ECBFs. We validate our approach through extensive simulations with up to 10 quadrotors and 20 obstacles, as well as real-world experiments with 3 quadrotors. Results demonstrate the effectiveness of the proposed framework in realistic settings, highlighting its potential for reliable quadrotor teams operations.

Keywords: Deep Learning Methods, Path Planning for Multiple Mobile Robots or Agents, Reinforcement Learning
Abstract: The Multi-Agent Path Finding (MAPF) problem aims to determine the shortest and collision-free paths for multiple agents in a known, potentially obstacle-ridden environment. It is the core challenge for robotic deployments in large-scale logistics and transportation. Decentralized learning-based approaches have shown great potential for addressing the MAPF problems, offering more reactive and scalable solutions. However, existing learning-based MAPF methods usually rely on agents making decisions based on a limited field of view (FOV), resulting in short-sighted policies and inefficient cooperation in complex scenarios. There, a critical challenge is to achieve consensus on potential movements between agents based on limited observations and communications. To tackle this challenge, we introduce a new framework that applies sheaf theory to decentralized deep reinforcement learning, enabling agents to learn geometric cross-dependencies between each other through local consensus and utilize them for tightly cooperative decision-making. In particular, sheaf theory provides a mathematical proof of conditions for achieving global consensus through local observation. Inspired by this, we incorporate a neural network to approximately model the consensus in latent space based on sheaf theory and train it through self-supervised learning. During the task, in addition to normal features for MAPF as in previous works, each agent distributedly reasons about a learned consensus feature, leading to efficient cooperation on pathfinding and collision avoidance. As a result, our proposed method demonstrates significant improvements over state-of-the-art learning-based MAPF planners, especially in relatively large and complex scenarios, demonstrating its superiority over baselines in various simulations and real-world robot experiments.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Multi-Robot Systems, Distributed Robot Systems
Abstract: This work presents an algorithm based on the Nondominated Sorting Genetic Algorithm II (NSGA-II) to solve multi-objective offline Multi-Robot Coverage Path Planning (MCPP) problems. The proposed algorithm embeds a donation-mutation operator and a multiple-parent crossover that generates solutions which maintain the longest path while minimizing the average path length. The algorithm also uses a library of elitism-selected high-fitness robot paths, and tournament-selected high min-max fitness paths, to construct high multi-objective fitness offspring. We evaluate the performance of our proposed algorithm against the state-of-the-art NSGA-II extended with an improved Heuristic Genetic Algorithm Crossover, and we demonstrate that for different instances of the MCPP problem, the Pareto-fronts of our proposed algorithm are not dominated by any of the points of the fronts generated by the state-of-the-art NSGA-II. A comparison has also been performed in a virtual environment simulating five drones inspecting three wind turbines. Results show that our approach exhibits a higher convergence rate for higher values of the ratio between the number of points to visit and the number of drones.

Keywords: Formal Methods in Robotics and Automation, Planning, Scheduling and Coordination, Multi-Robot Systems
Abstract: This paper presents an iterative approach for heterogeneous multi-agent route planning in environments with unknown resource distributions. We focus on a team of robots with diverse capabilities tasked with executing missions specified using Capability Temporal Logic (CaTL), a formal framework built on Signal Temporal Logic to handle spatial, temporal, capability, and resource constraints. The key challenge arises from the uncertainty in the initial distribution and quantity of resources in the environment. To address this, we introduce an iterative algorithm that dynamically balances exploration and task fulfillment. Robots are guided to explore the environment, identifying resource locations and quantities while progressively refining their understanding of the resource landscape. At the same time, they aim to maximally satisfy the mission objectives based on the current information, adapting their strategies as new data is uncovered. This approach provides a robust solution for planning in dynamic, resource-constrained environments, enabling efficient coordination of heterogeneous teams even under conditions of uncertainty. Our method's effectiveness and performance are demonstrated through simulated case studies.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Integrated Planning and Learning, Distributed Robot Systems
Abstract: This letter addresses the distributed informative path planning (IPP) problem for a mobile robot network to optimally explore a spatial field. Each robot is able to gather noisy environmental measurements while navigating the environment and build its own model of a spatial phenomenon using the Gaussian process and local data. The IPP optimization problem is formulated in an informative way through a multi-step prediction scheme constrained by connectivity preservation and collision avoidance. The shared hyperparameters of the local Gaussian process models are also arranged to be optimally computed in the path planning optimization problem. By the use of the proximal alternating direction method of multiplier, the optimization problem can be effectively solved in a distributed manner. It theoretically proves that the connectivity in the network is maintained over time whilst the solution of the optimization problem converges to a stationary point. The effectiveness of the proposed approach is verified in synthetic experiments by utilizing a real-world dataset.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Planning, Scheduling and Coordination, Task Planning
Abstract: The Multiple Depot Traveling Salesman Problem (MDTSP) is a variant of the NP-hard Traveling Salesman Problem (TSP) with more than one salesman to jointly visit all destinations, commonly found in task planning in multi-agent robotic systems. Traditional MDTSP overlooks practical constraints like limited battery level and inter-agent conflicts, often leading to infeasible or unsafe solutions in reality. In this work, we incorporate energy and resource consumption constraints to form the Constrained MDTSP (CMDTSP). We design a novel hierarchical framework to obtain high-quality solutions with low computational complexity. The framework decomposes a given CMDTSP instance into manageable sub-problems, each handled individually via a TSP solver and heuristic search to generate tours. The tours are then aggregated and processed through a Mixed-Integer Linear Program (MILP), which contains significantly fewer variables and constraints than the MILP for the exact CMDTSP, to form a feasible solution efficiently. We demonstrate the performance of our framework on both real-world and synthetic datasets. It reaches a mean 12.48% optimality gap and 41.7x speedup over the exact method on common instances and a 5.22%sim14.84% solution quality increase with more than 79.8x speedup over the best baseline on large instances where the exact method times out.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Environment Monitoring and Management, Integrated Planning and Learning
Abstract: Autonomous robots can survey and monitor large environments. However, these robots often have limited computational and power resources, making it crucial to develop an efficient and adaptive informative path planning (IPP) algorithm. Such an algorithm must quickly adapt to environmental data to maximize the information collected while accommodating path constraints, such as distance budgets and boundary limitations.

Current approaches to this problem often rely on maximizing mutual information using methods such as greedy algorithms, Bayesian optimization, and genetic algorithms. These methods can be slow and do not scale well to large or 3D environments. We present an adaptive IPP approach that is fully differentiable, significantly faster than previous methods, and scalable to 3D spaces. Our approach also supports continuous sensing robots, which collect data continuously along the entire path, by leveraging streaming sparse Gaussian processes.

Benchmark results on two real-world datasets demonstrate that our approach yields solutions that are on par with or better than baseline methods while being up to two orders of magnitude faster. Additionally, we showcase our adaptive IPP approach in a 3D space using a system-on-chip embedded computer with minimal computational resources. Our code is available in the SGP-Tools Python library with a companion ROS 2 package for deployment on ArduPilot-based robots.

Keywords: Environment Monitoring and Management, Motion and Path Planning, Reactive and Sensor-Based Planning
Abstract: Information gathering focuses on designing strategies for a robot to collect data about a physical process, aiming for accurate field reconstruction. While many recent methods have been proposed to address this problem, they often assume the model of the physical process is a priori known and stationary—assumptions that rarely hold in practice. This paper presents a novel informative motion planning approach for online information gathering of a non-stationary Gaussian process. Our approach comprises two key components: an informative path planner that explores the physical field and an adaptive velocity planner that adjusts the robot's velocity profile exploiting the field's spatial variability. Additionally, we propose a path smoothing and tracking strategy to ensure continuous robot motion. Extensive simulations on a bathymetric mapping task demonstrate the effectiveness of our approach, showing superior performance in reconstructing non-stationary physical fields compared to several baseline methods.

Keywords: Search and Rescue Robots, Path Planning for Multiple Mobile Robots or Agents, Robotics in Under-Resourced Settings
Abstract: Smart factories enhance production efficiency and sustainability, but emergencies like human errors, machinery failures and natural disasters pose significant risks. In critical situations, such as fires or earthquakes, collaborative robots can assist first-responders by entering damaged buildings and locating missing persons, mitigating potential losses. Unlike previous solutions that overlook the critical aspect of energy management, in this paper we propose REACT, a smart energy-aware orchestrator that optimizes the exploration phase, ensuring prolonged operational time and effective area coverage. Our solution leverages a fleet of collaborative robots equipped with advanced sensors and communication capabilities to explore and navigate unknown indoor environments, such as smart factories affected by fires or earthquakes, with high density of obstacles. By leveraging real-time data exchange and cooperative algorithms, the robots dynamically adjust their paths, minimize redundant movements and reduce energy consumption. Extensive simulations confirm that our approach significantly improves the efficiency and reliability of search and rescue missions in complex indoor environments, improving the exploration rate by 10% over existing methods and reaching a map coverage of 97% under time critical operations, up to nearly 100% under relaxed time constraint.

Keywords: Aerial Systems: Perception and Autonomy, Vision-Based Navigation, Path Planning for Multiple Mobile Robots or Agents
Abstract: In this work, we consider the problem of multi-agent informative path planning (IPP) for robots whose sensor visibility evolves over time as a consequence of a time-varying natural phenomenon. We leverage ergodic trajectory optimization (ETO), which generates paths such that the amount of time an agent spends in an area is proportional to the expected information in that area. We focus specifically on the problem of multi-agent drone search of a wildfire, where we use the time-varying environmental process of smoke diffusion to construct a sensor visibility model. This sensor visibility model is used to repeatedly calculate an expected information distribution (EID) to be used in the ETO algorithm. Our experiments show that our exploration method achieves improved information gathering over both baseline search methods and naive ergodic search formulations.

Keywords: Optimization and Optimal Control, Machine Learning for Robot Control, Model Learning for Control
Abstract: Model predictive control (MPC) has been applied to many platforms in robotics and autonomous systems for its capability to predict a system's future behavior while incorporating constraints that a system may have. To enhance the performance of a system with an MPC controller, one can manually tune the MPC's cost function. However, it can be challenging due to the possibly high dimension of the parameter space as well as the potential difference between the open-loop cost function in MPC and the overall closed-loop performance metric function. This paper presents DiffTune-MPC, a novel learning method, to learn the cost function of an MPC in a closed-loop manner. The proposed framework is compatible with the scenario where the time interval for performance evaluation and MPC's planning horizon have different lengths. We show the auxiliary problem whose solution admits the analytical gradients of MPC and discuss its variations in different MPC settings, including nonlinear MPCs that are solved using sequential quadratic programming. Simulation results demonstrate the learning capability of DiffTune-MPC and the generalization capability of the learned MPC parameters.

Keywords: Robot Safety, Motion Control, Robust/Adaptive Control
Abstract: 轮式自重构机器人 （WSRRs） 是一种新型的多机器人系统，具有灵活的配置和任务适应性，在非结构化任务环境中具有广泛的应用前景。该文基于非完整约束和拉格朗日方法，建立了具有任意重配置尺度的 WSRR 的组合模态运动学和动力学。在运动学层面，基于非完整约束，设计了基于安全地理围栏的平滑避障策略来确保安全。在动态层面，引入自适应容错机制，保证合理的扭矩分配，避免跟踪性能下降。同时，该文阐述了一种改进的扩展状态观测器（IESO），通过该算法可以抑制测量噪声的高频振荡和初始观测器误差的峰值现象，实现了未知集总扰动下鲁棒的速度跟踪控制

Keywords: Robot Safety, Intelligent Transportation Systems, Autonomous Vehicle Navigation
Abstract: Simulation is an indispensable tool in the development and testing of autonomous vehicles (AVs), offering an efficient and safe alternative to road testing. An outstanding challenge with simulation-based testing is the generation of safety-critical scenarios, which are essential to ensure that AVs can handle rare but potentially fatal situations. This paper addresses this challenge by introducing a novel framework, CaDRE, to generate realistic, diverse, and controllable safety-critical scenarios. Our approach optimizes for both the quality and diversity of scenarios by employing a unique formulation and algorithm that integrates real-world scenarios, domain knowledge, and black-box optimization. We validate the effectiveness of our framework through extensive testing in three representative types of traffic scenarios. The results demonstrate superior performance in generating diverse and high-quality scenarios with greater sample efficiency than existing reinforcement learning (RL) and sampling-based methods.

Keywords: Robot Safety, Reinforcement Learning, Machine Learning for Robot Control
Abstract: Control Barrier Functions (CBFs) have emerged as a prominent approach to designing safe navigation systems of robots. Despite their popularity, current CBF-based methods exhibit some limitations: optimization-based safe control techniques tend to be either myopic or computationally intensive, and they rely on simplified system models; conversely, the learning-based methods suffer from the lack of quantitative indication in terms of navigation performance and safety. In this paper, we present a new model-free reinforcement learning algorithm called Certificated Actor-Critic (CAC), which introduces a hierarchical reinforcement learning framework and well-defined reward functions derived from CBFs. We carry out theoretical analysis and proof of our algorithm, and propose several improvements in algorithm implementation. Our analysis is validated by two simulation experiments, showing the effectiveness of our proposed CAC algorithm.

Keywords: Robot Safety, Machine Learning for Robot Control
Abstract: Hamilton-Jacobi (HJ) reachability analysis is a widely adopted verification tool to provide safety and performance guarantees for autonomous systems. However, it involves solving a partial differential equation (PDE) to compute a safety value function, whose computational and memory complexity scales exponentially with the state dimension, making its direct application to large-scale systems intractable. To overcome these challenges, DeepReach,a recently proposed learning-based approach, approximates high-dimensional reachable tubes using neural networks (NNs). While shown to be effective, the accuracy of the learned solution decreases with system complexity. One of the reasons for this degradation is a soft imposition of safety constraints during the learning process, which corresponds to the boundary conditions of the PDE, resulting in inaccurate value functions. In this work, we propose ExactBC, a variant of DeepReach that imposes safety constraints exactly during the learning process by restructuring the overall value function as a weighted sum of the boundary condition and the NN output. Moreover, the proposed variant no longer needs a boundary loss term during the training process, thus eliminating the need to balance different loss terms. We demonstrate the efficacy of the proposed approach in significantly improving the accuracy of the learned value function for four challenging reachability tasks: a rimless wheel system with state resets, collision avoidance in a cluttered environment, autonomous rocket landing, and multi-aircraft collision avoidance.

Keywords: Robot Safety, AI-Based Methods, Probability and Statistical Methods
Abstract: AI-controlled robotic systems can introduce significant risks to both humans and the environment. Traditional reliability assessment methods fall short in addressing the complexities of these systems, particularly when dealing with black-box or dynamically changing control policies. The traditional approaches are applied manually and do not consider frequent software updates. In this paper, we present RelAIBotiX, a new methodology that enables dynamic and continuous reliability assessment, specifically tailored for robotic systems controlled by AI-Algorithms. RelAIBotiX is a dynamic reliability assessment framework that combines four methods: (i) Skill Detection that automatically identifies executed skills using deep learning techniques, (ii) Behavioral Analysis that creates an operational profile of the robotic system containing information about the skill execution sequence, active components for each skill, and their utilization intensity that influence their failure rate, (iii) Reliability Model Generation that automatically transforms the operational profile and reliability data of robotic hardware components into quantitative hybrid reliability models, and (iv) Reliability Model Solver for the numerical evaluation of the generated reliability models. Our evaluation included computing the reliability of the system, the probability of failure of individual skills, and component sensitivity analysis. We validated the applicability of the proposed framework in five simulative and real-world setups.

Keywords: Emotional Robotics, Gesture, Posture and Facial Expressions, Robot Companions
Abstract: Social robots have been extensively studied in recent decades, with many researchers exploring the use of modalities such as facial expressions to achieve more natural emotions in robots. Various methods have been attempted to generate and express robot emotions, including computational models that define an affect space and show dynamic emotion changes. However, the implementation of multimodal expression in previous models is ambiguous, and the generation of emotions in response to stimuli relies on heuristic methods. In this paper, we present a framework that enables robots to naturally express their emotions in a multimodal way, where the emotion can change over time based on the given stimulus values. By representing the robot’s emotion as a position in an affect space of a computational emotion model, we consider the given stimuli values as driving forces that can shift the emotion position dynamically. In order to examine the feasibility of our proposed method, a mobile robot prototype was implemented that can recognize touch and express different emotions with facial expressions and movements. The experiment demonstrated that the emotion elicited by a given stimulus is contingent upon the robot’s previous state, thereby imparting the impression that the robot possesses a distinctive emotion model. Furthermore, the Godspeed survey results indicated that our model was rated significantly higher than the baseline, which did not include a computational emotion model, in terms of anthropomorphism, animacy, and perceived intelligence. Notably, the unpredictabil ity of emotion switching contributed to a perception of greater lifelikeness, which in turn enhanced the overall interaction experience.

Keywords: Human-Robot Collaboration, Multi-Modal Perception for HRI, Embedded Systems for Robotic and Automation
Abstract: Intelligent virtual agents are used to accomplish complex multi-modal tasks such as human instruction comprehension in mixed-reality environments by increasingly adopting richer, energy-intensive sensors and processing pipelines. In such applications, the context for activating sensors and processing blocks required to accomplish a given task instance is usually manifested via multiple sensing modes. Based on this observation, we introduce a novel Commit-and-Switch (CAS) paradigm that simultaneously seeks to reduce both sensing and processing energy. In CAS, we first commit to a low-energy computational pipeline with a subset of available sensors. Then, the task context estimated by this pipeline is used to optionally switch to another energy-intensive DNN pipeline and activate additional sensors. We demonstrate how CAS’s paradigm of interweaving DNN computation and sensor triggering can be instantiated principally by constructing multi-head DNN models and jointly optimizing the accuracy and sensing costs associated with different heads. We exemplify CAS via the development of the RealGIN-MH model for multi-modal target acquisition tasks, a core enabler of immersive human-agent interaction. RealGIN-MH achieves 12.9x reduction in energy overheads, while outperforming baseline dynamic model optimization approaches.

Keywords: Social HRI, Physical Human-Robot Interaction
Abstract: Humor is a key element in human interactions, essential for building connections and rapport. To enhance human-robot communication, we developed a humor-aware chat framework that enables robots to deliver contextually appropriate humor. This framework takes into account the interaction environment, and user’s profile as well as emotional state. Two GPT models are used to generate responses. The initial one, named sensor-GPT, processes contextual data from the sensor along with the user’s response and conversation history to create prompts for the second one, chat-GPT. These prompts can guide the model on how to integrate appropriate humor elements into the conversation, ensuring that the dialogue is both contextually relevant and humorous. Our experiment compared the effectiveness of humor expression between our framework and the GPT-4o model. The results demonstrate that robots using our framework significantly outperform those using GPT-4o in humor expression, extending conversations, and improving overall interaction quality.

Keywords: Physical Human-Robot Interaction, Touch in HRI
Abstract: Humans are able to convey different messages using only touch. Equipping robots with the ability to understand social touch adds another modality in which humans and robots can communicate. In this paper, we present a social gesture recognition system using a fabric-based, large-scale tactile sensor integrated onto the arms of a humanoid robot. We built a social gesture dataset using multiple participants and extracted temporal features for classification. By collecting real-world data on a humanoid robot, our system provides valuable insights into human-robot social touch, further advancing the development of spHRI systems for more natural and effective communication.

Keywords: Gesture, Posture and Facial Expressions, Intention Recognition, Human-Robot Collaboration
Abstract: Mobile manipulators - robots with a moving base and an arm for grasping objects - are becoming more common in human-populated environments, such as hospitals, warehouses, and even homes. Yet most mobile manipulators lack clear ways to communicate intent to human interlocutors in a continuous, socially acceptable, and easy-to-interpret way. One possible solution for improving mobile manipulator communication is the addition of expressive eyes. This paper presents the design and evaluation of a custom expressive LED eye module for mobile manipulators, which can display both gaze and emotional expressions. Our evaluation study (N = 32) involved a mock teamwork task alongside a Hello Robot Stretch RE2 mobile manipulator with the custom LED eye module. The results showed that both gaze and emotional expressions supported better participant performance in the task and more feelings of social closeness. Emotional eye expressions also yielded higher ratings of robot social warmth and competence. This work can inform mobile manipulator design for smoother integration into human-populated spaces.

Keywords: Social HRI, Gesture, Posture and Facial Expressions, Emotional Robotics
Abstract: Equipping humanoid robots with the capability to understand emotional states of human interactants and express emotions appropriately according to situations is essential for affective human-robot interaction. However, enabling current vision-aware multimodal emotion recognition models for affective human-robot interaction in the real-world raises embodiment challenges: addressing the environmental noise issue and meeting real-time requirements. First, in multiparty conversation scenarios, the noises inherited in the visual observation of the robot, which may come from either 1)distracting objects in the scene or 2) inactive speakers appearing in the field of view of the robot, hinder the models from extracting emotional cues from vision inputs. Secondly, realtime response, a desired feature for an interactive system, is also challenging to achieve. To tackle both challenges, we introduce an affective human-robot interaction system called UGotMe designed specifically for multiparty conversations. Two denoising strategies are proposed and incorporated into the system to solve the first issue. Specifically, to filter out distracting objects in the scene, we propose extracting face images of the speakers from the raw images and introduce a customized active face extraction strategy to rule out inactive speakers. As for the second issue, we employ efficient data transmission from the robot to the local server to improve realtime response capability. We deploy UGotMe on a human robot named Ameca to validate its real-time inference capabilities in practical scenarios. Videos demonstrating real-world deployment are available at https://lipzh5.github.io/HumanoidVLE/

Keywords: Soft Robot Applications, Soft Robot Materials and Design, Robotics and Automation in Life Sciences
Abstract: Automating small-scale experiments in materials science presents challenges due to the heterogeneous nature of experimental setups. This study introduces the SCU-Hand (Soft Conical Universal Robot Hand), a novel end-effector designed to automate the task of scooping powdered samples from various container sizes using a robotic arm. The SCU-Hand employs a flexible, conical structure that adapts to different container geometries through deformation, maintaining consistent contact without complex force sensing or machine learning-based control methods. Its reconfigurable mechanism allows for size adjustment, enabling efficient scooping from diverse container types. By combining soft robotics principles with a sheet-morphing design, our end-effector achieves high flexibility while retaining the necessary rigidity for effective powder manipulation. We detail the design principles, fabrication process, and experimental validation of the SCU-Hand. Experimental validation showed that the scooping capacity is about 20% higher than that of a commercial tool, with a scooping performance of more than 95% for containers of sizes between 67 mm to 110 mm. This research contributes to laboratory automation by offering a cost-effective, easily implementable solution for automating tasks such as materials synthesis and characterization processes.

Keywords: Compliant Joints and Mechanisms, Multifingered Hands
Abstract: Soft robotic hands compensate for uncertainty in perception and actuation by leveraging passive deformation in their intrinsically compliant hardware, facilitating robust and dexterous interactions with their environment. The ability to adjust the level of compliance during operation has the potential to further improve the performance of these hands by enabling novel interaction strategies. However, achieving variable stiffness mechanically typically requires significant engineering complexity, making these systems difficult to manufacture, prone to error, and expensive. We present a novel, very simple mechanism for achieving variable stiffness. This mechanism employs tendon-driven antagonistic actuation, with Bowden cables connecting elastic elements to servomotors. It supports compact actuator designs, while the Bowden cables facilitate flexible component placement within a robotic system. Following our approach, variable stiffness actuators can be easily manufactured at low-cost from readily available materials. Despite its simplicity, we demonstrate that our mechanism provides consistent and precise control over stiffness levels and contact torques, showcasing its potential for a broad range of applications in soft robotic systems.

Keywords: Soft Robot Applications, Physical Human-Robot Interaction, Grippers and Other End-Effectors
Abstract: This paper presents a novel design concept of a pair of soft gripper hands that can establish a secure connection between them for bearing a large load with a low air pressure. The design was inspired by dovetail joints in carpentry that enable a tight, strong connection between two pieces of wood. We propose to mimic the dovetail joint mechanism by using soft robotic fingers that interlace to each other for secure connection. The work was motivated by the need for securing a connection between two soft robotic arms for holding a balance-impaired older adult in case of losing balance. First, the design principle of dovetail-like secure soft finger connection is presented, and its potential application to a portable fall prevention system is described. Details of the dovetail soft finger design, its rapid inflation method, and other implementation issues are then discussed. Through experiments of a proof-of-concept prototype, it is validated that the dovetail soft fingers can bear at least 18 kg of load with only 52 kPa of air chamber pressure filled in 250 ms of charging time. At the end, the proposed method is compared to alternative methods using a Pugh chart.

Keywords: Soft Robot Applications, Grippers and Other End-Effectors, Grasping
Abstract: For objects with complex topological and geometrical features, stochastic topological grasping can be executed without the necessity for feedback or precise planning. However, this grasping method has two significant limitations. First, the technique’s effectiveness is reduced when interacting with topologically and geometrically simple objects like spheres, cubes, and cylinders, due to the inherent variability in grasping patterns. Additionally, the method’s low stiffness restricts its ability to securely handling heavier objects. To address these challenges, this paper proposes an entanglement soft robotic gripper with variable stiffness and two transformed grasping modes (entanglement and clamping modes). The gripper contains three filaments, which can enhance the stiffness through the mechanism of layer jamming. Furthermore, the entanglement mode and the clamping mode, can be transformed by adjusting the working length of the filaments. The grasping performance comparison with and without variable stiffness was carried out, and the results indicated that the implementation of variable stiffness led to a 149 % increase in payload weight. Through experimental validation, we successfully employed the gripper in variable stiffness and transformed modes to grasp items with various shapes and weights. Demonstration of grasping heavier objects and transforming between two grasping modes were also conducted to showcase the adaptability and versatility of the gripper.

Keywords: In-Hand Manipulation, Modeling, Control, and Learning for Soft Robots, Dexterous Manipulation
Abstract: Dynamic in-hand manipulation remains a challenging task for soft robotic systems that have demonstrated advantages in safe compliant interactions but struggle with high-speed dynamic tasks. In this work, we present SoftSpin, a system for dynamic spinning using a soft and compliant robotic hand. Unlike previous works that rely on quasi-static actions and precise object models, the proposed system learns to spin a pen through trial-and-error using only real-world data without requiring explicit prior knowledge of the pen’s physical attributes. With self-labeled trials sampled from the real world, the system discovers the set of pen grasping and spinning primitive parameters that enables a soft hand to spin the pen robustly and reliably. After 130 sampled actions, SoftSpin achieves 100 % success rate across three pens with different weights and weight distributions, demonstrating the system’s generalizability and robustness to changes in object properties. The results highlight the potential for soft robotic end-effectors to perform dynamic tasks including rapid in-hand manipulation. We also demonstrate that SoftSpin generalizes to spinning tools with different shapes and weights such as a brush and a screwdriver which we spin with 10/10 and 5/10 success rates respectively. Videos, data, and code are available at https://soft-spin.github.io

Keywords: Soft Robot Applications, Modeling, Control, and Learning for Soft Robots, Grippers and Other End-Effectors
Abstract: Soft robots (SR) are a class of continuum robots that enable safe human interaction with task versatility beyond rigid robots. This has resulted in their rapid adoption in a number of applications that require manipulation of delicate and irregular objects. Despite their advantages, SR grippers typically require case-specific experimental characterization for shape and gripper retention force estimation. This letter presents a kinetostatic modeling approach based on strain energy minimization subject to mechanics and geometric constraints for shape estimation of SR grippers with external tendon routing (ETR), including those with composite structures. Additionally, Castigliano's First Theorem is used to estimate the retention force of the gripper. These models are evaluated across four different ETR SR grippers. The mechanics model predicted the fingertip position and orientation with an accuracy of 1.06±0.62 mm (1.79%±1.05% of length) and 3.58°±2.82° with respect to tendon force and 0.72±0.45 mm (1.22%±0.76% of length) and 2.86°±2.11° with respect to tendon retraction. The retention force of the gripper was predicted with an average error of 0.20±0.12 N.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, Wearable Robotics
Abstract: Recently, egocentric action anticipation for wearable robotics cameras has gained considerable attention due to its capability to analyze nouns and verbs from a first-person view. However, this field encounters challenges due to various uncertainties, such as action-irrelevant information and semantically fused representations of verbs and nouns. To overcome these issues, we introduce Ego-A3, designed to improve the robustness and reliability of egocentric action anticipation systems. Ego-A3 adaptively extracts action-relevant data to efficiently utilize additional information beyond visual data. Additionally, Ego-A3 produces effective disentangled representations for verbs and nouns by employing learnable verb and noun queries. Experiments on the EpicKitchens-100 and EGTEA Gaze+ datasets demonstrate that Ego-A3 outperforms existing methods in top-1 accuracy and mean top-5 recall. Our code is publicly available at https://github.com/alsgur0720/egocentric_anticipation.

Keywords: Haptics and Haptic Interfaces, Motion Control, Robust/Adaptive Control
Abstract: This paper proposes a new variable gain sliding mode filter augmented by variable windowing for achieving smooth and reactive response over a broad range of input frequencies. The proposed filter can be seen as a synergistic combination of Kikuuwe et al.’s [1] sliding mode filter with varying gain and sliding surfaces and a novel varying-length moving-window algorithm. In all schemes, the estimated input speed is employed for rendering the filter parameters between low and high settings. The discrete-time algorithm of the proposed filter does not suffer from chattering due to implicit (backward) Euler method. The effectiveness of the proposed filter in achieving better trade-off between noise attenuation and signal preservation is validated in both simulation and experimental scenarios by using the velocity signal obtained by differentiation of quantized position data.

Keywords: Telerobotics and Teleoperation, Physically Assistive Devices
Abstract: The prevailing and most effective approach to teleoperate a robotic arm involves a direct position-to-position mapping, imposing robotic end-effector movements that mirrors those of the user. However, due to this one-to-one mapping, the robot's motions are limited by the user's capability, particularly in translation. Drawing inspiration from head pointers utilized in the 1980s, originally designed to enable drawing with limited head motions for tetraplegic individuals, we proposed a "virtual wand" mapping which could be used by participants with reduced mobility. This mapping employs a virtual rigid linkage between the hand and the robot's end-effector. With this approach, rotations produce amplified translations through a lever arm, creating a "rotation-to-position" coupling and expanding the translation workspace at the expense of a reduced rotation space.

In this study, we compare the virtual wand approach to the one-to-one position mapping through the realization of 6-DoF reaching tasks. Results indicate that the two different mappings perform comparably well, are equally well-received by users, and exhibit similar motor control behaviors. Nevertheless, the virtual wand mapping is anticipated to outperform in tasks characterized by large translations and minimal effector rotations, whereas direct mapping is expected to demonstrate advantages in large rotations with minimal translations. These results pave the way for new interactions and interfaces, particularly in disability assistance utilizing residual body movements (instead of hands) as control input. Leveraging body parts with substantial rotations could enable the accomplishment of tasks previously deemed infeasible with standard direct coupling interfaces.

Keywords: Telerobotics and Teleoperation, Haptics and Haptic Interfaces, Humanoid and Bipedal Locomotion
Abstract: Teleoperated humanoid robot systems have made substantial advancements in recent years, offering a physical avatar that harnesses human skills and decision-making while safeguarding users from hazardous environments. However, current telelocomotion interfaces often fail to accurately represent the robot's environment, limiting the user’s ability to effectively navigate the robot through unstructured terrain. This paper presents an initial telelocomotion framework that integrates the ForceBot locomotion interface with the small-sized humanoid robot, HECTOR V2. The framework utilizes ForceBot to simulate walking motion and estimate the user’s Center of Mass (CoM) trajectory, which serves as a tracking reference for the robot. On the robot side, a model predictive control (MPC) approach, based on a reduced-order single rigid body model, is employed to track the user’s scaled trajectory. We present experimental results on ForceBot’s CoM estimation and the robot’s tracking performance, demonstrating the feasibility of this approach.

Keywords: Telerobotics and Teleoperation, Haptics and Haptic Interfaces, Physical Human-Robot Interaction
Abstract: Variable stiffness of a remote robot is crucial for a teleoperation system to deal with challenging tasks. External stiffness command interfaces have emerged as a promising solution to regulating the remote robot stiffness because of the benefits of their accuracy, ergonomics, and avoidance of the "coupling effect" that usually exists in muscle activity-based stiffness interfaces. However, the use of an external stiffness command interface requires good coordination between two limbs of an operator, which take care of the teleoperation task and the stiffness regulation task, respectively, at the same time, which is demanding for novice operators in dynamic situations necessitating agile and timely stiffness adjustments. In this paper, a new concept of Stiffness Regulation Co-pilot was proposed to facilitate the use of these interfaces. A co-pilot is a virtual agent that consists of a Stiffness Regulation Policy, which infers a reasonable stiffness regulation action from the task performance, and a feedback modality, which conveys the suggested stiffness regulation action to the operator. A preliminary user study was conducted to evaluate the efficacy of the co-pilot and the effect of different modalities of the co-pilot. The results showed that the cutaneous feedback or combined with another modality can potentially improve the task performance of the system and reduce the cognitive load of the operator compared to a teleoperation system without using the co-pilot.

Keywords: Motion Control, Human-Robot Collaboration, Grippers and Other End-Effectors
Abstract: This paper proposes an adaptive neural network synchronous tracking control strategy that can be suitable for event-triggered mechanism in response to the modeling uncertainties and communication delays in bilateral teleoperation systems. Through introducing the event-triggered mechanism with the aim of reducing the network communication frequency in teleoperation system, the master and slave robots communicate with each other only when the triggering conditions are fulfilled, which enhances the efficiency of the network communication. This control strategy can guarantee the exponential convergence of the position synchronization tracking error of the master-slave robot end-effector. Moreover, the event-triggered conditions do not require any empirical design, but can be derived inversely with the aid of the Lyapunov stability theory. And the triggering time interval between two neighboring events is verified to be non-zero. It is demonstrated by utilizing the Lyapunov principle that the presented adaptive neural network control strategy ensures the final asymptotic convergence and exponential convergence of the position synchronization tracking error for master-slave robots under the designed event-triggered mechanisms. Eventually, the feasibility and effectiveness of the developed control strategy are validated by comparative cases.

Keywords: Bimanual Manipulation, Dexterous Manipulation, Learning from Demonstration
Abstract: Aiming to replicate human-like dexterity, perceptual experiences, and motion patterns, we explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Two significant challenges exist: the lack of an affordable and accessible teleoperation system suitable for a dual-arm setup with multifingered hands, and the scarcity of multifingered hand hardware equipped with touch sensing. To tackle the first challenge, we develop HATO, a low-cost hands-arms teleoperation system that leverages off-the-shelf electronics, complemented with a software suite that enables efficient data collection; the comprehensive software suite also supports multimodal data processing, scalable policy learning, and smooth policy deployment. To tackle the latter challenge, we introduce a novel hardware adaptation by repurposing two prosthetic hands equipped with touch sensors for research. Using visuotactile data collected from our system, we learn skills to complete long-horizon, high-precision tasks which are difficult to achieve without multifingered dexterity and touch feedback. Furthermore, we empirically investigate the effects of dataset size, sensing modality, and visual input preprocessing on policy learning. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data. Videos, code, and datasets can be found on: https://toruowo.github.io/hato

Keywords: AI-Based Methods, Bimanual Manipulation, Imitation Learning
Abstract: When performing tasks like laundry, humans naturally coordinate both hands to manipulate objects and anticipate how their actions will change the state of the clothes. However, achieving such coordination in robotics remains challenging due to the need to model object movement, predict future states, and generate precise bimanual actions. In this work, we address these challenges by infusing the predictive nature of human manipulation strategies into robot imitation learning. Specifically, we disentangle task-related state transitions from agent-specific inverse dynamics modeling to enable effective bimanual coordination. Using a demonstration dataset, we train a diffusion model to predict future states given historical observations, envisioning how the scene evolves. Then, we use an inverse dynamics model to compute robot actions that achieve the predicted states. Our key insight is that modeling object movement can help learning policies for bimanual coordination manipulation tasks. Evaluating our framework across diverse simulation and real-world manipulation setups, including multimodal goal configurations, bimanual manipulation, deformable objects, and multi-object setups, we find that it consistently outperforms state-of-the-art state-to-action mapping policies. Our method demonstrates a remarkable capacity to navigate multimodal goal configurations and action distributions, maintain stability across different control modes, and synthesize a broader range of behaviors than those present in the demonstration dataset.

Keywords: Perception for Grasping and Manipulation, Deep Learning in Grasping and Manipulation, Data Sets for Robot Learning
Abstract: Cloth folding is a complex task due to the inevitable self-occlusions of clothes, their complicated dynamics, and the disparate materials, geometries, and textures that garments can have. In this work, we learn folding actions conditioned on text commands. Translating high-level, abstract instructions into precise robotic actions requires sophisticated language understanding and manipulation capabilities. To do that, we leverage a pre-trained vision-language model and repurpose it to predict manipulation actions. Our model, BiFold, can take context into account and achieves stateof-the-art performance on an existing language-conditioned folding benchmark. To address the lack of annotated bimanual folding data, we introduce a novel dataset with automatically parsed actions and language-aligned instructions, enabling better learning of text-conditioned manipulation. BiFold attains the best performance on our dataset and demonstrates strong generalization to new instructions, garments, and environments.

Keywords: Dual Arm Manipulation, Imitation Learning, Visual Servoing
Abstract: We introduce One-Shot Dual-Arm Imitation Learning (ODIL), which enables dual-arm robots to learn precise and coordinated everyday tasks from just a single demonstration of the task. ODIL uses a new three-stage visual servoing (3-VS) method for precise alignment between the end-effector and target object, after which replay of the demonstration trajectory is sufficient to perform the task. This is achieved without requiring prior task or object knowledge, or additional data collection and training following the single demonstration. Furthermore, we propose a new dual-arm coordination paradigm for learning dual-arm tasks from a single demonstration. ODIL was tested on a real-world dual-arm robot, demonstrating state-of-the-art performance across six precise and coordinated tasks in both 4-DoF and 6-DoF settings, and showing robustness in the presence of distractor objects and partial occlusions. Videos are available at https://www.robot-learning.uk/one-shot-dual-arm.

Keywords: Dual Arm Manipulation, Mobile Manipulation, Grasping
Abstract: Picking diverse objects in the real world is a fundamental robotics skill. However, many objects in such settings are bulky, heavy, or irregularly shaped, making them ungraspable by conventional end effectors like suction grippers and parallel jaw grippers (PJGs). In this paper, we expand the range of pickable items without hardware modifications using bimanual nonprehensile manipulation. We focus on a grocery shopping scenario, where a bimanual mobile manipulator equipped with a suction gripper and a PJG is tasked with re- trieving ungraspable items from tightly packed grocery shelves. From visual observations, our method first identifies optimal grasp points based on force closure and friction constraints. If the grasp points are occluded, a series of nonprehensile nudging motions are performed to clear the obstruction. A bimanual grasp utilizing contacts on the side of the end effectors is then executed to grasp the target item. In our replica grocery store, we achieved a 90% success rate over 102 trials in uncluttered scenes, and a 66% success rate over 45 trials in cluttered scenes. We also deployed our system to a real-world grocery store and successfully picked previously unseen items. Our results highlight the potential of bimanual nonprehensile manipulation for in-the-wild robotic picking tasks. A video summarizing this work can be found at youtu.be/g0hOrDuK8jM

Keywords: Bimanual Manipulation, Grasping, Dexterous Manipulation
Abstract: Humans naturally perform bimanual skills to handle large and heavy objects. To enhance a robot's object manipulation capabilities, generating effective bimanual grasp poses is essential. Nevertheless, bimanual grasp synthesis for dexterous hand manipulators remains underexplored. To bridge this gap, we propose the BimanGrasp algorithm for synthesizing bimanual grasps on 3D objects. The BimanGrasp algorithm generates grasp poses by optimizing an energy function that considers grasp stability and feasibility. Furthermore, the quality of the synthesized grasps is verified using the Isaac Gym physics simulation engine. These verified grasp poses form the BimanGrasp-Dataset, which is the first synthesized bimanual dexterous hand grasp pose dataset to our knowledge. The dataset comprises over 150k verified grasps on 900 objects, facilitating the synthesis of bimanual grasps through a data-driven approach. Last, we propose a diffusion model (BimanGrasp-DDPM) trained on the BimanGrasp-Dataset. This model achieved a grasp synthesis success rate of 69.87% and significant acceleration in computational speed compared to BimanGrasp algorithm.

Keywords: Grasping, Perception for Grasping and Manipulation, Cyborgs
Abstract: Achieving a real-time precise grasp of a specified target object in densely cluttered environments is an essential capability for autonomous robot operation. Recently, considerable investigations on planar and spatial grasp have been carried out, and significant results have been obtained. However,these point cloud-based grasp prediction methods often fail to ensure that the generated grasp configurations meet the precise requirements of the task. Additionally, some of the existing grasp pipelines are too time-consuming to meet the demand for real-time robot response. In more challenging cluttered scenes,the quality of pose and gripper jaw opening estimation in highdimensional space requires further improvement. Therefore,this paper introduces a data- and model-independent and efficient method to generate 7-DoF grasp configurations for arbitrary target objects from single-view point cloud data in dense cluttered scenes. In addition, this paper proposes a grasp framework that generates the grasp configuration for the target object while reducing the time consumed during the grasp process, to enable robots to efficiently grasp target objects for designated tasks. The grasp pipeline focuses on guided regions via target detection and rapidly adjusts grasp configurations through multi-region point cloud distribution perception. Extensive real-world robot experiments have demonstrated the effectiveness of the proposed method in grasping target objects in cluttered scenes, achieving higher success rates and reduced runtime compared to baseline methods.The realized code and video are available at https://github.com/L-tj/7DGCG.

Keywords: Grasping
Abstract: In general, humans would grasp an object differently for different tasks, e.g., ``grasping the handle of a knife to cut'' vs. ``grasping the blade to hand over''. In the field of robotic grasp pose detection research, some existing works consider this task-oriented grasping and made some progress, but they are generally constrained by low-DoF gripper type or non-cluttered setting, which is not applicable for human assistance in real life. With an aim to get more general and practical grasp models, in this paper, we investigate a new problem named Task-Oriented 6-DoF Grasp Pose Detection in Clutters (TO6DGC), which extends the task-oriented problem to a more general 6-DOF Grasp Pose Detection in Cluttered (multi-object) scenario. To this end, we construct a large-scale 6-DoF task-oriented grasping dataset, 6-DoF Task Grasp (6DTG), which features 4391 cluttered scenes with over 2 million 6-DoF grasp poses. Each grasp is annotated with a specific task, involving 6 tasks and 198 objects in total. Moreover, we propose One-Stage TaskGrasp (OSTG), a strong baseline to address the TO6DGC problem. Our OSTG adopts a task-oriented point selection strategy to detect where to grasp, and a task-oriented grasp generation module to decide how to grasp given a specific task. To evaluate the effectiveness of OSTG, extensive experiments are conducted on 6DTG. The results show that our method outperforms various baselines on multiple metrics. Real robot experiments also verify that our OSTG has a better perception of the task-oriented grasp points and 6-DoF grasp poses.

Keywords: Grasping, Manipulation Planning, Perception for Grasping and Manipulation
Abstract: Grasping has been a long-standing challenge in facilitating the final interface between a robot and the environment. As environments and tasks become complicated, the need to embed higher intelligence to infer from the surroundings and act on them has become necessary. Although most methods utilize techniques to estimate grasp pose by treating the problem via pure sampling-based approaches in the six-degree-of-freedom space or as a learning problem, they usually fail in real-life settings owing to poor generalization across domains. In addition, the time taken to generate the grasp plan and the lack of repeatability, owing to sampling inefficiency and the probabilistic nature of existing grasp planning approaches, severely limits their application in real-world tasks. This paper presents a lightweight analytical approach towards robotic grasp planning, particularly antipodal grasps, with little to no sampling in the six-degree-of-freedom space. The proposed grasp planning algorithm is formulated as an optimization problem towards estimating grasp points on the object surface instead of directly estimating the end-effector pose. To this extent, a soft-region-growing algorithm is presented for effective plane segmentation, even in the case of curved surfaces. An optimization-based quality metric is then used for evaluation of grasp points to ensure indirect force closure. The proposed grasp framework is compared with existing state-of-the-art grasp planning approach Grasp pose detection (GPD) as baseline over multiple simulated objects. The effectiveness of the proposed approach in comparison to GPD is also evaluated in real-world setting using image and point-cloud data, with the planned grasps being executed using a ROBOTIQ gripper and UR5 manipulator. The proposed approach shows better performance in terms of higher probability for force closure with a complete repeatability.

Keywords: Grasping, Grippers and Other End-Effectors, Manipulation Planning
Abstract: The use of machine learning to investigate grasp affordances has received extensive attention over the past several decades. The existing literature provides a robust basis to build upon, though a number of aspects may be improved. Results commonly work in terms of grasp configuration, with little consideration for the manner in which the grasp may be (re-)produced, from a reachability and trajectory planning perspective. We propose a different perspective on grasp affordance learning, explicitly accounting for grasp synthesis; that is, the manner in which manipulator kinematics are used to allow materialization of grasps. The approach allows to explicitly map the grasp policy space in terms of generated grasp types and associated grasp quality. Results of application to a range of objects illustrate merit of the method and highlight the manner in which it may promote a greater degree of explainability for otherwise intransparent reinforcement processes.

Keywords: Grasping
Abstract: Transparent object grasping remains a persistent challenge in robotics, largely due to the difficulty of acquiring precise 3D information. Conventional optical 3D sensors struggle to capture transparent objects, and machine learning methods are often hindered by their reliance on high-quality datasets. Leveraging NeRF’s capability for continuous spatial opacity modeling, our proposed architecture integrates a NeRF-based approach for reconstructing the 3D information of transparent objects. Despite this, certain portions of the reconstructed 3D information may remain incomplete. To address these deficiencies, we introduce a shape-prior-driven completion mechanism, further refined by a geometric pose estimation method we have developed. This allows us to obtain a complete and reliable 3D information of transparent objects. Utilizing this refined data, we perform scene-level grasp prediction and deploy the results in real-world robotic systems. Experimental validation demonstrates the efficacy of our architecture, showcasing its capability to reliably capture 3D information of various transparent objects in cluttered scenes, and correspondingly, achieve high-quality, stable, and executable grasp predictions.

Keywords: Deep Learning for Visual Perception, Data Sets for Robotic Vision, RGB-D Perception
Abstract: Object grasping is a fundamental challenge in robotics and computer vision, critical for advancing robotic manipulation capabilities. Deformable objects, like fabrics and cloths, pose additional challenges due to their non-rigid nature. In this work, we introduce CeDiRNet-3DoF, a deep-learning model for grasp point detection, with a particular focus on cloth objects. CeDiRNet-3DoF employs center direction regression alongside a localization network, attaining first place in the perception task of ICRA 2023's Cloth Manipulation Challenge. Recognizing the lack of standardized benchmarks in the literature that hinder effective method comparison, we present the ViCoS Towel Dataset. This extensive benchmark dataset comprises 8,000 real and 12,000 synthetic images, serving as a robust resource for training and evaluating contemporary data-driven deep-learning approaches. Extensive evaluation revealed CeDiRNet-3DoF's robustness in real-world performance, outperforming state-of-the-art methods, including the latest transformer-based models. Our work bridges a crucial gap, offering a robust solution and benchmark for cloth grasping in computer vision and robotics.

Keywords: Localization, Probability and Statistical Methods, Robot Safety, Autonomous Vehicle Navigation
Abstract: Deriving safe bounds on particle filter estimate is a research problem that, if solved, could greatly benefit robots in life-critical applications, a field that is facing increasing interest as more robots are being deployed near humans. In response, this paper introduces a new fault detector and derives a performance measure for particle filter: integrity risk. Integrity risk is defined as the probability of having large estimate errors without triggering an alarm, all while considering measurement faults, unknown deterministic errors that cannot be modeled via normal white noise. In this work, the faults come in the form of incorrectly associated features when using the local nearest neighbors. Simulations and experiments assess the efficiency of the introduced safety metric. The results show that safety improves as map density increases as long as the number of particles is sufficient to shape the error distribution and the landmarks are well separated. Also, the results indicate that, when landmarks are poorly separated, particle filter is safer than Kalman filter, whereas, when landmarks are well separated, particle filter is often, but not always, safer than Kalman filter.

Keywords: Localization, Multi-Robot Systems, Wheeled Robots
Abstract: In this paper, we apply lighthouse localization, originally designed for virtual reality motion tracking, to positioning and localization of indoor robots. We first present a lighthouse decoding and tracking algorithm on a low-power wireless microcontroller with hardware implemented in a cm-scale form factor. One-time scene solving is performed on a computer using a variety of standard computer vision tech-niques. Three different robotic localization scenarios are analyzed in this work. The first is a planar scene with a single lighthouse with a four-point pre-calibration. The second is a planar scene with two light-houses that self calibrates with either multiple robots in the experiment or a single robot in motion. The third extends to a 3D scene with two lighthouses and a self-calibration algorithm. The absolute accuracy, measured against a camera-based tracking system, was found to be 7.25 mm RMS for the 2D case and 11.2 mm RMS for the 3D case, respectively. This demonstrates the viability of lighthouse tracking both for small-scale robotics and as an inexpensive and compact alternative to camera-based setups.

Keywords: Localization, Deep Learning for Visual Perception
Abstract: Event cameras are bioinspired sensors with some notable features, including high dynamic range and low latency, which makes them exceptionally suitable for perception in challenging scenarios such as high-speed motion and extreme lighting conditions. In this paper, we explore their potential for localization within pre-existing LiDAR maps, a critical task for applications that require precise navigation and mobile manipulation. Our framework follows a paradigm based on the refinement of an initial pose. Specifically, we first project LiDAR points into 2D space based on a rough initial pose to obtain depth maps, and then employ an optical flow estimation network to align events with LiDAR points in 2D space, followed by camera pose estimation using a PnP solver. To enhance geometric consistency between these two inherently different modalities, we develop a novel frame-based event representation that improves structural clarity. Additionally, given the varying degrees of bias observed in the ground truth poses, we design a module that predicts an auxiliary variable as a regularization term to mitigate the impact of this bias on network convergence. Experimental results on several public datasets demonstrate the effectiveness of our proposed method. To facilitate future research, both the code and the pre-trained models are made available online.

Keywords: Localization, Deep Learning for Visual Perception, Recognition
Abstract: In recent years, robust matching methods using deep learning-based approaches have been actively studied and improved in computer vision tasks. However, there remains a persistent demand for both robust and fast matching techniques. To address this, we propose a novel Mamba-based local feature matching approach, called MambaGlue, where Mamba is an emerging state-of-the-art architecture rapidly gaining recognition for its superior speed in both training and inference, and promising performance compared with Transformer architectures. In particular, we propose two modules: a) MambaAttention mixer to simultaneously and selectively understand the local and global context through the Mamba-based self-attention structure and b) deep confidence score regressor, which is a multi-layer perceptron (MLP)-based architecture that evaluates a score indicating how confidently matching predictions correspond to the ground-truth correspondences. Consequently, our MambaGlue achieves a balance between robustness and efficiency in real-world applications. As verified on various public datasets, we demonstrate that our MambaGlue yields a substantial performance improvement over baseline approaches while maintaining fast inference speed. Our code will be available on https://github.com/url-kaist/MambaGlue.

Keywords: Localization, Range Sensing, Autonomous Vehicle Navigation
Abstract: While UWB-based methods can achieve high localization accuracy in small-scale areas, their accuracy and reliability are significantly challenged in large-scale environments. In this paper, we propose a learning-based framework for Ultra-Wideband (UWB) based localization in such complex large-scale environments, named ULOC. First, anchors are deployed in the environment without knowledge of their actual position. Then, UWB observations are collected when the vehicle travels in the environment. At the same time, map-consistent pose estimates are developed from registering (onboard self-localization) data with the prior map to provide the training labels. We then propose a recurrent neural network (RNN) based on MAMBA that learns the ranging patterns of UWBs over a complex large-scale environment. The experiment demonstrates that our solution can ensure high localization accuracy on a large scale compared to the state-of-the-art. We release our source code to benefit the community at https://github.com/brytsknguyen/uloc.

Keywords: Localization
Abstract: We present a deterministic approach for the localization of an Unmanned Aerial Vehicle (UAV) in a known indoor environment by using only a few downward distance measurements and the corresponding odometries between measurements. For each distance measurement and odometry, we look at the preimage of that distance measurement under the downwards distance function combined with the corresponding odometry where the motion between every two measurements has four degrees of freedom: three of translation and one of azimuth change. The intersection of these preimages yields the set of all possible locations for the UAV. In this work, we present an efficient method for approximating that intersection of preimages. We perform a spatial subdivision search, which splits only voxels containing that intersection. We present a novel technique, based on geometric insights, for correctly evaluating whether a voxel indeed contains a true localization. This technique is also robust under different kinds of errors that might occur. Our method is guaranteed to contain the ground truth location, and its runtime complexity is output sensitive, in the Hausdorff dimension and measure of the resulting intersection of preimages. We demonstrate the effectiveness of this method in various indoor scenarios, showing that it can be used to significantly decrease the uncertainty of localization when solving the kidnapped robot problem in simulation and on a physical drone. Our method can be performed in real-time. Furthermore, our method requires only a map of the environment, odometry and ToF sensors, which is advantageous in terms of cost, privacy and transmission bandwidth. Our open-source software and supplementary materials are available at https://github.com/TAU-CGL/uav-fdml-public.

Keywords: Software Tools for Robot Programming, Software Tools for Benchmarking and Reproducibility
Abstract: Roboticists compare robot motions for tasks such as parameter tuning, troubleshooting, and deciding between possible motions. However, most existing visualization tools are designed for individual motions and lack the features necessary to facilitate robot motion comparison. In this paper, we follow a rigorous design process to create Motion Comparator, a web-based tool that facilitates the comprehension, comparison, and communication of robot motions. Our design process identified roboticists' needs, articulated design challenges, and provided corresponding strategies. Motion Comparator includes several key features such as multi-view coordination, quaternion visualization, time warping, and comparative designs. To demonstrate the applications of Motion Comparator, we discuss four case studies in which our tool is used for motion selection, troubleshooting, parameter tuning, and motion review.

Keywords: Methods and Tools for Robot System Design, Evolutionary Robotics
Abstract: Robot design has traditionally been costly and labor-intensive. Despite advancements in automated processes, it remains challenging to navigate a vast design space while producing physically manufacturable robots. We introduce Text2Robot, a framework that converts user text specifications and performance preferences into physical quadrupedal robots. Within minutes, Text2Robot can use text-to-3D models to provide strong initializations of diverse morphologies. Within a day, our geometric processing algorithms and body-control co-optimization produce a walking robot by explicitly considering real-world electronics and manufacturability. Text2Robot enables rapid prototyping and opens new opportunities for robot design with generative models.

Keywords: Deep Learning Methods, Engineering for Robotic Systems, Software Tools for Robot Programming
Abstract: CAD models are widely used in industry and are essential for robotic automation processes. However, these models are rarely considered in novel AI-based approaches, such as the automatic synthesis of robot programs, as there are no readily available methods that would allow CAD models to be incorporated for the analysis, interpretation, or extraction of information. To address these limitations, we propose QueryCAD, the first system designed for CAD question answering, enabling the extraction of precise information from CAD models using natural language queries. QueryCAD incorporates SegCAD, an open-vocabulary instance segmentation model we developed to identify and select specific parts of the CAD model based on part descriptions. We further propose a CAD question answering benchmark to evaluate QueryCAD and establish a foundation for future research. Lastly, we integrate QueryCAD within an automatic robot program synthesis framework, validating its ability to enhance deep-learning solutions for robotics by enabling them to process CAD models.

Keywords: Software Tools for Robot Programming, Software, Middleware and Programming Environments, Multi-Robot Systems
Abstract: Heterogeneous robots can work together to accomplish a variety of complex tasks and have shown great potential in many fields. There are many efforts to make robot task orchestration more efficient. However, current methods still have some limitations, including the lack of a high-level abstraction for programming method and fault handling mechanism. In this paper, we design a state machine-based, fault-tolerant framework for heterogeneous multi-robot collaboration named HeRo, to effectively support the development of heterogeneous multi-robot systems. HeRo has three key techniques: (1) a state machine-based programming language to flexibly model robot behaviors and tasks; (2) a state synchronization mechanism to achieve information exchange and maintain the consistency among heterogeneous robots in distributed environments; (3) a fault detection and recovery mechanism to monitor the system's runtime states and use Large Language Model (LLM) combined with Planning Domain Definition Language (PDDL) to enable automated recovery. We evaluate the effectiveness and fault recovery capability of the framework by setting up manufacturing task and fault scenarios with varying difficulty in the ARIAC simulation environment, achieving a 100% task completion rate, with low system overhead and flexible scalability.

Keywords: Methods and Tools for Robot System Design, Engineering for Robotic Systems, Optimization and Optimal Control
Abstract: It is challenging to find optimum kinematic designs for non-standard robotic manipulators, e.g., medical, nuclear, and space manipulators, which are demanded to adapt to arbitrary complex tasks in constraints. Such design optimization can be modelled as a multi-dimensional non-convex optimization problem with nonlinear constrained conditions. However, it is non-trivial to ensure the essential reachability condition, i.e., the existence of continuous trajectories between demand positions for serial articulated manipulators, given complex spatial constraints, like obstacles and boundaries. Traditional solutions integrate standard motion planning or inverse kinematics algorithms within a kinematic-design optimization process, resulting in significant demand for time and computing resources. To accelerate design optimization at improved efficiency, we design a novel robust design framework built on a new kinematic design synthesis, which allows for simultaneously optimizing dimension and topology of a serial manipulator's kinematics for arbitrary tasks in constrained environments, using a generalised parametric kinematic model. Significantly, in contrast to standard solutions, we develop a novel computationally effective reachability verification method, which rapidly aborts infeasible motions by exploiting efficient collision checks, based on the Rapidly-exploring Random Tree (RRT) algorithm. The effectiveness of the proposed design framework is verified and evaluated by comparing to baseline benchmarks. Results demonstrate the novel design framework can accelerate kinematic design optimization by an order of magnitude compared to the current state-of-the-art, and optimise link dimension and joint type simultaneously of serial robots for cluttered environments.

Keywords: Embedded Systems for Robotic and Automation, Engineering for Robotic Systems, Methods and Tools for Robot System Design
Abstract: Connected and automated vehicles and robot swarms hold transformative potential for enhancing safety, efficiency, and sustainability in the transportation and manufacturing sectors. Extensive testing and validation of these technologies is crucial for their deployment in the real world. While simulations are essential for initial testing, they often have limitations in capturing the complex dynamics of real-world interactions. This limitation underscores the importance of small-scale testbeds. These testbeds provide a realistic, cost-effective, and controlled environment for testing and validating algorithms, acting as an essential intermediary between simulation and full-scale experiments. This work serves to facilitate researchers' efforts in identifying existing small-scale testbeds suitable for their experiments and provide insights for those who want to build their own. In addition, it delivers a comprehensive survey of the current landscape of these testbeds. We derive 62 characteristics of testbeds based on the well-known sense-plan-act paradigm and offer an online table comparing 23 small-scale testbeds based on these characteristics. The online table is hosted on our designated public webpage https://bassamlab.github.io/testbeds-survey, and we invite testbed creators and developers to contribute to it. We closely examine nine testbeds in this paper, demonstrating how the derived characteristics can be used to present testbeds. Furthermore, we discuss three ongoing ch

Keywords: Transfer Learning, Force and Tactile Sensing, Representation Learning
Abstract: Tactile perception is essential for human interaction with the environment and is becoming increasingly crucial in robotics. Tactile sensors like the BioTac mimic human fingertips and provide detailed interaction data. Despite its utility in applications like slip detection and object identification, this sensor is now deprecated, making many valuable datasets obsolete. However, recreating similar datasets with newer sensor technologies is both tedious and time-consuming. Therefore, adapting these existing datasets for use with new setups and modalities is crucial. In response, we introduce ACROSS, a novel framework for translating data between tactile sensors by exploiting sensor deformation information. We demonstrate the approach by translating BioTac signals into the DIGIT sensor. Our framework consists of first converting the input signals into 3D deformation meshes. We then transition from the 3D deformation mesh of one sensor to the mesh of another, and finally convert the generated 3D deformation mesh into the corresponding output space. We demonstrate our approach to the most challenging problem of going from a low-dimensional tactile representation to a high-dimensional one. In particular, we transfer the tactile signals of a BioTac sensor to DIGIT tactile images. Our approach enables the continued use of valuable datasets and data exchange between groups with different setups.

Keywords: Force and Tactile Sensing, Perception for Grasping and Manipulation, Perception-Action Coupling
Abstract: Manipulating arbitrary objects in unstructured environments is a significant challenge in robotics, primarily due to difficulties in determining an object's center of mass. This paper introduces U-GRAPH: Uncertainty-Guided Rotational Active Perception with Haptics, a novel framework to enhance the center of mass estimation using active perception. Traditional methods often rely on singular interactions and are limited by the inherent inaccuracies of Force-Torque (F/T) sensors. Our approach circumvents these limitations by integrating a Bayesian Neural Network (BNN) to quantify uncertainty and guide the robotic system through multiple, information-rich interactions via grid search and ActiveNet. We demonstrate the remarkable generalizability and transferability of our method with training on a small dataset with limited variation yet still perform well on unseen complex real-world objects.

Keywords: Force and Tactile Sensing, In-Hand Manipulation, Reinforcement Learning
Abstract: Recent progress in reinforcement learning (RL) and tactile sensing has significantly advanced dexterous manipulation. However, these methods often utilize simplified tactile signals due to the gap between tactile simulation and the real world. We introduce a sensor model for tactile skin that enables zero-shot sim-to-real transfer of ternary shear and binary normal forces. Using this model, we develop an RL policy that leverages sliding contact for dexterous in-hand translation. We conduct extensive real-world experiments to assess how tactile sensing facilitates policy adaptation to various unseen object properties and robot hand orientations. We demonstrate that our 3-axis tactile policies consistently outperform baselines that use only shear forces, only normal forces, or only proprioception. Videos and details available on the project website.

Keywords: Representation Learning, Force and Tactile Sensing, Deep Learning in Grasping and Manipulation
Abstract: Tactile sensors differ greatly in design, making it challenging to develop general-purpose methods for processing tactile feedback. In this paper, we introduce a contrastive self-supervised learning approach that represents tactile feedback across different sensor types. Our method utilizes paired tactile data—where two distinct sensors, in our case Soft Bubbles and GelSlims, grasp the same object in the same configuration—to learn a unified latent representation. Unlike current approaches that focus on reconstruction or task-specific supervision, our method employs contrastive learning to create a latent space that captures shared information between sensors. By treating paired tactile signals as positives and unpaired signals as negatives, we show that our model effectively learns a rich, sensor-agnostic representation. Despite significant differences between Soft Bubble and GelSlim sensors, the learned representation enables strong downstream task performance, including zero-shot and few-shot classification and pose estimation. This work provides a scalable solution for integrating tactile data across diverse sensor modalities, advancing the development of generalizable tactile representations.

Keywords: Perception for Grasping and Manipulation, Force and Tactile Sensing
Abstract: An essential problem in robotic systems that are to be deployed in unstructured environments is the accurate and autonomous perception of the shapes of previously unseen objects. Existing methods for shape estimation or reconstruction have leveraged either visual or tactile interactive exploration techniques, or have relied on comprehensive visual or tactile information acquired in an offline manner. In this work, a novel visuo-tactile interactive perception framework - ViTract is introduced for shape estimation of unseen objects. Our framework estimates the shape of diverse objects robustly using low-dimensional, efficient, and generalizable shape primitives, which are superquadrics.

The probabilistic formulation within our framework takes advantage of the complementary information provided by vision and tactile observations while accounting for associated noise. As part of our framework, we propose a novel modality-specific information gain to select the most informative and reliable exploratory action (using vision/tactile) to obtain iterative visuo/tactile information. Our real-robot experiments demonstrate superior and robust performance compared to state-of-the-art visuo-tactile-based shape estimation techniques.

Keywords: Force and Tactile Sensing, Perception for Grasping and Manipulation, Grippers and Other End-Effectors, Barometric Sensing
Abstract: The adoption of tactile sensors in robotics is hindered by their high cost and fragility. We designed and validated a cost-effective and robust barometric tactile sensor array, whose material cost is below 80 USD. Unlike past work, we do not mold the rubber surface over the barometers but instead keep it as a separate element, leading to a design that is easy to fabricate and repair. Machine learning techniques are applied to enhance the sensor’s localization precision, increasing the effective resolution from 6 mm (the distance between adjacent barometers) to 0.284 mm. To investigate the localization model’s robustness, we utilized an E-TRoll robotic gripper to roll differently shaped prismatic objects across the sensing surface mounted on one finger. Under these uncontrolled settings, we achieved a satisfactory real-time localization resolution of within 2.68 mm. Furthermore, we demonstrate a novel practical application: The E-TRoll mimics a 1-DoF parallel gripper inferring a cube’s orientation relative to the sensor. The range of orientations is split into 4 classes, which a trained CNN-LSTM model can predict with an 86.91% five-fold cross-validated accuracy.

Keywords: Human-Centered Automation, Computer Vision for Automation, Continual Learning
Abstract: Robot person following (RPF) is a crucial capability in human-robot interaction (HRI) applications, allowing a robot to persistently follow a designated person. In practical RPF scenarios, the person can often be occluded by other objects or people. Consequently, it is necessary to re-identify the person when he/she reappears within the robot's field of view. Previous person re-identification (ReID) approaches to person following rely on a fixed feature extractor. Such an approach often fails to generalize to different viewpoints and lighting conditions in practical RPF environments. In other words, it suffers from the so-called domain shift problem where it cannot re-identify the person when his re-appearance is out of the domain modeled by the fixed feature extractor. To mitigate this problem, we propose a ReID framework for RPF where we use a feature extractor that is optimized online with both short-term and long-term experiences (i.e., recently and previously observed samples during RPF) using the online continual learning (OCL) framework. The long-term experiences are maintained by a memory manager to enable OCL to update the feature extractor. Our experiments demonstrate that even in the presence of severe appearance changes and distractions from visually similar people, the proposed method can still re-identify the person more accurately than the state-of-the-art methods.

Keywords: Datasets for Human Motion, Wearable Robotics, SLAM
Abstract: Helmet-mounted wearable positioning systems are crucial for enhancing safety and facilitating coordination in industrial, construction, and emergency rescue environments. These systems, including LiDAR-Inertial Odometry (LIO) and Visual-Inertial Odometry (VIO), often face challenges in localization due to adverse environmental conditions such as dust, smoke, and limited visual features. To address these limitations, we propose a novel head-mounted Inertial Measurement Unit (IMU) dataset with ground truth, aimed at advancing data-driven IMU pose estimation. Our dataset captures human head motion patterns using a helmet-mounted system, with data from ten participants performing various activities. We explore the application of neural networks, specifically Long Short-Term Memory (LSTM) and Transformer networks, to correct IMU biases and improve localization accuracy. Additionally, we evaluate the performance of these methods across different IMU data window dimensions, motion patterns, and sensor types. We release a publicly available dataset, demonstrate the feasibility of advanced neural network approaches for helmet-based localization, and provide evaluation metrics to establish a baseline for future studies in this field. Data and code can be found at url{https://lqiutong.github.io/HelmetPoser.github.io/}

Keywords: Human-Robot Collaboration, Cognitive Modeling, Collision Avoidance
Abstract: Human brain possesses the ability to effectively focus on important environmental components, which enhances perception, learning, reasoning, and decision-making. Inspired by this cognitive mechanism, we introduced a novel concept termed relevance for Human-Robot Collaboration (HRC). Relevance is a dimensionality reduction process that incorporates a continuously operating perception module, evaluates cue sufficiency within the scene, and applies a flexible formulation and computation framework. In this paper, we present an enhanced two-loop framework that integrates real-time and asynchronous processing to quantify relevance and leverage it for safer and more efficient human-robot collaboration (HRC). The two-loop framework integrates an asynchronous loop, which leverages an LLM’s world knowledge to quantify relevance, and a real-time loop, which performs scene understanding, human intent prediction, and decision-making based on relevance. HRC decision-making is enhanced by a relevance-based task allocation method, as well as a motion generation and collision avoidance approach that incorporates human trajectory prediction. Simulations and experiments show that our methodology for relevance quantification can accurately and robustly predict the human objective and relevance, with an average accuracy of up to 0.90 for objective prediction and up to 0.96 for relevance prediction. Moreover, our motion generation methodology reduces collision cases by 63.76% and collision frames by 44.74% when compared with a state-of-the-art (SOTA) collision avoidance method. Our framework and methodologies, with relevance, guide the robot on how to best assist humans and generate safer and more efficient actions for HRC.

Keywords: Telerobotics and Teleoperation, Compliance and Impedance Control, Intention Recognition
Abstract: For various teleoperation tasks, position-based control is not practical. An impedance-based control is superior e.g. for handling fragile objects, like harvesting fruits or grasping a paper cup. However, only a few researchers have focused on impedance control for teleoperation. In tele-impedance, the stiffness of a human is measured and transferred to a controller of a robot. Until now, human stiffness was mostly measured either for specific joints or in 2D Cartesian space. We introduce a new way of measuring Cartesian stiffness in 3D using electromyography. Users were asked to perform a peg-in-hole task in three different orientations (0°, 45°, 90°). Meanwhile, electromyography measurements at shoulder and elbow muscle groups are performed. In a proof-of-concept study, we showed that the measured stiffness matrix in Cartesian space differed significantly across the three differently oriented peg-in-hole scenarios. This demonstrates that human stiffness could be predicted in 3D Cartesian space based on the type of task at hand.

Keywords: Gesture, Posture and Facial Expressions, Human Detection and Tracking, Omnidirectional Vision
Abstract: Fisheye cameras offer robots the ability to capture human movements across a wider field of view (FOV) than standard pinhole cameras, making them particularly useful for applications in human-robot interaction and automotive contexts. However, accurately detecting human poses in fisheye images is challenging due to the curved distortions inherent to fisheye optics. While various methods for undistorting fisheye images have been proposed, their effectiveness and limitations for poses that cover a wide FOV has not been systematically evaluated in the context of absolute human pose estimation from monocular fisheye images. To address this gap, we evaluate the impact of pinhole, equidistant and double sphere camera models, as well as cylindrical projection methods, on 3D human pose estimation accuracy. We find that in close-up scenarios, pinhole projection is inadequate, and the optimal projection method varies with the FOV covered by the human pose. The usage of advanced fisheye models like the double sphere model significantly enhances 3D human pose estimation accuracy. We propose a heuristic for selecting the appropriate projection model based on the detection bounding box to enhance prediction quality. Additionally, we introduce and evaluate on our novel FISHnCHIPS dataset, which features 3D human skeleton annotations in fisheye images, including images from unconventional angles, such as extreme close-ups, ground-mounted cameras, and wide-FOV poses.

Keywords: Modeling and Simulating Humans, Human and Humanoid Motion Analysis and Synthesis, Intention Recognition
Abstract: Predicting human motion may lead to considerable advantages for human-robot interaction, particularly when precise synchronization between the robot’s motion and the user’s movement is imperative. The inherent stochastic nature of human behavior, combined with the restricted window of response, can give rise to residual and undesirable forces during interactions, potentially harming the user. Therefore, efficient prediction of human joint movements may enhance the performance of various interaction control frameworks used in wearable robots. This paper proposes the HuMAn algorithm for predicting human joint motion based on a recurrent neural network. This algorithm consists of a long-term memory network, used to interpret sequences of poses, and a prediction layer, employed to build the most likely future user poses within a specified time horizon. Network training was performed using datasets encompassing various subjects and types of motion. The results demonstrate the effectiveness of the proposed algorithm, as evidenced by average general prediction errors below 0.1 radians for predictive horizons of up to 500 milliseconds. Furthermore, a mean absolute error of 0.026 radians was achieved for a periodic treadmill walk. Simulation results demonstrate a large improvement in transparency control performance in a case study with an upper limb exoskeleton robot.

Keywords: Perception-Action Coupling, Representation Learning
Abstract: Vision language models have played a key role in extracting meaningful features for various robotic applications. Among these, Contrastive Language-Image Pretraining (CLIP) is widely used in robotic tasks that require both vision and natural language understanding. However, CLIP was trained solely on static images paired with text prompts and has not yet been fully adapted for robotic tasks involving dynamic actions. In this paper, we introduce Robotic-CLIP to enhance robotic perception capabilities. We first gather and label large-scale action data, and then build our Robotic-CLIP by fine-tuning CLIP on 309,433 videos (~7.4 million frames) of action data using contrastive learning. By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts. Intensive experiments show that our Robotic-CLIP outperforms other CLIP-based models across various language-driven robotic tasks. Additionally, we demonstrate the practical effectiveness of Robotic-CLIP in real-world grasping applications.

Keywords: Learning from Demonstration, Imitation Learning, Data Sets for Robot Learning
Abstract: In-context imitation learning is the capability to perform novel tasks when prompted with task demonstration examples. In-Context Robot Transformer (ICRT) is a causal transformer that performs autoregressive prediction on sensorimotor trajectories, which include images, proprioceptive states, and actions. This approach supports flexible and trainingfree execution of new tasks at test time. Experiments with a Franka Emika robot demonstrate that ICRT can adapt to new environment configurations that differ from both the prompt and the training data. In a multi-task environment setup, ICRT significantly outperforms current state-of-the-art robot foundation models on generalization to unseen tasks. Code, data, and appendix are available on https://icrt.dev.

Keywords: Incremental Learning, Deep Learning for Visual Perception, Planning under Uncertainty
Abstract: Current methods based on Neural Radiance Fields fail in the low data limit, particularly when training on incomplete scene data. Prior works augment training data only in next-best-view applications, which lead to hallucinations and model collapse with sparse data. In contrast, we propose adding a set of views during training by rejection sampling from a posterior uncertainty distribution, generated by combining a volumetric uncertainty estimator with spatial coverage. We validate our results on partially observed scenes; on average, our method performs 39.9% better with 87.5% less variability across established scene reconstruction benchmarks, as compared to state of the art baselines. We further demonstrate that augmenting the training set by sampling from any distribution leads to better, more consistent scene reconstruction in sparse environments. This work is foundational for robotic tasks where augmenting a dataset with informative data is critical in resource-constrained, a priori unknown environments. Videos and source code are available at https://murpheylab.github.io/low-data-nerf.

Keywords: Imitation Learning, Learning from Demonstration, Representation Learning
Abstract: In this paper, we leverage self-supervised vision transformer models and their emergent semantic abilities to improve the generalization abilities of imitation learning policies. We introduce DVK, an imitation learning algorithm that leverages rich pre-trained Visual Transformer patch-level embeddings to obtain better generalization when learning through demonstrations. Our learner sees the world by clustering appearance features into groups associated with semantic concepts, forming stable keypoints that generalize across a wide range of appearance variations and object types. We demonstrate how this representation enables generalized behaviour by evaluating imitation learning across a diverse dataset of object manipulation tasks. To facilitate further study of generalization in Imitation Learning, all of our code for the method and evaluation, as well as the dataset, is made available.

Keywords: Big Data in Robotics and Automation, Sensorimotor Learning, Learning from Demonstration
Abstract: Interacting with the world is a multi-sensory experience: achieving effective general-purpose interaction requires making use of all available modalities -- including vision, touch, and audio -- to fill in gaps from partial observation. For example, when vision is occluded reaching into a bag, a robot should rely on its senses of touch and sound. However, state-of-the-art generalist robot policies are typically trained on large datasets to predict robot actions solely from visual and proprioceptive observations. In this work, we propose FuSe, a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities for which large datasets are not readily available by leveraging natural language as a common cross-modal grounding. We combine a multimodal contrastive loss with a sensory-grounded language generation loss to encode high-level semantics. In the context of robot manipulation, we show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound in a zero-shot setting, such as multimodal prompting, compositional cross-modal prompting, and descriptions of objects it interacts with. We show that the same recipe is applicable to widely different generalist policies, including both diffusion-based generalist policies and large vision-language-action (VLA) models. Extensive experiments in the real world show that FuSe is able to increase success rates by over 20% compared to all considered baselines.

Keywords: Deep Learning for Visual Perception, Perception for Grasping and Manipulation, Deep Learning Methods
Abstract: Humans have the remarkable ability to use held objects as tools to interact with their environment. For this to occur, humans internally estimate how hand movements affect the object's movement. We wish to endow robots with this capability. We contribute methodology to jointly estimate the geometry and pose of objects grasped by a robot, from RGB images captured by an external camera. Notably, our method transforms the estimated geometry into the robot's coordinate frame, while not requiring the extrinsic parameters of the external camera to be calibrated. Our approach leverages 3D foundation models, large models pre-trained on huge datasets for 3D vision tasks, to produce initial estimates of the in-hand object. These initial estimations do not have physically correct scales and are in the camera's frame. Then, we formulate, and efficiently solve, a coordinate-alignment problem to recover accurate scales, along with a transformation of the objects to the coordinate frame of the robot. Forward kinematics mappings can subsequently be defined from the manipulator's joint angles to specified points on the object. These mappings enable the estimation of points on the held object at arbitrary configurations, enabling robot motion to be designed with respect to coordinates on the grasped objects. We empirically evaluate our approach on a robot manipulator holding a diverse set of real-world objects.

Keywords: Machine Learning for Robot Control, Model Learning for Control, Compliance and Impedance Control
Abstract: This paper proposes a novel force-based task-orientation controller for interaction tasks with environmental orientation uncertainties. The main aim of the controller is to align the robot tool along the main task direction (e.g., along screwing, insertion, polishing, etc.) without the use of any external sensors (e.g., vision systems), relying only on end-effector wrench measurements/estimations. We propose a gradient descent-based orientation controller, enhancing its performance with the orientation predictions provided by a Gaussian Process model. Derivation of the controller is presented, together with simulation results (considering a probing task) and experimental results involving various re-orientation scenarios, i.e., i) a task with the robot in interaction with a soft environment, ii) a task with the robot in interaction with a stiff and inclined environment, and iii) a task to enable the assembly of a gear into its shaft. The proposed controller is compared against a state-of-the-art approach, highlighting its ability to re-orient the robot tool even in complex tasks (where the state-of-the-art method fails).

Keywords: Reinforcement Learning, Imitation Learning, Machine Learning for Robot Control
Abstract: This paper proposes a data-driven model-free inverse reinforcement learning (IRL) algorithm tailored for solving an inverse H_infty control problem. In the problem, both an expert and a learner engage in H_infty control to reject disturbances and the learner's objective is to imitate the expert's behavior by reconstructing the expert's performance function through IRL techniques. Introducing zero-sum game principles, we first formulate a model-based single-loop IRL policy iteration algorithm that includes three key steps: updating the policy, action, and performance function using a new correction formula and the standard inverse optimal control principles. Building upon the model-based approach, we propose a model-free single-loop off-policy IRL algorithm that eliminates the need for initial stabilizing policies and prior knowledge of the dynamics of expert and learner. Also, we provide rigorous proof of convergence, stability, and Nash optimality to guarantee the effectiveness and reliability of the proposed algorithms. Furthermore, we showcase the efficiency of our algorithm through simulations and experiments, highlighting its advantages compared to the existing methods.

Keywords: Machine Learning for Robot Control, Sensorimotor Learning, Learning from Demonstration
Abstract: Differentiable simulation has become a powerful tool for system identification. While prior work has focused on identifying robot properties using robot-specific data or object properties using object-specific data, our approach calibrates object properties by using information from the robot, without relying on data from the object itself. Specifically, we utilize robot joint encoder information, which is commonly available in standard robotic systems. Our key observation is that by analyzing the robot's reactions to manipulated objects, we can infer properties of those objects, such as inertia and softness. Leveraging this insight, we develop differentiable simulations of robot-object interactions to inversely identify the properties of the manipulated objects. Our approach relies solely on proprioception — the robot’s internal sensing capabilities — and does not require external measurement tools or vision-based tracking systems. This general method is applicable to any articulated robot and requires only joint position information. We demonstrate the effectiveness of our method on a low-cost robotic platform, achieving accurate mass and elastic modulus estimations of manipulated objects with just a few seconds of computation on a laptop.

Keywords: Machine Learning for Robot Control, Deep Learning Methods, Reinforcement Learning
Abstract: Adapting reinforcement learning (RL) policies to various robot body configurations is a significant challenge for creating flexible autonomous systems. This study presents a novel framework that integrates Reservoir Computing (RC) with the First-Order Reduced and Controlled Error (FORCE) learning rule to enhance policy adaptability in RL. The RC serves as a dynamic feature extractor, capturing temporal dependencies by pre-training on state transitions generated through random actions. This pre-training acts as regularization, reducing variance and preventing overfitting to specific configurations Subsequently, the control policy network is trained on a limited set of body variations using the enriched features from the RC. Experimental results across three distinct environments demonstrate that the proposed RC+FORCE framework significantly improves policy performance and adaptability to unseen robot configurations compared to traditional reinforcement learning through domain randomization. These findings highlight the effectiveness of combining RC-based feature extraction with FORCE-based training in developing robust RL agents.

Keywords: Machine Learning for Robot Control, Aerial Systems: Mechanics and Control, Robust/Adaptive Control
Abstract: A critical goal of adaptive control is enabling robots to rapidly adapt in dynamic environments. Recent studies have developed a meta-learning-based adaptive control scheme, which uses meta-learning to extract nonlinear features (represented by Deep Neural Networks (DNNs)) from offline data, and uses adaptive control to update linear coefficients online. However, such a scheme is fundamentally limited by the linear parameterization of uncertainties and does not fully unleash the capability of DNNs. This paper introduces a novel learning-based adaptive control framework that pretrains a DNN via self-supervised meta-learning (SSML) from offline trajectories and online adapts the full DNN via composite adaptation. In particular, the offline SSML stage leverages the time consistency in trajectory data to train the DNN to predict future disturbances from history, in a self-supervised manner without environment condition labels. The online stage carefully designs a control law and an adaptation law to update the full DNN with stability guarantees. Empirically, the proposed framework significantly outperforms (19-39%) various classic and learning-based adaptive control baselines, in challenging real-world quadrotor tracking problems under large dynamic wind disturbance.

Keywords: Machine Learning for Robot Control, Vision-Based Navigation, Legged Robots
Abstract: First-order Policy Gradient (FoPG) algorithms such as Backpropagation through Time and Analytical Policy Gradients leverage local simulation physics to accelerate policy search, significantly improving sample efficiency in robot control compared to standard model-free reinforcement learning. However, FoPG algorithms can exhibit poor learning dynamics in contact-rich tasks like locomotion. Previous approaches address this issue by alleviating contact dynamics via algorithmic or simulation innovations. In contrast, we propose guiding the policy search by learning a residual over a simple baseline policy. For quadruped locomotion, we find that the role of residual policy learning in FoPG-based training (FoPG RPL) is primarily to improve asymptotic rewards, compared to improving sample efficiency for model-free RL. Additionally, we provide insights on applying FoPG's to pixel-based local navigation, training a point-mass robot to convergence within seconds. Finally, we showcase the versatility of FoPG RPL by using it to train locomotion and perceptive navigation end-to-end on a quadruped in minutes.

Keywords: Imitation Learning, Integrated Planning and Learning, Sensor Fusion
Abstract: Multimodal end-to-end autonomous driving has shown promising advancements in recent work. By embedding more modalities into end-to-end networks, the system’s understanding of both static and dynamic aspects of the driving environment is enhanced, thereby improving the safety of autonomous driving. In this paper, we introduce METDrive, an end-to-end system that leverages temporal guidance from the embedded time series features of ego states, including rotation angles, steering, throttle signals, and waypoint vectors. The geometric features derived from perception sensor data and the time series features of ego state data jointly guide the waypoint prediction with the proposed temporal guidance loss function. We evaluated METDrive on the CARLA leaderboard benchmarks, achieving a driving score of 70%, a route completion score of 94%, and an infraction score of 0.78.

Keywords: Learning from Demonstration, Representation Learning, Imitation Learning
Abstract: Large real-world driving datasets have sparked significant research into various aspects of learning-based motion planners for autonomous driving. These include data augmentation, model architecture, reward design, training strategies, and planner pipelines. In this paper, we review and benchmark previous methods. Experiments show that many of these approaches have limited generalization abilities in planning performance due to overly complex designs or training paradigms. Experiments further reveal that as models are appropriately scaled, many designs become redundant. Therefore, we introduce StateTransformer-2 (STR2), a scalable, decoder-only motion planner. STR2uses a Vision Transformer (ViT) encoder and a mix-of-experts (MoE) causal transformer architecture. The MoE backbone addresses modality collapse and reward balancing by expert routing during training. Extensive experiments on the NuPlan dataset show that our method generalizes better than previous approaches across different test sets and closed-loop simulations. We evaluate its scalability on billions of real-world urban driving scenarios, demonstrating consistent accuracy improvements as both data and model size grow.

Keywords: Computer Vision for Automation, Autonomous Vehicle Navigation
Abstract: All-weather image restoration (AWIR) is crucial for reliable autonomous navigation under adverse weather conditions. AWIR models are trained to address a specific set of weather conditions such as fog, rain, and snow. But this causes them to often struggle with out-of-distribution (OoD) samples or unseen degradations which limits their effectiveness for real-world autonomous navigation. To overcome this issue, existing models must either be retrained or fine-tuned, both of which are inefficient and impractical, with retraining needing access to large datasets, and fine-tuning involving many parameters. In this paper, we propose using Low-Rank Adaptation (LoRA) to efficiently adapt a pre-trained all-weather model to novel weather restoration tasks. Furthermore, we observe that LoRA lowers the performance of the adapted model on the pre-trained restoration tasks. To address this issue, we introduce a LoRA-based fine-tuning method called LoRA-Align (LoRA-A) which seeks to align the singular vectors of the fine-tuned and pre-trained weight matrices using Singular Value Decomposition (SVD). This alignment helps preserve the model's knowledge of its original tasks while adapting it to unseen tasks. We show that images restored with LoRA and LoRA-A can be effectively used for computer vision tasks in autonomous navigation, such as semantic segmentation and depth estimation. Project page: https://sudraj2002.github.io/loraapage/.

Keywords: Computer Vision for Automation, AI-Based Methods, Vision-Based Navigation
Abstract: 3D detection of vehicles is an essential component for autonomous driving applications. Nevertheless, collecting the supervised training data for learning 3D vehicle detectors would be costly (e.g. utilization of expensive LiDAR sensors) and labor-intensive (for human annotation). In comparison to 3D detection, 2D object detection has achieved a well-developed status, boosting stable and robust performance with widespread application in numerous fields, thanks to the large scale (i.e. amount of samples) of existing training datasets of 2D object detection. Hence, in our work, we propose to realize 3D detection via leveraging the robustness of 2D detectors and developing a network that lifts 2D detections to 3D. With the flexibility of building upon various backbone models (e.g. the models which take image regions detected by 2D detector as inputs to predict their corresponding 3D bounding boxes, or the existing monocular 3D detection models which have the intermediate output of 2D bounding boxes), we propose several geometry-driven objectives, including projection consistency loss, geometry depth loss, and opposite bin loss, to improve the training upon 2D-to-3D lifting. Our extensive experimental results demonstrate that our proposed geometry-driven objectives not only contribute to the superior results of 3D detection but also provide better generalizability across datasets.

Keywords: Autonomous Vehicle Navigation, Simulation and Animation, AI-Based Methods
Abstract: We present LidarDM, a novel LiDAR generative model capable of producing realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. LidarDM stands out with two unprecedented capabilities in LiDAR generative modeling: (i) LiDAR generation guided by driving scenarios, offering significant potential for autonomous driving simulations, and (ii) 4D LiDAR point cloud generation, enabling the creation of realistic and temporally coherent sequences. At the heart of our model is a novel integrated 4D world generation framework. Specifically, we employ latent diffusion models to generate the 3D scene, combine it with dynamic actors to form the underlying 4D world, and subsequently produce realistic sensory observations within this virtual environment. Our experiments indicate that our approach outperforms competing algorithms in realism, temporal coherency, and layout consistency. We additionally show that LidarDM can be used as a generative world model simulator for training and testing perception models.

Keywords: Computer Vision for Automation, Planning, Scheduling and Coordination, Object Detection, Segmentation and Categorization
Abstract: End-to-end autonomous driving with vision-only is not only more cost-effective compared to LiDAR-vision fusion but also more reliable than traditional methods. To achieve a economical and robust purely visual autonomous driving system, we propose RenderWorld, a vision-only end-to-end autonomous driving framework, which generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes the labels by AM-VAE, and use world model for forecasting and planning. RenderWorld employs Gaussian Splatting to represent 3D scenes and render 2D images greatly improves segmentation accuracy and reduces GPU memory consumption compared with NeRF-based methods. By applying AM-VAE to encode air and non-air separately, RenderWorld achieves more fine-grained scene element representation, lead- ing to state-of-the-art performance in both 4D occupancy forecasting and motion planning from autoregressive world model.

Keywords: SLAM, Mapping, Localization
Abstract: Simultaneous Localization and Mapping (SLAM) allows mobile robots to navigate without external positioning systems or pre-existing maps. Radar is emerging as a valuable sensing tool, especially in vision-obstructed environments, as it is less affected by particles than lidars or cameras. Modern 4D imaging radars provide three-dimensional geometric information and relative velocity measurements, but they bring challenges such as a small field of view and sparse, noisy point clouds. Detecting loop closures in SLAM is critical for reducing trajectory drift and maintaining map accuracy. However, the directional nature of 4D radar data makes identifying loop closures, especially from reverse viewpoints, difficult due to limited scan overlap. This article explores using 4D radar for loop closure in SLAM, focusing on similar and opposing viewpoints. We generate submaps for a denser environment representation and use introspective measures to reject false detections in feature-degenerate environments. Our experiments show accurate loop closure detection in geometrically diverse settings for both similar and opposing viewpoints, improving trajectory estimation with up to 82 % improvement in ATE and rejecting false positives in self-similar environments.

Keywords: SLAM, Mapping, Range Sensing
Abstract: This paper presents range-based 6-DoF Monte Carlo SLAM with a gradient-guided particle update strategy. While non-parametric state estimation methods, such as particle filters, are robust in situations with high ambiguity, they are known to be unsuitable for high-dimensional problems due to the curse of dimensionality. To address this issue, we propose a particle update strategy that improves the sampling efficiency by using the gradient information of the likelihood function to guide particles toward its local maxima. Additionally, we introduce a keyframe-based map representation that represents the global map as a set of past frames (i.e., keyframes) to mitigate memory consumption. The keyframe poses for each particle are corrected using a simple loop closure method to maintain trajectory consistency. The combination of gradient information and keyframe-based map representation significantly enhances sampling efficiency and reduces memory usage compared to traditional RBPF approaches. To process a large number of particles (e.g., 100,000 particles) in real-time, the proposed framework is designed to fully exploit GPU parallel processing. Experimental results demonstrate that the proposed method exhibits extreme robustness to state ambiguity and can even deal with kidnapping situations, such as when the sensor moves to different floors via an elevator, with minimal heuristics.

Keywords: Multi-Robot SLAM, Range Sensing
Abstract: Reliable simultaneous localization and mapping (SLAM) algorithms are necessary for safety-critical autonomous navigation. In the communication-constrained multi-agent setting, navigation systems increasingly use point-to-point range sensors as they afford measurements with low bandwidth requirements and known data association. The state estimation problem for these systems takes the form of range-aided (RA) SLAM. However, distributed algorithms for solving the RA-SLAM problem lack formal guarantees on the quality of the returned estimate. To this end, we present the first distributed algorithm for RA-SLAM that can efficiently recover certifiably globally optimal solutions. Our algorithm, distributed certifiably correct RA-SLAM (DCORA), achieves this via the Riemannian Staircase method, where computational procedures developed for distributed certifiably correct pose graph optimization are generalized to the RA-SLAM problem. We demonstrate DCORA's efficacy on real-world multi-agent datasets by achieving absolute trajectory errors comparable to those of a state-of-the-art centralized certifiably correct RA-SLAM algorithm. Additionally, we perform a parametric study on the structure of the RA-SLAM problem using synthetic data, revealing how common parameters affect DCORA's performance.

Keywords: SLAM, Mapping, Embedded Systems for Robotic and Automation
Abstract: A dense SLAM system is essential for mobile robots, as it provides localization and allows navigation, path planning, obstacle avoidance, and decision making in unstructured environments. Due to increasing computational demands, the use of GPUs in dense SLAM is expanding. In this work, we present coVoxSLAM, a novel GPU-accelerated volumetric SLAM system that takes full advantage of the parallel processing power of the GPU to build globally consistent maps even in large-scale environments. It was deployed on different platforms (discrete and embedded GPUs) and compared with the state-of-the-art. The results obtained using public datasets show that coVoxSLAM delivers a significant performance improvement considering execution times while maintaining accurate localization. The presented system is available as an open-source system on GitHub: https://github.com/lrse-uba/coVoxSLAM.

Keywords: SLAM, Mapping, Range Sensing
Abstract: Compared to conventional 3D radar, the 4D imaging radar provides additional height data and finer resolution measurements. Moreover, compared to LiDAR sensors, 4D imaging radar is more cost-effective and offers enhanced durability against challenging weather conditions. Despite these advantages, radar-based localization systems face several challenges, including limited resolution, leading to scattered object recognition and less precise localization. Additionally, existing methods that form submaps from filtered results can accumulate errors, leading to blurred submaps and reducing the accuracy of the SLAM and odometry. To address these challenges, this paper introduces Radar4VoxMap, a novel approach designed to enhance radar-only odometry. The method includes an RCS-weighted voxel distribution map that improves registration accuracy. Furthermore, fixed-lag optimization with the graph is used to optimize both the submap and pose, effectively reducing cumulative errors. The proposed method has shown strong performance on open datasets. The code is available at: url{https://github.com/ailab-hanyang/Radar4VoxMap

Keywords: SLAM, Localization, Mapping
Abstract: Light detection and ranging (LiDAR)-based odometry has been widely utilized for pose estimation due to its use of high-accuracy range measurements and immunity to ambient light conditions. However, the performance of LiDAR odometry varies depending on the environment and deteriorates in degenerative environments such as long corridors. This issue stems from the dependence on a single error metric, which has different strengths and weaknesses depending on the geometrical characteristics of the surroundings. To address these problems, this study proposes a novel iterative closest point (ICP) method called GenZ-ICP. We revisited both point-to-plane and point-topoint error metrics and propose a method that leverages their strengths in a complementary manner. Moreover, adaptability to diverse environments was enhanced by utilizing an adaptive weight that is adjusted based on the geometrical characteristics of the surroundings. As demonstrated in our experimental evaluation, the proposed GenZ-ICP exhibits high adaptability to various environments and resilience to optimization degradation in corridor-like degenerative scenarios by preventing ill-posed problems during the optimization process. Our code is available at https://github.com/cocel-postech/genz-icp.

Keywords: SLAM, Localization, Mapping
Abstract: Robust initialization is crucial for online systems. In the letter, a high-frequency and resilient initialization framework is designed for LiDAR-inertial systems, leveraging both inertial sensors and Doppler LiDAR. The innovative FMCW Doppler LiDAR opens up a novel avenue for robotic sensing by capturing not only point range but also Doppler velocity via the intrinsic Doppler effect. By fusing point-wise Doppler velocity with inertial measurements under non-inertial kinematics, the proposed framework, Free-Init, eliminates reliance on motion undistortion of LiDAR scans, excitation motions, and map correspondences during the initialization phase. Free-Init is also plug-and-play compatible with typical LiDAR-inertial systems and is versatile to handle a wide range of initial motions when the system starts, including stationary, dynamic, and even violent motions. The embedded Doppler-inertial velocimeter ensures fast convergence and high-frequency performance, delivering outputs exceeding 10 kHz. Comprehensive experiments on diverse platforms and across myriad motion scenes validate the framework's effectiveness. The results demonstrate the superior performance of Free-Init, highlighting the necessity of fast, resilient, and dynamic initialization for online systems.

Keywords: Actuation and Joint Mechanisms, Mechanism Design
Abstract: In this paper, we report the development of an innovative two-degrees-of-freedom (2-DOF) spherical parallel remote center of motion mechanism (SPRCMM), which can offer a wide range of movement in both DOFs without encountering singularities. To facilitate the design process, the paper briefly reviews the existing spherical joints, including serial and parallel structures with and without the remote center of motion (RCM). Aiming at combining the advantages of these existing spherical joints, this paper proposes a novel design that utilizes the parallelogram mechanism to form a parallel RCM mechanism without using universal or spherical joints. Forward and inverse kinematics were constructed using the product of the exponentials. Moreover, space and closed Jacobians were derived, accompanied by manipulability in the available workspace for the mechanism. The prototype of the 2-DOF SPRCMM was built and experimentally evaluated. The experimental results confirm that the singularity-free motion of the two DOFs of the mechanism in a wide range is feasible, and the root mean squared errors in the trajectory tracking of the mechanism in most states were less than 10% of the motion range.

Keywords: Mechanism Design, Force Control, Industrial Robots
Abstract: Robot interaction control is often limited to low dynamics or low flexibility, depending on whether an active or passive approach is chosen. In this work, we introduce a hybrid control scheme that combines the advantages of active and passive interaction control. To accomplish this, we propose the design of a novel Active Remote Center of Compliance (ARCC), which is based on a passive and active element which can be used to directly control the interaction forces. We introduce surrogate models for a dynamic comparison against purely robot-based interaction schemes. In a comparative validation, ARCC drastically improves the interaction dynamics, leading to an increase in the motion bandwidth of up to 31 times. We introduce further our control approach as well as the integration in the robot controller. Finally, we analyze ARCC on different industrial benchmarks like peg-in-hole, top-hat rail assembly and contour following problems and compare it against the state of the art, to highlight the dynamic and flexibility. The proposed system is especially suited if the application requires a low cycle time combined with a sensitive manipulation.

Keywords: Mechanism Design, Legged Robots, Compliant Joints and Mechanisms
Abstract: Arboreal environments challenge current robots but are deftly traversed by many familiar animals such as squirrels. We present a small, 450 g robot "Pinto" developed for tree-jumping, a behavior seen in squirrels but rarely in legged robots: jumping from the ground onto a vertical tree trunk. We develop a powerful and lightweight latched series-elastic actuator using a twisted string and carbon fiber springs. We consider the effects of scaling down conventional quadrupeds and experimentally show how storing energy in a parallel-elastic fashion using a latch increases jump energy compared to series-elastic or springless strategies. By switching between series and parallel-elastic modes with our latched 5-bar leg mechanism, Pinto executes energetic jumps as well as maintains continuous control during shorter bounding motions. We also develop sprung 2-DoF arms equipped with spined grippers to grasp tree bark for high-speed perching following a jump.

Keywords: Tendon/Wire Mechanism, Mechanism Design, Robot Safety
Abstract: Cable transmission enables motors of robotic arm to operate lightweight and low-inertia joints remotely in various environments, but it also creates issues with motion coupling and cable routing that can reduce arm's control precision and performance. In this paper, we present a novel motion decoupling mechanism with low-friction to align the cables and efficiently transmit the motor's power. By arranging these mechanisms at the joints, we fabricate a fully decoupled and lightweight cable-driven robotic arm called D3-Arm with all the electrical components be placed at the base. Its 776 mm length moving part boasts six degrees of freedom (DOF) and only 1.6 kg weights. To address the issue of cable slack, a cable-pretension mechanism is integrated to enhance the stability of long-distance cable transmission. Through a series of comprehensive tests, D3-Arm demonstrated 1.29 mm average positioning error and 2 kg payload capacity, proving the practicality of the proposed decoupling mechanisms in cable-driven robotic arm.

Keywords: Mechanism Design, Kinematics, Multi-Robot Systems
Abstract: Articulation between body segments of small insects and animals is a three degree-of-freedom (DOF) motion. Implementing this kind of motion in a compact robot is usually not tractable due to limitations in small actuator technologies. In this work, we concede full 3-DOF control and instead select a one degree-of-freedom curve in SO(3) to articulate segments of a caterpillar robot. The curve is approximated with a spherical four-bar, which is synthesized through optimal rigid body guidance. We specify the desired SO(3) motion using discrete task positions, then solve for candidate mechanisms by computing all roots of the stationary conditions using numerical homotopy continuation. A caterpillar robot prototype demonstrates the utility of this approach. This synthesis procedure is also used to design prolegs for the caterpillar robot. Each segment contains two DC motors and a shape memory alloy, which is used for latching and unlatching between segments. The caterpillar robot is capable of walking, steering, object manipulation, body articulation, and climbing.

Keywords: Mechanism Design, Legged Robots, Compliant Joints and Mechanisms
Abstract: Redesigning and remanufacturing robots are infeasible for resource-constrained environments like space or undersea. This work thus studies how to evaluate and repurpose existing, complementary, quadruped legs for new tasks. We implement this approach on 15 robot designs generated from combining six pre-selected leg designs. The performance maps for force-based locomotion tasks like pulling, pushing, and carrying objects are constructed via a learned policy that works across all designs and adapts to the limits of each. Performance predictions agree well with real-world validation results. The robot can locomote at 0.5 body lengths per second while exerting a force that is almost 60% of its weight.

Keywords: Mechanism Design, Methods and Tools for Robot System Design, Deep Learning Methods
Abstract: Recent advances in foundation models are significantly expanding the capabilities of AI models. As part of this progress, this paper introduces a robot design framework that uses a diffusion model approach for generating 3D mesh structures. Specifically, we focus on generating directly fabricable robot structures that require no post-processing guided by human-imposed design constraints. Our approach can find the optimal design of the robot by optimizing or composing embedding vectors of the model. The efficacy of the framework is validated through an application to design, fabricate, and evaluate a jumping robot. Our solution is an optimized jumping robot with a 41% increase in jump height compared to the state-of-the-art design. Additionally, when the robot is augmented with an optimized foot, it can land reliably with a success ratio of 88% in contrast to the 4% success ratio of the base robot.

Keywords: Sensor Fusion, Probabilistic Inference, SLAM
Abstract: This paper proposes a novel Hessian approximation for Maximum a Posteriori estimation problems in robotics involving Gaussian mixture likelihoods. Previous approaches manipulate the Gaussian mixture likelihood into a form that allows the problem to be represented as a nonlinear least squares (NLS) problem. The resulting Hessian approximation used within NLS solvers from these approaches neglects certain nonlinearities. The proposed Hessian approximation is derived by setting the Hessians of the Gaussian mixture component errors to zero, which is the same starting point as for the Gauss-Newton Hessian approximation for NLS, and using the chain rule to account for additional nonlinearities. The proposed Hessian approximation results in improved convergence speed and uncertainty characterization for simulated experiments, and similar performance to the state of the art on real-world experiments. A method to maintain compatibility with existing solvers, such as ceres, is also presented. Accompanying software and supplementary material can be found at https://github.com/decargroup/hessian_sum_mixtures.

Keywords: Deep Learning for Visual Perception, Sensor Fusion, RGB-D Perception
Abstract: Monocular depth estimation from RGB images plays a pivotal role in 3D vision. However, its accuracy can deteriorate in challenging environments such as nighttime or adverse weather conditions. While long-wave infrared cameras offer stable imaging in such challenging conditions, they are inherently low-resolution, lacking rich texture and semantics as delivered by the RGB image. Current methods focus solely on a single modality due to the difficulties to identify and integrate faithful depth cues from both sources. To address these issues, this paper presents a novel approach that identifies and integrates dominant cross-modality depth features with a learning-based framework. Concretely, we independently compute the coarse depth maps with separate networks by fully utilizing the individual depth cues from each modality. As the advantageous depth spreads across both modalities, we propose a novel confidence loss steering a confidence predictor network to yield a confidence map specifying latent potential depth areas. With the resulting confidence map, we propose a multi-modal fusion network that fuses the final depth in an end-to-end manner. Harnessing the proposed pipeline, our method demonstrates the ability of robust depth estimation in a variety of difficult scenarios. Experimental results on the challenging MS^2 and ViViD++ datasets demonstrate the effectiveness and robustness of our method.

Keywords: Computer Vision for Automation, Sensor Fusion
Abstract: Camera and LiDAR serve as informative sensors for accurate and robust autonomous driving systems. However, these sensors often exhibit heterogeneous natures, resulting in distributional modality gaps that present significant challenges for fusion. To address this, a robust fusion technique is crucial, particularly for enhancing 3D object detection. In this paper, we introduce a dynamic adjustment technology aimed at aligning modal distributions and learning effective modality representations to enhance the fusion process. Specifically, we propose a triphase domain aligning module. This module adjusts the feature distributions from both the camera and LiDAR, bringing them closer to the ground truth domain and minimizing differences. Additionally, we explore improved representation acquisition methods for dynamic fusion, which includes modal interaction and specialty enhancement. Finally, an adaptive learning technique that merges the semantics and geometry information for dynamical instance optimization. Extensive experiments in the nuScenes dataset present competitive performance with state-of-the-art approaches. Our code will be released in the future.

Keywords: Computer Vision for Transportation, Sensor Fusion, Deep Learning for Visual Perception
Abstract: Deploying depth estimation networks in the real world requires high-level robustness against various adverse conditions to ensure safe and reliable autonomy. For this purpose, many autonomous vehicles employ multi-modal sensor systems, including an RGB camera, NIR camera, thermal camera, LiDAR, or Radar. They mainly adopt two strategies to use multiple sensors: modality-wise and multi-modal fused inference. The former method is flexible but memory-inefficient, unreliable, and vulnerable. Multi-modal fusion can provide high-level reliability, yet it needs a specialized architecture. In this paper, we propose an effective solution, named align-and-fuse strategy, for the depth estimation from multi-spectral images. In the align stage, we align embedding spaces between multiple spectrum bands to learn shareable representation across multi-spectral images by minimizing contrastive loss of global and spatially aligned local features with geometry cues. After that, in the fuse stage, we train an attachable feature fusion module that can selectively aggregate the multi-spectral features for reliable and robust prediction results. Based on the proposed method, a single-depth network can achieve both spectral-invariant and multi-spectral fused depth estimation while preserving reliability, memory efficiency, and flexibility.

Keywords: RGB-D Perception, Deep Learning for Visual Perception, Sensor Fusion
Abstract: Mobile robots operating indoors must be prepared to navigate challenging scenes that contain transparent surfaces. This paper proposes a novel method for the fusion of acoustic and visual sensing modalities through implicit neural represen- tations to enable dense reconstruction of transparent surfaces in indoor scenes. We propose a novel model that leverages generative latent optimization to learn an implicit representation of indoor scenes consisting of transparent surfaces. We demonstrate that we can query the implicit representation to enable volumetric rendering in image space or 3D geometry reconstruction (point clouds or mesh) with transparent surface prediction. We evaluate our method’s effectiveness qualitatively and quantitatively on a new dataset collected using a custom, low-cost sensing platform featuring RGB-D cameras and ultrasonic sensors. Our method exhibits significant improvement over state-of-the-art for transparent surface reconstruction.

Keywords: Sensor Fusion, Deep Learning for Visual Perception
Abstract: Multi-modal fusion can improve perceptual robustness and accuracy by fully utilizing multi-source sensor data. Current RGB-T fusion methods still falter with adverse illumination and weather. Recent advances in generative methods have shown the ability to enhance and restore visible images in adverse conditions. However, the fusion of RGB-T based on generative methods has not been studied in depth, due to limited attention given to the degradation of multi-modal features under challenging circumstances. Motivated by this observation, we propose CDMFusion, a three-branch conditional diffusion model that achieves fusion with dynamically enhancing multi-modal features and suppressing high-frequency interference. Specifically, we achieve feature-preserving fusion through three branches and establish a dynamic gating prediction module to adjust the enhancement of multi-modal features adaptively. In addition, considering the high time cost of existing diffusion models for generating fused images, we propose a skip patrol mechanism to achieve accelerated high-quality generation with no need for additional training. Experiments demonstrate our method achieves excellent performance in multiple datasets. The code and datasets are available at https://github.com/yangluojie/CDMFusion.

Keywords: Sensor Fusion, Object Detection, Segmentation and Categorization, AI-Based Methods
Abstract: 4D millimeter-wave (MMW) radar, which provides both height information and dense point cloud data over 3D MMW radar, has become increasingly popular in 3D object detection. In recent years, radar-vision fusion models have demonstrated performance close to that of LiDAR-based models, offering advantages in terms of lower hardware costs and better resilience in extreme conditions. However, many radar-vision fusion models treat radar as a sparse LiDAR, underutilizing radar-specific information. Additionally, these multi-modal networks are often sensitive to the failure of a single modality, particularly vision. To address these challenges, we propose the Radar Depth Lift-Splat-Shoot (RDL) module, which integrates radar-specific data into the depth prediction process, enhancing the quality of visual Bird’s-Eye View (BEV) features. We further introduce a Unified Feature Fusion (UFF) approach that extracts BEV features across different modalities using shared module. To assess the robustness of multi-modal models, we develop a novel Failure Test (FT) ablation experiment, which simulates vision modality failure by injecting Gaussian noise. We conduct extensive experiments on the View-of-Delft (VoD) and TJ4D datasets. The results demonstrated that our proposed Unified BEVFusion (UniBEVFusion) network significantly outperforms state-of-the-art models on the TJ4D dataset, with improvements of 3.96% in 3D and 4.17% in BEV object detection accuracy.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications, Motion Control
Abstract: This paper addresses the problem of thrust estimation and control for the rotors of small-sized multirotors Uncrewed Aerial Vehicles (UAVs). Accurate control of the thrust generated by each rotor during flight is one of the main challenges for robust control of quadrotors. The most common approach is to approximate the mapping of rotor speed to thrust with a simple quadratic model. This model is known to fail under non-hovering flight conditions, introducing errors into the control pipeline. One of the approaches to modeling the aerodynamics around the propellers is the Blade Element Momentum Theory (BEMT). Here, we propose a novel BEMT-based closed-loop thrust estimator and control to eliminate the laborious calibration step of finding several aerodynamic coefficients. We aim to reuse known values as a baseline and fit the thrust estimate to values closest to the real ones with a simple test bench experiment, resulting in a single scaling value. A feedforward PID thrust control was implemented for each rotor, and the methods were validated by outdoor experiments with two multirotor UAV platforms: 250mm and 500mm. A statistical analysis of the results showed that the thrust estimation and control provided better robustness under aerodynamically varying flight conditions compared to the quadratic model.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications
Abstract: Efficient real-time trajectory planning and control for fixed-wing unmanned aerial vehicles is challenging due to their non-holonomic nature, complex dynamics, and the additional uncertainties introduced by unknown aerodynamic effects. In this paper, we present a fast and efficient real-time trajectory planning and control approach for fixed-wing unmanned aerial vehicles, leveraging the differential flatness property of fixed-wing aircraft in coordinated flight conditions to generate dynamically feasible trajectories. The approach provides the ability to continuously replan trajectories, which we show is useful to dynamically account for the curvature constraint as the aircraft advances along its path. Extensive simulations and real-world experiments validate our approach, showcasing its effectiveness in generating trajectories across various flight conditions, including wind disturbances.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Perception and Autonomy
Abstract: This paper introduces a safety filter to ensure collision avoidance for multirotor aerial robots. The proposed formalism leverages a single Composite Control Barrier Function from all position constraints acting on a third-order nonlinear representation of the robot's dynamics. We analyze the recursive feasibility of the safety filter under the composite constraint and demonstrate that the infeasible set is negligible. The proposed method allows computational scalability against thousands of constraints and, thus, complex scenes with numerous obstacles. We experimentally demonstrate its ability to guarantee the safety of a quadrotor with an onboard LiDAR, operating in both indoor and outdoor cluttered environments against both naive and adversarial nominal policies.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications
Abstract: This paper presents the Spinning Blimp, a novel lighter-than-air (LTA) aerial vehicle designed for low-energy stable flight. Using an oblate spheroid helium balloon for buoyancy, the vehicle achieves minimal energy consumption while maintaining prolonged airborne states. The unique and low-cost design employs a passively arranged wing coupled with a propeller to induce a spinning behavior, providing inherent pendulum-like stabilization. We propose a control strategy that takes advantage of the continuous revolving nature of the spinning blimp to control translational motion. The cost-effectiveness of the vehicle makes it highly suitable for a variety of applications, such as patrolling, localization, air and turbulence monitoring, and domestic surveillance. Experimental evaluations affirm the design's efficacy and underscore its potential as a versatile and economically viable solution for aerial applications.

Keywords: Aerial Systems: Mechanics and Control, Reinforcement Learning, Robust/Adaptive Control
Abstract: In high-speed quadcopter racing, finding a single controller that works well across different platforms remains challenging. This work presents the first neural network controller for drone racing that generalizes across physically distinct quadcopters. We demonstrate that a single network, trained with domain randomization, can robustly control various types of quadcopters. The network relies solely on the current state to directly compute motor commands. The effectiveness of this generalized controller is validated through real-world tests on two substantially different crafts (3-inch and 5-inch race quadcopters). We further compare the performance of this generalized controller with controllers specifically trained for the 3-inch and 5-inch drone, using their identified model parameters with varying levels of domain randomization (0%, 10%, 20%, 30%). While the generalized controller shows slightly slower speeds compared to the fine-tuned models, it excels in adaptability across different platforms. Our results show that no randomization fails sim-to-real transfer while increasing randomization improves robustness but reduces speed. Despite this trade-off, our findings highlight the potential of domain randomization for generalizing controllers, paving the way for universal AI controllers that can adapt to any platform.

Keywords: Aerial Systems: Mechanics and Control
Abstract: This paper introduces a novel method for controlling multirotor aerial robots connected by passive flexible elements. Despite the growing popularity of multirotor aerial robots, their real-world applications remain limited due to difficulties adapting to complex environments. Soft robotics, due to their inherent flexibility, offer a potential solution, though research on integrating flexible elements into aerial robots is still in the early stages. In this study, we propose control methods for a system where multiple aerial robots are interconnected with passive flexible elements. These robotic systems enhance adaptability, enabling tasks like object manipulation. We model the flexible parts using the piecewise constant strain (PCS) model, which allows for model-based closed-loop control and stabilizes various configurations of the system. Through simulations and experiments, we validated that the proposed method achieves both stable flight and flexible deformation. Notably, we succeeded in maintaining stable flight, which was not possible with traditional methods, and demonstrated both positional controllability and the ability of the flexible parts to bend dynamically during flight.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Perception and Autonomy, Compliant Joints and Mechanisms
Abstract: Aerial robots are a well-established solution for environmental surveying, exploration, and inspection, thanks to their superior maneuverability and agility. Nowadays, the algorithms that provide these capabilities rely on GNSS and Vision, which are obstructed in some environments of interest, e.g., indoors and underground or in smoke and dust. In similar conditions, animals rely on the sense of touch and compliant responses to interactions embodied in the body morphology. This way, they can navigate safely using tactile cues by feeling the environment surrounding them. In this work, we take inspiration from the natural example and propose an approach that allows a quadrotor to navigate using tactile information from the environment. We propose to endow a conventional quadrotor with a novel robotic finger that embodies compliance and sensing capabilities. We complete this design with a navigation approach that generates new waypoints based on the robotic finger's contact information to follow the unknown environment. The overall system's evaluation shows successful, repeatable results in 36 flight experiments with various relative angles between the drone and a planar surface.

Keywords: Deep Learning for Visual Perception, Visual Learning, Computer Vision for Transportation
Abstract: Hybrid visual odometry methods achieve state-of-the-art performance by fusing both data-based deep learning networks and rule-based localization approaches. However, these methods also suffer from deep learning domain gap problems, which leads to an accuracy drop of the hybrid visual odometry approach when new type of data is considered. This paper is the first to explore a practical solution to this problem. Indeed, the deep learning network in the hybrid visual odometry predicts the stereo disparity with fixed searching space. However, the disparity distribution is unbalanced in stereo images acquired in different environments. We propose an adaptive network structure to overcome this problem. Secondly, the rule-based localization module has a robust performance by online optimizing the camera pose in test data, which motivates us to introduce test-time training machine learning method for improving the data-based part of the hybrid visual odometry.

Keywords: Deep Learning Methods, Deep Learning for Visual Perception, Recognition
Abstract: This paper proposes SOLVR, a unified pipeline for learning based LiDAR-Visual re-localisation which performs place recognition and 6-DoF registration across sensor modalities. We propose a strategy to align the input sensor modalities by leveraging stereo image streams to produce metric depth predictions with pose information, followed by fusing multiple scene views from a local window using a probabilistic occupancy framework to expand the limited field-of-view of the camera. Additionally, SOLVR adopts a flexible definition of what constitutes positive examples for different training losses, allowing us to simultaneously optimise place recognition and registration performance. Furthermore, we replace RANSAC with a registration function that weights a simple least-squares fitting with the estimated inlier likelihood of sparse keypoint correspondences, improving performance in scenarios with a low inlier ratio between the query and retrieved place.

Keywords: Deep Learning Methods, Computer Vision for Transportation, Object Detection, Segmentation and Categorization
Abstract: Scene flow enables an understanding of the motion characteristics of the environment in the 3D world. It gains particular significance in the long-range, where object-based perception methods might fail due to sparse observations far away. Although significant advancements have been made in scene flow pipelines to handle large-scale point clouds, a gap remains in scalability with respect to long-range. We attribute this limitation to the common design choice of using dense feature grids, which scale quadratically with range. In this paper, we propose Sparse Scene Flow (SSF), a general pipeline for long-range scene flow, adopting a sparse convolution based backbone for feature extraction. This approach introduces a new challenge: a mismatch in size and ordering of sparse feature maps between time-sequential point scans. To address this, we propose a sparse feature fusion scheme, that augments the feature maps with virtual voxels at missing locations. Additionally, we propose a range-wise metric that implicitly gives greater importance to faraway points. Our method, SSF, achieves state-of-the-art results on the Argoverse2 dataset, demonstrating strong performance in long-range scene flow estimation. Our source code is open-sourced at https://github.com/KTH-RPL/SSF.

Keywords: Deep Learning Methods, Autonomous Agents, Task and Motion Planning
Abstract: While humans can successfully navigate using abstractions, ignoring details that are irrelevant to the task at hand, most of the existing approaches in robotics require detailed environment representations which consume a significant amount of sensing, computing, and storage; these issues become particularly important in resource-constrained settings with limited power budgets. Deep learning methods can learn from prior experience to abstract knowledge from novel environments, and use it to more efficiently execute tasks such as frontier exploration, object search, or scene understanding. We propose BoxMap, a Detection-Transformer-based architecture that takes advantage of the structure of the sensed partial environment to update a topological graph of the environment as a set of semantic entities (rooms and doors) and their relations (connectivity). The predictions from low-level measurements can be leveraged to achieve high-level goals with lower computational costs than methods based on detailed representations. As an example application, we consider a robot equipped with a 2-D laser scanner tasked with exploring a residential building. Our BoxMap representation scales quadratically with the number of rooms (with a small constant), resulting in significant savings over a full geometric map. Moreover, our high-level topological representation results in 30.9% shorter trajectories in the exploration task with respect to a standard method. Code is available at: bit.ly/3F6w2Yl.

Keywords: Vision-Based Navigation, Integrated Planning and Learning, Computer Vision for Transportation
Abstract: End-to-end autonomous driving aims to produce planning trajectories from raw sensors directly. Currently, most approaches integrate perception, prediction, and planning modules into a fully differentiable network, promising great scalability. However, these methods typically rely on deterministic modeling of online maps in the perception module for guiding or constraining vehicle planning, which may incorporate erroneous perception information and further compromise planning safety. To address this issue, we delve into the importance of online map uncertainty for enhancing autonomous driving safety and propose a novel paradigm named UncAD. Specifically, UncAD first estimates the uncertainty of the online map in the perception module. It then leverages the uncertainty to guide motion prediction and planning modules to produce multi-modal trajectories. Finally, to achieve safer autonomous driving, UncAD proposes an uncertainty-collision-aware planning selection strategy according to the online map uncertainty to evaluate and select the best trajectory. In this study, we incorporate UncAD into various state-of-the-art (SOTA) end-to-end methods. Experiments on the nuScenes dataset show that integrating UncAD, with only a 1.9% increase in parameters, can reduce collision rates by up to 26% and drivable area conflict rate by up to 42%. Codes, pre-trained models, and demo videos can be accessed at https://github.com/pengxuanyang/UncAD.

Keywords: Vision-Based Navigation, Embodied Cognitive Science, Visual Learning
Abstract: Object navigation in multi-floor environments presents a formidable challenge in robotics, requiring sophisticated spatial reasoning and adaptive exploration strategies. Traditional approaches have primarily focused on single-floor scenarios, overlooking the complexities introduced by multi-floor structures. To address these challenges, we first propose a Multi-floor Navigation Policy (MFNP) and implement it in Zero-Shot object navigation tasks. Our framework comprises three key components: (i) Multi-floor Navigation Policy, which enables an agent to explore across multiple floors; (ii) Multi-modal Large Language Models (MLLMs) for reasoning in the navigation process; and (iii) Inter-Floor Navigation, ensuring efficient floor transitions. We evaluate MFNP on the Habitat-Matterport 3D (HM3D) and Matterport 3D (MP3D) datasets, both include multi-floor scenes. Our experiment results demonstrate that MFNP significantly outperforms all the existing methods in Zero-Shot object navigation, achieving higher success rates and improved exploration efficiency. Ablation studies further highlight the effectiveness of each component in addressing the unique challenges of multi-floor navigation. Meanwhile, we conducted real-world experiments to evaluate the feasibility of our policy. Upon deployment of MFNP, the Unitree quadruped robot demonstrated successful multi-floor navigation and found the target object in a completely unseen environment. By introducing MFNP, we offer a new paradigm for tackling complex, multi-floor environments in object navigation tasks, opening avenues for future research in visual-based navigation in realistic, multi-floor settings.

Keywords: Distributed Robot Systems, Vision-Based Navigation, Field Robots
Abstract: Centralized learning requires data to be aggregated at a central server, which poses significant challenges in terms of data privacy and bandwidth consumption. Federated learning presents a compelling alternative, however, vanilla Federated Learning methods deployed in robotics aim to learn a single global model across robots that works ideally for all. But in practice one model may not be well suited for robots deployed in various environments. This paper proposes Federated-EmbedCluster (Fed-EC), a clustering-based federated learning framework that is deployed with vision based autonomous robot navigation in diverse outdoor environments. The framework addresses the key federated learning challenge of deteriorating model performance of a single global model due to the presence of non-IID data across real-world robots. Extensive real-world experiments validate that Fed-EC reduces the communication size by 23x for each robot while matching the performance of centralized learning for goal-oriented navigation and outperforms local learning. Fed-EC can transfer previously learnt models to new robots that join the cluster.

Keywords: Range Sensing, Deep Learning for Visual Perception, Vision-Based Navigation
Abstract: We investigate methods for determining if a planar surface contains geometric deviations (e.g. protrusions, objects, divots, or cliffs) using only an instantaneous measurement from a miniature optical time-of-flight sensor. The key to our method is to utilize the entirety of information encoded in raw time-of-flight data captured by off-the-shelf distance sensors. We provide an analysis of the problem in which we identify the key ambiguity between geometry and surface photometrics. To overcome this challenging ambiguity, we fit a Gaussian mixture model to a small dataset of planar surface measurements. This model implicitly captures the expected geometry and distribution of photometrics of the planar surface and is used to identify measurements that are likely to contain deviations. We characterize our method on a variety of surfaces and planar deviations across a range of scenarios. We find that our method utilizing raw time-of-flight data outperforms baselines which use only derived distance estimates. We build an example application in which our method enables mobile robot obstacle and cliff avoidance over a wide field-of-view.

Keywords: Localization, SLAM, Range Sensing
Abstract: We often encounter limited FOV situations due to various factors such as sensor fusion or sensor mount in real-world robot navigation. However, the limited FOV interrupts the generation of descriptions and impacts place recognition adversely. Therefore, we suffer from correcting accumulated drift errors in a consistent map using LiDAR-based place recognition with limited FOV. Thus, in this paper, we propose a robust LiDAR-based place recognition method for handling narrow FOV scenarios. The proposed method establishes spatial organization based on the range-elevation bin and azimuth-elevation bin to represent places. In addition, we achieve a robust place description through reweighting based on vertical direction information. Based on these representations, our method enables addressing rotational changes and determining the initial heading. Additionally, we designed a lightweight and fast approach for the robot's onboard autonomy. For rigorous validation, the proposed method was tested across various LiDAR place recognition scenarios (i.e., single-session, multi-session, and multi-robot scenarios). To the best of our knowledge, we report the first method to cope with the restricted FOV. Our place description and SLAM codes will be released. Also, the supplementary materials of our descriptor are available at https://sites.google.com/view/lidar-solid.

Keywords: Visual Tracking, Deep Learning for Visual Perception, Data Sets for Robotic Vision
Abstract: 事件数据最近成为 object 的有价值的辅助对象 跟踪，提供具有密集时间分辨率的提示，以及 高动态范围。现有的 RGB 事件跟踪器通常 在使用 仅靠 RGB 功能无法实现的复杂运动轨迹 提供足够的鉴别力。为了解决这个问题，我们 提出了一个创新的 RGB 事件跟踪框架，称为 EventTPT，通过触发 嵌入在历史 轨迹。具体来说，EventTPT 集成了 多个相邻帧的轨迹转换为单个 事件图像使用时间加权聚合和 随后将其作为视觉提示输入到 跟踪器进行当前帧定位。跨模态自适应 融合模块进一步设计用于 光度不一致的情况。此外，我们 提出了一种新颖的具有挑战性的 RGB 事件跟踪基准， EventUAV，包含具有高运动复杂&#

Keywords: Vision-Based Navigation, Visual Tracking
Abstract: Endoscope tracking is commonly utilized to provide surgeons with in-body camera poses and visual fields during invasive procedures. The fundamental aspect of endoscopic navigation lies in precisely and continuously tracing the position and orientation of the endoscope within monocular endoscopic video sequences in a preoperative data space. This work proposes a new spatially constrained and deeply learned bilateral structural intensity-depth 2D-3D registration framework for autonomously navigating a flexible endoscope. Concretely, a novel bilateral structural intensity-depth similarity function is defined to tackle the deficiency of using image intensity, while a cross-domain monocular depth estimation model trained on virtual image data is used to accurately predict real image dense depth. Additionally, a spatial constraint is introduced to precisely reinitialize an optimizer to reduce accumulative tracking errors. We validate our method on clinical data, with the experimental results showing that our method significantly outperforms current vision-based navigation methods. Particularly, the average of position and orientation errors were reduced from (4.59mm, 9.22degree) to (1.65mm, 4.67degree).

Keywords: Visual Tracking, Representation Learning, Deep Learning Methods
Abstract: High-speed object tracking holds significant relevance across robotic domains, such as drones and autonomous driving. Compared to conventional cameras, event cameras are equipped with the ability to capture object motion information at exceptionally high temporal resolution with relatively low power consumption and remain immune from motion-blurring effects. Regrettably, many existing methods adopt a frame-based approach by stacking events into Event Frame, which overlooks the sparsity and high temporal resolution of events. This approach is reliant on the pre-training backbone and reaches a performance plateau but demands unrealistically large networks and high power consumption, rendering it impractical for real-time applications in battery-constrained scenarios. In this paper, we propose an efficient and effective single-modality tracker using Point Cloud representation named E2B (Event to Box). By directly handling the raw output of event cameras without dataformat transformation, E2B leverages events' coordinate guidance to accurately map Event Cloud features to 2D bounding boxes. Moreover, E2B incorporates the pyramid structure into the multi-stage feature extraction architecture to effectively track objects across diverse scales. In the experiments, E2B performs outstandingly on two large-scale and one synthetic event-based tracking datasets, covering both indoor and outdoor environments, as well as rigid and non-rigid objects.

Keywords: Deep Learning for Visual Perception
Abstract: Most stereo matching networks assume that the stereo images are perfectly rectified, ignoring the perturbation of extrinsic parameters due to collisions, mechanical vibrations, and thermal expansion. This leads to poor rectification robustness in real-world stereo systems. That is, even minor rectification errors can lead to failure, making stereo systems unreliable for long-term autonomous operation in complex environments. In this paper, we are the first to propose a frequency filtering-based rectification robustness (F²R²) method for stereo matching, which aims to enhance the robustness of existing stereo networks to rectification errors. Specifically, we propose a sensitive frequency filter (SFF) to remove components susceptible to rectification errors within the frequency domain. SFF achieves the filtering through the learning-based adaptive filtering mask (AFM) guided by the spatial-frequency mapping modulation mask (SFM). Moreover, we build the matching feature reconstruction module (MFRM) to recover the features lost during filtering to benefit cost aggregation. Comprehensive experiments on simulated datasets and self-collected data validate that our method can significantly enhance the rectification robustness of stereo matching networks.

Keywords: Vision-Based Navigation, Aerial Systems: Mechanics and Control, Aerial Systems: Perception and Autonomy
Abstract: This paper presents VisTune, a method for automatic controller tuning, specifically designed for UAVs using vision-based localization for position control. In contrast to existing methods that involve flying the UAV manually to collect the data for system identification and tuning, our approach leverages relay-based system identification and tuning that autonomously generates stable oscillations, without the need for stabilizing controller. The whole process concludes within few seconds. Prior work in vision-based position control of the UAVs often ignores the delay from the perception pipeline, which is quite significant and results in suboptimal tuning and poor control performance. Our approach accounts for perception delay and addresses practical issues, such as varying delays due to varying computation requirements and inevitable estimation errors, which pose challenges in applying relay-based identification and tuning. Typically, VBL system introduces over 100 ms delay, compared to less than 20 ms delay when motion capture system is used. Moreover, we show that the perception delay identified by VisTune can be effectively used to temporally advance the feedforward acceleration signal to achieve better tracking performance. Finally, we demonstrate the robustness of the tuned controllers on a trajectory tracking task, reaching speed up to 2.1 m/s with RMS control error of only 0.054 m while under wind disturbance of 5 m/s we report RMSE of 0.116 m. A video of experiments is available at https://youtu.be/hJoT8bn0K0o

Keywords: Learning from Experience, Software Architecture for Robotic and Automation, Cognitive Control Architectures
Abstract: Humans can autonomously learn the relationship between sensation and motion in their own bodies, estimate and control their own body states, and move while continuously adapting to the current environment. On the other hand, current robots control their bodies by learning the network structure described by humans from their experiences, making certain assumptions on the relationship between sensors and actuators. In addition, the network model does not adapt to changes in the robot's body, the tools that are grasped, or the environment, and there is no unified theory, not only for control but also for state estimation, anomaly detection, simulation, and so on. In this study, we propose a Generalized Multisensory Correlational Model (GeMuCo), in which the robot itself acquires a body schema describing the correlation between sensors and actuators from its own experience, including model structures such as network input/output. The robot adapts to the current environment by updating this body schema model online, estimates and controls its body state, and even performs anomaly detection and simulation. We demonstrate the effectiveness of this method by applying it to tool-use co