Keywords: Intention Recognition, Deep Learning Methods, Neurorobotics
Abstract: Human intention detection with hand motion prediction is critical to drive the upper-extremity assistive robots in neurorehabilitation applications. However, the traditional methods relying on physiological signal measurement are restrictive and often lack environmental context. We propose a novel approach that predicts future sequences of both hand poses and joint positions.This method integrates gaze information, historical hand motion sequences, and environmental object data, adapting dynamically to the assistive needs of the patient without prior knowledge of the intended object for grasping. Specifically, we use a vector-quantized variational autoencoder for robust hand pose encoding with an autoregressive generative transformer for effective hand motion sequence prediction. We demonstrate the usability of these novel techniques in a pilot study with healthy subjects.To train and evaluate the proposed method, we collect a dataset consisting of various types of grasp actions on different objects from multiple subjects. Through extensive experiments, we demonstrate that the proposed method can successfully predict sequential hand movement. Especially, the gaze information shows significant enhancements in prediction capabilities, particularly with fewer input frames, highlighting the potential of the proposed method for real-world applications.

Keywords: Telerobotics and Teleoperation, Human-Robot Collaboration, Intention Recognition
Abstract: For direct teleoperation tasks, the follower robot accomplishes tasks by strictly executing the inputs from the operator. However, the operator's physiological tremor seriously reduces the smoothness of the trajectory, especially in tasks relying on operator’s experience such as gluing, while the random tremor is hard to be described and suppressed online. To navigate this challenge, this paper proposes an Online Hidden Markov Model with two-layer Bayesian (TLB-OHMM) method to suppress the tremor by estimating the operator's expected speed, which can recognize new intention online and is adaptive to complex trajectories. First, the actual moving speed of the human hand is modeled as an HMM. Then an Online-HMM method based on online expectation maximum (EM) algorithm is introduced to shorten the training time and realize the online updating of the HMM parameters. Finally, a two-layer Bayesian method is proposed to estimate the expected speed of the human hand. Experimental results in simulation and real teleoperated gluing task show that the proposed method greatly reduces computation time and improves the quality of trajectories, especially for complex curve trajectories, compared with the traditional HMM based method.

Keywords: Intention Recognition, Intelligent Transportation Systems, Human Detection and Tracking
Abstract: Accurate prediction of pedestrian trajectories is crucial for improving the safety of autonomous driving. However, this task is generally nontrivial due to the inherent stochasticity of human motion, which naturally requires the predictor to generate multi-modal prediction. Previous works leverage various generative methods, such as GAN and VAE, for pedestrian trajectory prediction. Nevertheless, these methods may suffer from mode collapse and relatively low-quality results. The denoising diffusion probabilistic model (DDPM) has recently been applied to trajectory prediction due to its simple training process and powerful reconstruction ability. However, current diffusion-based methods do not fully utilize input information and usually require many denoising iterations that lead to a long inference time or an additional network for initialization. To address these challenges and facilitate the use of diffusion models in multi-modal trajectory prediction, we propose GDTS, a novel Goal-Guided Diffusion Model with Tree Sampling for multi-modal trajectory prediction. Considering the "goal-driven" characteristics of human motion, GDTS leverages goal estimation to guide the generation of the diffusion network. A two-stage tree sampling algorithm is presented, which leverages common features to reduce the inference time and improve accuracy for multi-modal prediction. Experimental results demonstrate that our proposed framework achieves comparable state-of-the-art performance with real-time inference speed in public datasets.

Keywords: Intention Recognition, Multi-Modal Perception for HRI, Deep Learning Methods
Abstract: Recognition of human manipulation actions in real-time is essential for safe and effective human-robot interaction and collaboration. The challenge lies in developing a model that is both lightweight enough for real-time execution and capable of generalization. While some existing methods in the literature can run in real-time, they struggle with temporal scalability, i.e., they fail to adapt to long-duration manipulations effectively. To address this, leveraging the generalizable scene graph representations, we propose a new Factorized Graph Sequence Encoder network that not only runs in real-time but also scales effectively in the temporal dimension, thanks to its factorized encoder architecture. Additionally, we introduce Hand Pooling operation, a simple pooling operation for more focused extraction of the graph-level embeddings. Our model outperforms the previous state-of-the-art real-time approach, achieving a 14.3% and 5.6% improvement in F1-macro score on the KIT Bimanual Action (Bimacs) Dataset and Collaborative Action (CoAx) Dataset, respectively. Moreover, we conduct an extensive ablation study to validate our network design choices. Finally, we compare our model with its architecturally similar RGB-based model on the Bimacs dataset and show the limitations of this model in contrast to ours on such an object-centric manipulation dataset. Our code and trained models are available at url{https://github.com/eneserdo/FGSE}.

Keywords: Intention Recognition, Prosthetics and Exoskeletons, Rehabilitation Robotics
Abstract: The growing prevalence of stroke necessitates advanced lower-limb exoskeleton control. This paper proposes HybridFusionAtt, a novel model for continuous joint angle estimation using surface electromyography (sEMG). Unlike conventional approaches, our framework uniquely integrates traditional time-domain features with CNN-extracted high-dimensional spatial features through an attention mechanism, where traditional features dynamically guide feature fusion as attention queries. The model was validated using data collected from eight participants performing four activities of daily living (walking, stair climbing, stair descending, and obstacle crossing). The proposed model achieves average R² values for knee and hip joint angle prediction of 0.8682 (walking), 0.8482 (obstacle crossing), 0.9294 (stair climbing), and 0.8676 (stair descending). Experimental results show that the proposed model significantly outperforms traditional LSTM and CNN-LSTM models in terms of accuracy and robustness, particularly in handling non-periodic actions such as obstacle crossing. The model achieves high performance by effectively fusing features and adaptively focusing on key features, enabling it to maintain robustness even under noisy conditions and significant individual differences. This demonstrates the model's broad application potential, especially in rehabilitation and prosthetic control systems.

Keywords: Intention Recognition, Rehabilitation Robotics
Abstract: The quantitative evaluation of the improvement of physical function is crucial for patients with impaired motor function, such as stroke, in conducting related rehabilitation training activities. Specially, a practical and easy-to-operate gait feature detection and extraction system for a home is urgently needed. In this study, a home gait feature extraction method based on the inertial measurement unit is proposed. The subjects' walking distance and speed are calculated using the double integral and the number of strides is calculated using the local maximum peak approach, while the stance phase and swing phase are calculated using the local trough approach. The compared result shows that the average walking distance accuracy is about 91.32% and the average stride accuracy is about 96.55%. The proportion of the stance period (59.01%) and swing period (40.99%) estimated by the inertial measurement unit is close to the ratio of the two at normal speed. The experimental results demonstrate that the great accuracy of the gait spatio-temporal features is retrieved. The proposed method facilitates gait evaluation in clinics and at home, including the extraction of gait features and real-time evaluation.

Keywords: Intention Recognition, Human-Robot Collaboration, Learning from Demonstration
Abstract: Learning from Demonstration (LfD) is a powerful type of machine learning that can allow novices to teach robots to complete various tasks. However, the learning process for these systems may still be difficult for novices to interpret and understand, making effective teaching challenging. Explainable artificial intelligence (XAI) aims to address this challenge by explaining a system to the user. In this work, we investigate XAI within LfD by implementing an adaptive explanatory feedback system on an inverse reinforcement learning (IRL) algorithm. The feedback is implemented by demonstrating selected learnt trajectories to users. The system adapts to user teaching by categorizing and then selectively sampling trajectories shown to a user, to show a representative sample of both successful and unsuccessful trajectories. The system was evaluated through a user study with 26 participants teaching a robot a multi-goal navigation task. The results of the user study demonstrated that the proposed explanatory feedback system can improve robot performance, teaching efficiency and the user's ability to predict the robot's goals and actions.

Keywords: Intention Recognition, Deep Learning Methods, Cognitive Modeling
Abstract: Accurate and fast motion prediction such as pedestrian motion prediction (PMP) is crucial for safe autonomous driving. Much research effort has been devoted to studying the reactive behaviors of pedestrians, such as the interaction between pedestrians or the interaction between pedestrians and the environment. However, compared with behavioral logic, the current motion state of pedestrians has a greater influence on the future trajectory. In this work, we propose a motion logic network (MLN) to improve both the accuracy and efficiency of pedestrian motion prediction. Compared with the traditional data-driven neural networks, the concept of motion logic is directly introduced into the network so that the training of the network is not required. In particular, instead of fitting the network based on inputs and target outputs, the proposed method directly adopts motion logic to predict the future trajectory. To illustrate the performance of the proposed MLN, several experiments have been performed. For acceleration and deceleration motion in practical experiment, the average displacement error (ADE) of MLN has an improvement of as high as 6.25% than CVM's, while the final displacement error (FDE) of MLN has an improvement of 11.1%. In terms of efficiency, MLN is a hundred times faster than LSTM. It indicates that motion logic plays an important role in the development of prediction algorithms for pedestrian motion.

Note to Practitioners-This article is motivated by the effects of pedestrians' different motion states on the future trajectory. For example, the pedestrians' state may also change drastically in some situations such as dashing across the road or stopping in anticipation of a bicycle crossing the path. This work explores the use of pedestrians' physical motion states to develop a network that emphasizes the logical features of pedestrian motion so as to directly estimate the future motion trajectory and trend based on the motion logic. For the first time, the concept of motion logic and the resultant training-free motion logic network (MLN) is proposed, which takes into account the accuracy and efficiency performance of the algorithm. Compared with the state-of-the-art pedestrian prediction methods, the proposed method can better predict the trajectory of pedestrians in more complex motion states. Moreover, a series of simulations and practical experiments with pseudo-constant velocity motion and acceleration/deceleration motion was taken to verify the performance of the proposed MLN.

Keywords: Industrial Robots, Manipulation Planning, Optimization and Optimal Control
Abstract: Time-optimal trajectory generation (TOTG) is critical in robotics applications to minimize travel time and increase robot task efficiency. To ensure the trajectory is feasible and executable by the robot, it is important to constrain the trajectory kinodynamics subject to the robot actuator limits. A typical actuator has multiple limits, 1) peak limit, and 2) multilevel continuous limits with different operation time windows. The peak limit bounds the instantaneous kinodynamics (IKD), whereas the continuous limits bound the system continuous kinodynamics (CKD). Existing works only constrain IKD, usually by the actuator peak limit, to achieve time optimality. However, a joint capable of operating at its peak limit momentarily will overheat and damage robot life if the motion continues. Alternatively, users can constrain the IKD with a reduced peak limit to avoid violating continuous limits. However, the reduced peak limit would inevitably sacrifice task efficiency. To address the challenge, this paper studies TOTG with both IKD and CKD, and proposes TOTG-C. It formulates the TOTG as a nonlinear programming (NLP). In particular, it proposes a novel formulation to encode the multi-level CKD constraints efficiently. To the best of our knowledge, TOTG-C is the first work that explicitly considers multi-level CKD constraints. We demonstrate the effectiveness and robustness of the proposed TOTG-C both in simulation and real robot experiments.

Keywords: Industrial Robots, SLAM, Performance Evaluation and Benchmarking
Abstract: Performance evaluation is critical for ensuring the accuracy, efficiency, and reliability of industrial robotic arms. Traditional measurement methods offer high precision but are often costly, complex to install, and constrained by environmental factors. To address these limitations, this study proposes a SLAM-based performance evaluation method that leverages LiDAR to track robotic motion without requiring external calibration references. This approach provides a cost-effective and flexible alternative to conventional metrology techniques. However, integrating SLAM into the ISO 9283 framework presents challenges related to accuracy, stability, and measurement consistency. To assess its feasibility, this study evaluates the SLAM-based system by analyzing key performance parameters, ensuring its alignment with industrial requirements. The results demonstrate that the LiDAR-based SLAM system achieves an RMSE of 0.0353 mm in trajectory estimation, confirming its precision and stability. These findings validate the system’s capability as a reliable benchmarking tool for robotic arm performance assessment.

Keywords: Hydraulic/Pneumatic Actuators, Force Control, Robust/Adaptive Control
Abstract: A “planning and control” scheme with hysteresis compensation is developed in this article for a compliant grinding device (CGD) driven by a pneumatic actuator with asymmetric full-state constraints. First, in the planning part, we propose an improved hysteresis model and construct its inverse model through the modified inverse multiplicative structure method to deal with the hysteresis nonlinearity between air pressure and contact force in CGD. In the control part, a neural state observer (SO) is developed to observe the rate of change of air pressure by combining a neural network (NN) and an SO. By fusing an NN and a disturbance observer (DO), a neural DO is presented to observe external disturbances. Asymmetric full-state constraints are realized without the feasibility conditions (FCs) of the virtual controller by the fusion of the backstepping design process and nonlinear state-dependent transformations. It is proven that the air pressure closely tracks the planned desired air pressure, as the tracking error of the air pressure is uniformly ultimately bounded. Finally, hardware experiments show that the force control accuracy of the proposed method is within 2N, which achieves a satisfactory force control effect and verifies the effectiveness and robustness of the presented method.

Keywords: Probabilistic Inference, Industrial Robots, Intelligent and Flexible Manufacturing
Abstract: Controller tuning and parameter optimization are crucial in system design to improve closed-loop system performance. Bayesian optimization has been established as an efficient model-free controller tuning and adaptation method. However, Bayesian optimization methods are computationally expensive and therefore difficult to use in real-time critical scenarios. In this work, we propose a real-time purely data-driven, model-free approach for adaptive control, by online tuning low-level controller parameters. We base our algorithm on GoOSE, an algorithm for safe and sample-efficient Bayesian optimization, for handling performance and stability criteria. We introduce multiple computational and algorithmic modifications for computational efficiency and parallelization of optimization steps. We further evaluate the algorithm’s performance on a real precision-motion system utilized in the semiconductor industry applications by modifying the payload and reference stepsize and comparing it to an interpolated constrained optimization-based baseline approach. Note to Practitioners—This work is motivated by developing a comprehensive control and optimization framework for high-precision motion systems. Precision motion is an integral application of advanced mechatronics and a cornerstone technology for high-value industrial processes such as semiconductor manufacturing. The proposed method framework relies on data-driven optimization methods that can be designed by prescribing desired system performance. By using a method based on Bayesian Optimization and safe exploration, our method optimizes desired parameters based on the prescribed system performance. A key benefit is the incorporation of input and output constraints, which are satisfied throughout the optimization procedure. Therefore, the method is suitable for use in practical systems where safety or operational constraints are of concern. Our method includes a variable to incorporate contextual information, which we name the task parameter. Using this variable, users can input external changes, such as changing step sizes for a motion system, or changing weight on top of the motion system. We provide two parallel implementation variants of our framework to make it suitable for run-time operation under context changes and applicable for continuous operation in industrial systems. We demonstrate the optimization framework on simulated examples and experiment on an industrial motion system to showcase its applicability in practice.

Keywords: Industrial Robots, Parallel Robots, Mechanism Design
Abstract: The parallel kinematic mechanism (PKM) is typically equipped with multiple actuators to realize the precise and arbitrary spatial motion, but resulting in complex mechanical and control systems and high cost. Actually, in many specific fields, such as heavy load metal forming process, the motion of PKM is only needed to be specific and the motion precision requirement is not extremely high. This paper proposes a new approximate mechanism synthesis method for PKM with single actuator (PKMSA), which not only can simplify the complexity of PKM and reduce the cost, but also can realize the specific motion with permissible error. Firstly, the design criteria of the consistency between the motion pattern of the PKMSA and required motion DoF is determined to avoid the occurrence of kinematic redundancy error. Then a general kinematic model for PKMSA is derived based on the screw theory, and the general constraints are obtained for PKMSA to realize the specific motion. On this basis, a 3-RSS/S PKMSA configured with a single input and triple output actuator layout realized by a gear set is proposed. Finally, a heavy load multi-DoF forming machine (load of 200kN) with PKMSA is developed and, with that, the multi-DoF forming experiment of a typical metal component is conducted with forming load of about 180kN. The geometric deviation of formed component is in the range of -40~55 um (it can completely meet the forming accuracy requirement), validating the feasibility of the proposed approximate mechanism synthesis method for PKMSA.

Keywords: Industrial Robots, Grippers and Other End-Effectors
Abstract: Vacuum grippers are popular for their simplicity, reliability, and strength. However, their rigidity limits them to grip only relatively flat objects. To design a soft vacuum gripper, leakage must be prevented through a suction cup, not in contact with the object. Various soft valves have been proposed that mimic the working principle of conventional rigid valves. Still, they have limitations, such as the inability to grip objects with various surface characteristics and slow gripping speed. In this paper, we propose a new soft valve for a versatile and fast soft vacuum gripper. The proposed valve is inspired by the working principle of the air fuse, and the feasibility of the mechanism has been experimentally confirmed. Two performance indices are proposed for the optimization of the proposed valve. Analytical models are proposed to predict the pressure conservation and operating time according to the geometric parameters of the proposed valve, and the validity of the model is experimentally confirmed. Finally, a soft vacuum gripper with the proposed valves was fabricated to verify that the proposed valve can grip objects with various surface characteristics rapidly.

Keywords: Industrial Robots, Human-Robot Collaboration, Physical Human-Robot Interaction
Abstract: In industrial collaborative robotics, safety, fluency as well as productivity are among the most important requirements to implement effective applications. In this work, a generalized formulation that combines the Power and Force Limiting and the Speed and Separation Monitoring criteria is firstly developed; then, a novel method that combines it with the possibility of executing an escape trajectory with

the aim to enhance the levels of fluency and performance while guaranteeing human safety is presented. In particular, starting from a reference trajectory assigned to the manipulator to execute a specific task, a suitable joint acceleration is searched by formulating the problem as a Quadratic Programming problem under linearized kinematic and safety constraints. The method is then tested and experimentally validated. Results show, through fluency metrics, the effectiveness of the developed method with respect to the Power and Force Limiting criterion, the Speed and Separation Monitoring criterion as well as their combination.

Keywords: Industrial Robots, Multi-Robot Systems, Planning, Scheduling and Coordination
Abstract: With the rapid development of intelligent manufacturing, multi-robot collaborative systems are increasingly integrated into various production processes. In the flexible job shop environment of automotive stamping, achieving smooth operation and efficient manufacturing of production lines hinges on solving the critical issues of multi-robot task allocation and scheduling. However, for such fixed-type multi-robot collaboration problems, robots are constrained by specific areas or predetermined trajectories, and processing times can only be adjusted by varying the number of available robots. Therefore, the scheduling problem in multi-robot collaborative flexible job shop problems (MCFJSP) is divided into two sub-problems: FJSP with controllable processing times and multi-robot collaborative task balancing. To address these, we propose three distinct methods: mixed integer linear programming (MILP), constraint programming (CP), and a hybrid genetic algorithm-constraint programming (GA-CP). Finally, a set of 48 benchmark cases and two real-world cases are developed to test these methods. Comparative experiments demonstrate that the MILP model is superior in small-scale cases, while the GA-CP model exhibits the best overall performance in medium to large-scale cases. Furthermore, through comparisons with two advanced algorithms, the effectiveness and superiority of the GA-CP method in addressing real-world cases are confirmed.[8pt]Note to Practitioners—In modern manufacturing environments, particularly in industries like automotive manufacturing, multiple robots working together on complex tasks are increasingly common. This paper addresses the practical challenge of effectively scheduling these robots to maximize efficiency while reducing the number of robots assigned to each task. This study introduces and compares different methods, including MILP, CP, and GA-CP methods, that can help practitioners determine the best way to allocate tasks among robots and schedule them efficiently. For example, in small-scale tasks, the MILP model can quickly provide the best solution. However, as the complexity and scale of the task increase, the GA-CP method becomes more practical, offering high-quality solutions within a reasonable timeframe. The study provides actionable insights that can be applied directly to real-world production scenarios, helping practitioners in industries like automotive stamping to maximize job shop productivity while reducing energy consumption losses in robot processing.

Keywords: Physical Human-Robot Interaction, Rehabilitation Robotics, Compliance and Impedance Control
Abstract: Robot-assisted rehabilitation focuses in part on path-based assist-as-needed reaching rehabilitation, which dynamically adapts the level of robot assistance during physical therapy to ensure patient progress along a predefined trajectory without over-reliance on the system. Additionally, bimanual exoskeletons have enabled asymmetric rehabilitation schemes, which leverage the patient's healthy side to guide the rehabilitation through interactions with objects in virtual reality that replicate activities of daily living. Within the context of physical human-robot interaction, these tasks can be formulated as constraints on the space of allowable motions. This study introduces a novel feedback linearization-inspired time-invariant admittance control scheme that enforces these motion constraints by isolating and stabilizing the component of the virtual dynamics transversal to the constraint. The methodology is applied to two rehabilitation tasks: (1) a path-guided reaching task with restoring force field, and (2) a bimanual interaction with a virtual object. Each task is then evaluated on one of two drastically different exoskeleton systems: (1) the V-Rex, a non-anthropomorphic full-body haptic device, and (2) the EXO-UL8, an anthropomorphic bimanual upper-limb exoskeleton. The two systems exist on opposite ends of the task/joint space control, non-redundant/redundant, off-the-shelf (industrial)/custom, non-anthropomorphic/anthropomorphic spectra. Experimental results validate and support the methodology as a generalizable approach to enabling constrained admittance control for rehabilitation robots.

Keywords: Deep Learning Methods, Physical Human-Robot Interaction, Machine Learning for Robot Control
Abstract: This paper presents a novel framework for Human Activity Recognition (HAR) by unifying all sensor streams, visual, audio, and inertial, into a single textual domain, enabling the direct application of GPT-3 for multimodal data classification. Unlike traditional pipelines that use dedicated encoders for each modality, we show that converting sensor outputs into text tokens offers both simplicity and a powerful proof-of-concept for large language models (LLMs). To further boost performance, we introduce a composite loss function combining cross-entropy, Kullback-Leibler divergence, total variation, and multimodal consistency terms, ensuring both temporal smoothness and cross-modal alignment. We conduct extensive experiments on the CMU-MMAC dataset, achieving up to {98%} accuracy and significantly outperforming baseline methods. We also demonstrate robustness under missing sensor streams via partial tokenization, maintaining strong performance despite sensor failures. These results highlight the potential of LLM-driven HAR for enhanced human-robot interaction in real-world scenarios, and pave the way for broader multimodal applications of next-generation language models.

Keywords: Physical Human-Robot Interaction, Model Learning for Control, Safety in HRI
Abstract: Robots that can physically interact with humans in a safe and comfortable manner have the potential to revolutionize application domains like home assistance and nursing care. However, to become long-term companions, such robots must learn user-specific preferences and adapt their behaviors in real time. We propose a Constrained Partially Observable Markov Decision Process framework for modeling human safety preferences over representative variables like force, velocity, and proximity. These variables are modeled as adaptive linear constraints, with a belief over their upper bounds that is updated online based on noisy human feedback. By modeling the belief as phase dependent, the model captures varying preferences across different task phases. The robot then solves a hierarchical optimization to select actions that respect both the learned constraints and robot motion limits. Our method does not require offline training data and can be applied directly to diverse physical interaction tasks and operation modes (tele-operated or autonomous). A pilot study shows that our approach effectively learns user preferences and improves perceived safety while reducing user effort compared to baselines.

Keywords: Physical Human-Robot Interaction, Physically Assistive Devices, Simulation and Animation
Abstract: Robot-assisted feeding has the potential to enhance the independence of individuals requiring assistance, yet the bite transfer process remains particularly challenging, especially for those with complex conditions. In this paper, we present ORBiT, a novel Real2Sim2Real framework designed to optimize bite transfer in robot-assisted feeding. By integrating motion capture-driven, high-fidelity soft-body simulation with systematic parameter tuning, ORBiT effectively replicates realistic head, neck and jaw dynamics during feeding interactions to provide a safe simulation-driven approach to optimize bite transfer strategies. In our approach, motion capture data drives a personalized dynamic head model that, together with a comprehensive parameter search over variables such as entry angle, exit angle, exit depth, height offset, and distance to mouth, identifies the bite transfer parameters that minimize contact forces on the user. The optimal parameters are then transferred to a real-world robotic system and validated through a pilot user study involving five subjects. Results from real user evaluations mirror the trends in simulation, indicating that bite transfer parameters, especially those related to entry and exit angles, substantially affect user comfort and overall satisfaction. Our findings validate that simulation-derived optimizations can effectively guide improvements in bite transfer strategies, laying the groundwork for a safe, personalized approach to robot-assisted feeding.

Keywords: Physical Human-Robot Interaction, Compliance and Impedance Control, Optimization and Optimal Control
Abstract: Variable impedance control is a control strategy widely used in physical human-robot collaboration (pHRC) and physical human-robot interaction (pHRI). Variable stiffness and damping parameters improve adaptability to changing environments and enhance safety in human-robot interaction. However, these adaptive parameters can compromise the stability of the system without proper management, particularly in dynamic environments. To address this, we propose a real-time parameter prediction method for variable impedance control using model predictive control (MPC) with Control Lyapunov Function (CLF). Unlike the method that sets the terminal constraint as the equilibrium position, the proposed method guarantees system stability even when parameters change or external disturbances occur, ensuring safe and adaptive robot behavior. Moreover, the infeasibility issue is resolved by applying CLF instead of relying on the equilibrium position. Furthermore, considering stability throughout the prediction horizon, the stability of the system is strictly guaranteed. The proposed method was validated through comparative experiments with the method that sets the terminal constraint as the equilibrium position in both simulations and real-world environments using the Franka Emika Panda robot. Through these experiments, the proposed controller demonstrated a significant reduction in parameter computation time, achieving approximately 97.13% and 96.20% faster computation in simulation tests compared to conventional method, while consistently ensuring stability under various disturbances including human interaction, tool vibration, and contact loss scenarios.

Keywords: Physical Human-Robot Interaction, Rehabilitation Robotics, Modeling and Simulating Humans
Abstract: Focus of attention is one of the most influential factors facilitating motor training performance. Most of robotic training methods have not well solved the negative effect of divided-attention on motor execution performance, resulting in limited rehabilitation efficiency for motor-cognitive dysfunction. In this study, we propose a novel visuomotor human-robot interaction framework by integrating a gaze-visual game and force-movement robot, to realize more efficient training for both attentional and motor function. An important novelty of this framework is to design a dynamical pattern recognition scheme for the hierarchical-coupled behavior of attentional and motor execution, to facilitate efficient human-robot interaction in both cognitive and motor perspectives. Specifically, an attentional-motor dynamical system modeling method is first developed by using the gaze, force and movement data collected from the human under different attentional-motor behavior. Then, an online dynamical pattern recognition scheme can be design with these models to online recognizing the human’s attentional and motor behavior states. The training robot system can dynamically adjust the parameters according to the recognition results, to guide the collaboration of both attentional and motor training. Experimental study are conducted to demonstrate the desired accuracy and efficiency of our designed approaches in attentional-motor behavior recognition and training.

Keywords: Physical Human-Robot Interaction, Model Learning for Control, Collision Avoidance
Abstract: With the advancement of robot manipulator technologies, developing custom-made low-cost manipulator arms has gained increasing popularity in the field of robotics. However, these low-cost systems often lack precise sensing and control capabilities, making reliable collision detection particularly critical to ensure safe operation. Popular methods of collision detection, which estimate external disturbances such as generalized Momentum Observer (MO) critically rely on an accurate dynamics model, and precise system identification requires joint torque sensors that are expensive for low-cost manipulators. This paper presents a novel approach to improve collision detection for low-cost robot manipulators where modeling error is present and torque sensing is not available. We propose a probabilistic framework using Gaussian Mixture Models (GMM) to capture friction and other unmodeled dynamics of a robot manipulator. Instead of explicitly identifying model parameters, GMM is created on the total residual torque of the MO while running an excitation trajectory. The GMM is then deployed in the main control loop to identify external disturbances from the residual torque of the same momentum observer. The approach is validated on a custom-made 4-Degrees-of-Freedom (DoF) robot arm with modeling error, unmodeled dynamics, and with only joint position measurement. Our method achieves reliable detection on both hard and soft collisions, demonstrating a reduction in the false positive rate by more than 50% compared to conventional MO-based methods with the same true positive rate on the same hardware.

Keywords: AI-Enabled Robotics, Control Architectures and Programming, Reactive and Sensor-Based Planning
Abstract: Precision is a crucial performance indicator for robot arms. During interacting with human, high precision enables a robot arm to be used effectively and safely, while low precision may lead to safety issues. Traditional methods for improving robot arm precision rely on error compensation. However, these methods are often not robust and lack adaptability. Learning-based methods offer greater flexibility and adaptability, while current researches show that they often fall short in achieving high precision and struggle to handle many scenarios requiring high precision. In this paper, we propose a novel high-precision robot arm manipulation framework based on online iterative learning and forward simulation, which can achieve positioning error (precision) less than end-effector physical minimum displacement. In other words, our proposed method can compensate for the precision-limitation of the hardware structure of the robot arms. Furthermore, we consider the joint angular resolution of the real robot arm, which is usually neglected in related works. A series of experiments on both simulation and real UR3 robot arm platforms demonstrate that our proposed method is effective and promising. The related code will be available soon.

Keywords: AI-Enabled Robotics, Task Planning, Manipulation Planning
Abstract: Leveraging Large Language Models (LLMs) to write policy code for controlling robots has gained significant attention. However, in long-horizon implicative tasks, this approach often results in API parameter, comments and sequencing errors, leading to task failure. To address this problem, we propose a collaborative Triple-S framework that involves multiple LLMs. Through In-Context Learning, different LLMs assume specific roles in a closed-loop Simplification-Solution-Summary process, effectively improving success rates and robustness in long-horizon implicative tasks. Additionally, a novel demonstration library update mechanism which learned from success allows it to generalize to previously failed tasks. We validate the framework in the Long-horizon Desktop Implicative Placement (LDIP) dataset across various baseline models, where Triple-S successfully executes 89% of tasks in both observable and partially observable scenarios. Experiments in both simulation and real-world robot settings further validated the effectiveness of Triple-S. Our code and dataset is available at: https://github.com/Ghbbbbb/Triple-S.

Keywords: AI-Enabled Robotics, Humanoid and Bipedal Locomotion, Machine Learning for Robot Control
Abstract: Training and deploying reinforcement learning (RL) policies for robots is a complex task, requiring careful design of reward functions, sim-to-real transfer, and performance evaluation across various robot configurations. These tasks traditionally demand significant human expertise and effort. To address these challenges, this paper introduces textit{Anybipe}, a novel, fully automated, end-to-end framework for training and deploying bipedal robots, leveraging large language models (LLMs) for reward function generation, while supervising model training, evaluation, and deployment. The framework integrates comprehensive quantitative metrics to assess policy performance, deployment effectiveness, and safety. Additionally, it allows users to incorporate prior knowledge and preferences, improving the accuracy and alignment of generated policies with expectations. We demonstrate how textit{Anybipe} reduces human labor while maintaining high levels of accuracy and safety, examined on three different bipedal robots, showcasing its potential for autonomous RL training and deployment.

Keywords: AI-Enabled Robotics, Model Learning for Control
Abstract: High maneuverability is essential to the autonomous operation of underwater robots. To achieve real-time maneuvering motion, the control strategy must take into account nonlinear hydrodynamic effects, which are extremely difficult to accurately capture during motion and therefore a balance must be struck between accuracy and real-time computational efficiency. Therefore, this paper proposes a data-driven approach to model the dynamics of the underwater robot using Sparse Identification of Nonlinear Dynamics (SINDy). Compared with existing works, our method does not require any physical prior knowledge and only uses a short period of onboard sensor data. Subsequently, the learned dynamic model is incorporated into a model predictive controller (MPC) to enable precise attitude control. Finally, the proposed method is implemented on our developed fully vectored propulsion underwater robot, and a series of attitude tracking experiments are conducted in an indoor water tank. Experimental results reveal that our approach significantly improves the model accuracy and reduces the attitude tracking errors by over 79% at a control frequency of 20 Hz, which proves the effectiveness and real-time performance of the method.

Keywords: AI-Enabled Robotics, Engineering for Robotic Systems
Abstract: Terrain classification is crucial for robotic navigation especially in unknown environment. Existing terrain classification methods usually have high requirements for environment conditions and robot motions, making them challenging to apply to real-world scenarios. In this paper, we develop a novel terrain classification system with the planar electrical capacitance tomography (ECT) sensor, which provides a non-contact, real-time, and cost-effective way for terrain classification. Specifically, we design a planar ECT sensor and integrate it at the bottom of a mobile robot. The proposed system leverages the collected capacitance measurements to reflect the inherent differences in dielectric permittivity across various terrain types. And a multilayer perception networks is used to fuse the collected capacitance and IMU measurements for classification. Additionally, a large scale ECT dataset including 10 different types of terrains is collected with the proposed system. Extensive experiments are conducted demonstrating the effectiveness and robustness of the proposed system.

Keywords: AI-Enabled Robotics, Task Planning, Embedded Systems for Robotic and Automation
Abstract: In recent years, lightweight large language models (LLMs) have garnered significant attention in the robotics field due to their low computational resource requirements and suitability for edge deployment. However, in task planning—particularly for complex tasks that involve dynamic semantic logic reasoning—lightweight LLMs have underperformed. To address this limitation, we propose a novel task planner, LightPlanner, which enhances the performance of lightweight LLMs in complex task planning by fully leveraging their reasoning capabilities. Unlike conventional planners that use fixed skill templates, LightPlanner controls robot actions via parameterized function calls, dynamically generating parameter values. This approach allows for fine-grained skill control and improves task planning success rates in complex scenarios. Furthermore, we introduce hierarchical deep reasoning. Before generating each action decision step, LightPlanner thoroughly considers three levels: action execution (feedback verification), semantic parsing (goal consistency verification), and parameter generation (parameter validity verification). This ensures the correctness of subsequent action controls. Additionally, we incorporate a memory module to store historical actions, thereby reducing context length and enhancing planning efficiency for long-term tasks. We train the LightPlanner-1.5B model on our LightPlan-3k dataset, which comprises action chains consisting of tasks with 2 to 8 steps. Experiments demonstrate that our model achieves the highest task success rate despite having the smallest number of parameters. In tasks involving spatial semantic reasoning, the success rate exceeds that of ReAct by 14.9%. Moreover, we demonstrate LightPlanner's potential to operate on edge devices.

Keywords: AI-Enabled Robotics, Natural Dialog for HRI, Deep Learning for Visual Perception
Abstract: Language-conditioned policies have recently gained substantial adoption in robotics as they allow users to specify tasks using natural language, making them highly versatile. While much research has focused on improving the action prediction of language-conditioned policies, reasoning about task descriptions has been largely overlooked. Ambiguous task descriptions often lead to downstream policy failures due to misinterpretation by the robotic agent. To address this challenge, we introduce AmbResVLM, a novel method that grounds language goals in the observed scene and explicitly reasons about task ambiguity. We extensively evaluate its effectiveness in both simulated and real-world domains, demonstrating superior task ambiguity detection and resolution compared to recent state-of-the-art baselines. Finally, real robot experiments show that our model improves the performance of downstream robot policies, increasing the average success rate from 69.6% to 97.1%. We make the data, code, and trained models publicly available at https://ambres.cs.uni-freiburg.de.

Keywords: AI-Enabled Robotics, Autonomous Agents, Vision-Based Navigation
Abstract: Robotic instruction following tasks require seamless integration of visual perception, task planning, target localization, and motion execution. However, existing task planning methods for instruction following are either data-driven or underperform in zero-shot scenarios due to difficulties in grounding lengthy instructions into actionable plans under operational constraints. To address this, we propose FlowPlan, a structured multi-stage LLM workflow that elevates zero-shot pipeline and bridges the performance gap between zero-shot and data-driven in-context learning methods. By decomposing the planning process into modular stages--task information retrieval, language-level reasoning, symbolic-level planning, and logical evaluation--FlowPlan generates logically coherent action sequences while adhering to operational constraints and further extracts contextual guidance for precise instance-level target localization. Benchmarked on the ALFRED and validated in real-world applications, our method achieves competitive performance relative to data-driven in-context learning methods and demonstrates adaptability across diverse environments. This work advances zero-shot task planning in robotic systems without reliance on labeled data. Project website: https://instruction-following-project.github.io/.

Keywords: Force Control, Machine Learning for Robot Control, Industrial Robots
Abstract: This paper proposes a novel Q-learning-based dual-loop force tracking control framework for robot grinding tasks in uncertain environments. A complete system state-space model is established, incorporating interaction dynamics and the desired force. By augmenting the system state, a discount cost function is defined to quantify the tracking errors of the force and reference trajectory. The modified Q-learning method is systematically designed to iteratively compute the optimal control gain in a model-free manner. To mitigate force overshoot during the transition from free space to contact space, a force reference model and a transition mechanism for the control gain are designed. Simulations and experiments validate the method's effectiveness in precise force tracking with minimal overshoot and robustness to environmental variations.

Keywords: Force Control, Actuation and Joint Mechanisms, Soft Sensors and Actuators
Abstract: This article proposes a method for controlling force and position in a synchronous direct-drive electrostatic linear motor. Using a spring-like behavior of synchronous motors, the proposed controller regulates the contact force in a manner similar to that of series elastic actuators. Discussions regarding similarities and differences between series elastic actuators and the proposed method imply that their dynamic behaviors are different. The proposed controller consists of a force control part and a position control part, one of which is automatically selected based on the operating conditions. The position and force controllers each independently command the velocity based on the position or force error. The lower of the two velocities is selected, allowing a smooth automatic transition between the two control modes. Based on the selected velocity, the driving signal is calculated under the assumption of synchronous operation and fed to the actuator. The proposed control method is simulated and experiments are performed using prototype electrostatic linear motors. The experimental results confirm the smooth transition between the position and force control modes, as well as the good controllability of the contact force.

Keywords: Formal Methods in Robotics and Automation, Autonomous Vehicle Navigation
Abstract: Simultaneously accurate and reliable tracking control for quadrotors in complex dynamic environments is challenging. The chaotic nature of aerodynamics, derived from drag forces and moment variations, makes precise identification difficult. Consequently, many existing quadrotor tracking systems treat these aerodynamic effects as simple `disturbances' in conventional control approaches. We propose a novel and interpretable trajectory tracker integrating a distributional Reinforcement Learning (RL) disturbance estimator for unknown aerodynamic effects with a Stochastic Model Predictive Controller (SMPC). Specifically, the proposed estimator `Constrained Distributional REinforced-Disturbance-estimator' (ConsDRED) effectively identifies uncertainties between the true and estimated values of aerodynamic effects. Control parameterization employs simplified affine disturbance feedback to ensure convexity, which is seamlessly integrated with the SMPC. We theoretically guarantee that ConsDRED achieves an optimal global convergence rate, and sublinear rates if constraints are violated with certain error decreases as neural network dimensions increase. To demonstrate practicality, we show convergent training, in simulation and real-world experiments, and empirically verify that ConsDRED is less sensitive to hyperparameter settings compared with canonical constrained RL. Our system substantially improves accumulative tracking errors by at least 70%, compared with the recent art. Importantly, the proposed ConsDRED-SMPC framework balances the trade-off between pursuing high performance and obeying conservative constraints for practical implementations.

Keywords: Formal Methods in Robotics and Automation, Robot Safety, Reinforcement Learning
Abstract: Incorporating formal methods into reinforcement learning (RL) has the potential to result in the best of both worlds, combining the robustness of formal guarantees with the adaptability and learning capabilities of RL, though careful design is needed to balance safety and exploration. In this work, we propose a framework to mitigate this loss of exploration while still allowing for the safety of the system to be ensured. Specifically, we introduce a less restrictive method that can reduce the conservativeness of formal methods by refining a disturbance model using online collected data and it evaluates the safety of a learning-based controller, using computationally efficient zonotopic reachability analysis for the safety analysis to facilitate a real-time implementation. We validate the framework in a real-world drone flight through a canyon, where the drone is subjected to unknown external disturbances and the framework is tasked with learning those disturbances online and adjusting the safety guarantees accordingly. The results show that the framework enables a less restrictive online training of learning-based controllers without compromising the safety of the system.

Keywords: Formal Methods in Robotics and Automation, Autonomous Vehicle Navigation
Abstract: Human drivers naturally balance the risks of different concerns while driving, including traffic rule violations, minor accidents, and fatalities. However, achieving the same behavior in autonomous driving systems remains an open problem. This paper extends a risk metric that has been verified in human-like driving studies to encompass more complex driving scenarios specified by linear temporal logic (LTL) that go beyond just collision risks. This extension incorporates the timing and severity of events into LTL specifications, thereby reflecting a human-like risk awareness. Without sacrificing expressivity for traffic rules, we adopt LTL specifications composed of safety and co-safety formulas, allowing the control synthesis problem to be reformulated as a reachability problem. By leveraging occupation measures, we further formulate a linear programming (LP) problem for this LTL-based risk metric. Consequently, the synthesized policy balances different types of driving risks, including both collision risks and traffic rule violations. The effectiveness of the proposed approach is validated by three typical traffic scenarios in Carla simulator.

Keywords: Formal Methods in Robotics and Automation, Reactive and Sensor-Based Planning, Robot Safety
Abstract: Control Barrier Functions (CBFs) have emerged as an effective and non-invasive safety filter for ensuring the safety of autonomous systems in dynamic environments with formal guarantees. However, most existing works on CBF synthesis focus on fully known settings. Synthesizing CBFs online based on perception data in unknown environments poses particular challenges. Specifically, this requires the construction of CBFs from high-dimensional data efficiently in real time. This paper proposes a new approach for online synthesis of CBFs directly from local Occupancy Grid Maps (OGMs). Inspired by steady-state thermal fields, we show that the smoothness requirement of CBFs corresponds to the solution of the steady-state heat conduction equation with suitably chosen boundary conditions. By leveraging the sparsity of the coefficient matrix in Laplace’s equation, our approach allows for efficient computation of safety values for each grid cell in the map. Simulation and real-world experiments demonstrate the effectiveness of our approach. Specifically, the results show that our CBFs can be synthesized in an average of milliseconds on a 200×200 grid map, highlighting its real-time applicability.

Keywords: Formal Methods in Robotics and Automation, Robot Safety, Dynamics
Abstract: This paper presents a guaranteed model-based approach for monitoring drone trajectory, providing real-time guarantees with a simplified dynamic model. ROS components are introduced for real-time implementation, enabling monitoring and adjustments in both simulations and actual systems. We extend the application of set-based simulation by formalizing timing conditions with Signal Temporal Logic (STL) and incorporating Boolean interval arithmetic to handle undetermined behaviors. The method compares model-based fault prediction using a stochastic approach with a set-based method, which manages bounded uncertainties and offers guarantees. Experimental validation, including comparisons against Monte Carlo methods, demonstrates the approach ability to ensure safety in worst-case scenarios while remaining suitable for real-time processing.

Keywords: Deep Learning in Grasping and Manipulation, Imitation Learning
Abstract: Grasp-based manipulation tasks are fundamental to robots interacting with their environments, yet gripper state ambiguity significantly reduces the robustness of imitation learning policies for these tasks. Data-driven solutions face the challenge of high real-world data costs, while simulation data, despite its low costs, is limited by the sim-to-real gap. We identify the root cause of gripper state ambiguity as the lack of tactile feedback. To address this, we propose a novel approach employing pseudo-tactile as feedback, inspired by the idea of using a force-controlled gripper as a tactile sensor. This method enhances policy robustness without additional data collection and hardware involvement, while providing a noise-free binary gripper state observation for the policy and thus facilitating pure simulation learning to unleash the power of simulation. Experimental results across three real-world grasp-based tasks demonstrate the necessity, effectiveness, and efficiency of our approach.

Keywords: Deep Learning in Grasping and Manipulation, Deep Learning for Visual Perception, Deep Learning Methods
Abstract: Achieving zero-shot peg insertion, where inserting an arbitrary peg into an unseen hole without task-specific training, remains a fundamental challenge in robotics. This task demands a highly generalizable perception system capable of detecting potential holes, selecting the correct mating hole from multiple candidates, estimating its precise pose, and executing insertion despite uncertainties. While learning-based methods have been applied to peg insertion, they often fail to generalize beyond the specific peg-hole pairs encountered during training. Recent advancements in Vision-Language Models (VLMs) offer a promising alternative, leveraging large-scale datasets to enable robust generalization across diverse tasks. Inspired by their success, we introduce a novel zero-shot peg insertion framework that utilizes a VLM to identify mating holes and estimate their poses without prior knowledge of their geometry. Extensive experiments demonstrate that our method achieves 90.2% accuracy, significantly outperforming baselines in identifying the correct mating hole across a wide range of previously unseen peg-hole pairs, including 3D-printed objects, toy puzzles, and industrial connectors. Furthermore, we validate the effectiveness of our approach in a real-world connector insertion task on a backpanel of a PC, where our system successfully detects holes, identifies the correct mating hole, estimates its pose, and completes the insertion with a success rate of 88.3%. These results highlight the potential of VLM-driven zero-shot reasoning for enabling robust and generalizable robotic assembly.

Keywords: Deep Learning in Grasping and Manipulation, Grasping, Data Sets for Robot Learning
Abstract: Robotic grasping, crucial for robot interaction with objects, still struggles with counter-intuitive or long-tailed scenarios like uncommon materials and shapes. Humans, however, intuitively adjust grasps with their physics-informed interpretations of the object, using visual and linguistic cues. This work introduces PhyGrasp, a large multimodal model and dataset that enhance robotic manipulation by combining natural language and 3D point clouds using a bridge module to integrate these inputs. The language modality exhibits robust reasoning capabilities concerning the impacts of diverse physical properties on grasping, while the 3D modality comprehends object shapes and parts. With these two capabilities, PhyGrasp is able to accurately assess the physical properties of object parts and determine optimal grasping poses. Additionally, the model’s language comprehension enables human instruction interpretation, generating grasping poses that align with human preferences. To train PhyGrasp, we construct a dataset PhyPartNet with 195K object instances with varying physical properties and human preferences, alongside their corresponding language descriptions. Extensive experiments conducted in the simulation and on the real robots demonstrate that PhyGrasp achieves state-of-the-art performance, particularly in long-tailed cases, e.g., about 10% improvement in success rate over GraspNet. More demos and information are available on https://sites.google.com/view/phygrasp

Keywords: AI-Based Methods, Deep Learning in Grasping and Manipulation, Task and Motion Planning
Abstract: Zero-shot generalization across various robots, tasks and environments remains a significant challenge in robotic manipulation. Policy code generation methods use executable code to connect high-level task descriptions and low-level action sequences, leveraging the generalization capabilities of large language models and atomic skill libraries. In this work, we propose Robotic Programmer (RoboPro), a robotic foundation model, enabling the capability of perceiving visual information and following free-form instructions to perform robotic manipulation with policy code in a zero-shot manner. To address low efficiency and high cost in collecting runtime code data for robotic tasks, we devise Video2Code to synthesize executable code from extensive videos in-the-wild with off-the-shelf vision-language model and code-domain large language model. Extensive experiments show that RoboPro achieves the state-of-the-art zero-shot performance on robotic manipulation in both simulators and real-world environments. Specifically, the zero-shot success rate of RoboPro on RLBench surpasses Code-as-Policies equipped with the state-of-the-art model GPT-4o by 11.6%. Furthermore, RoboPro is robust to variations on API formats and skill sets. Our website can be found at https://video2code.github.io/RoboPro-website/.

Keywords: Deep Learning in Grasping and Manipulation, Perception for Grasping and Manipulation, Grasping
Abstract: We propose VISO-Grasp, a novel vision-language-informed system designed to systematically address visibility constraints for grasping in severely occluded environments. By leveraging Foundation Models (FMs) for spatial reasoning and active view planning, our framework constructs and updates an instance-centric representation of spatial relationships, enhancing grasp success under challenging occlusions. Furthermore, this representation facilitates active Next-Best-View (NBV) planning and optimizes sequential grasping strategies when direct grasping is infeasible. Additionally, we introduce a multi-view uncertainty-driven grasp fusion mechanism that refines grasp confidence and directional uncertainty in real-time, ensuring robust and stable grasp execution. Extensive real-world experiments demonstrate that VISO-Grasp achieves a success rate of 87.5% in target-oriented grasping with the fewest grasp attempts outperforming baselines. To the best of our knowledge, VISO-Grasp is the first unified framework integrating FMs into target-aware active view planning and 6-DoF grasping in environments with severe occlusions and entire invisibility constraints. Code is available at: https://github.com/YitianShi/vMF-Contact

Keywords: Deep Learning in Grasping and Manipulation, AI-Enabled Robotics
Abstract: Language-driven grasp detection has the potential to revolutionize human-robot interaction by allowing robots to understand and execute grasping tasks based on natural language commands. However, existing approaches face two key challenges. First, they often struggle to interpret complex text instructions or operate ineffectively in densely cluttered environments. Second, most methods require a training or fine-tuning step to adapt to new domains, limiting their generation in real-world applications. In this paper, we introduce GraspMAS, a new multi-agent system framework for language-driven grasp detection. GraspMAS is designed to reason through ambiguities and improve decision-making in real-world scenarios. Our framework consists of three specialized agents: Planner, responsible for strategizing complex queries; Coder, which generates and executes source code; and Observer, which evaluates the outcomes. Intensive experiments on two large-scale public datasets demonstrate that our GraspMAS significantly outperforms existing baselines. Additionally, robot experiments conducted in both simulation and real-world settings further validate the effectiveness of our approach.

Keywords: Deep Learning in Robotics and Automation, Computer Vision for Other Robotic Applications, Sensor Fusion, Visual Tracking
Abstract: We present PK-ROKED, a learning-based pipeline for probabilistic robot pose estimation relative to a camera, addressing inaccuracies in forward kinematics, particularly in systems with elastic and lightweight modules. Our approach integrates a probabilistic 2D keypoint detection mechanism that leverages prior knowledge derived from the robot’s imprecise kinematics. We further improve the detection accuracy and geometric understanding by incorporating segmentation of the robot arm. The method computes reliable uncertainty estimates, enabling a robust 2D-6D fusion for precise robot arm pose estimation from a single detected keypoint. PK-ROKED requires only synthetic training data, effectively exploits imperfect kinematics as valuable prior knowledge, and introduces a novel fusion framework for enhanced robot pose estimation. We validate our method on the Panda-Orb dataset, demonstrating competitive performance against state-of-the-art approaches. Additionally, we evaluate on two other robotic systems in real-world scenarios and show its practicality by using the predictions to initialize a tracking algorithm. Code and pre-trained models will be made available.

Keywords: Human Performance Augmentation, Mechanism Design, Wearable Robotics
Abstract: The execution of long-distance load-carrying tasks across multiple terrains remains a frequent requirement. These tasks often involve heavy loads, resulting in fatigue, decreased efficiency, and potential safety risks. To address this issue, this paper proposes a wearable centaur robot with wheel-legged transformation for human load-carrying assistance. The key feature of this robotic mechanism is the independent wheel-legged transformable structure, enabling transitions between the wheeled and legged modes. The wheeled mode ensures high load-carrying efficiency, while in the legged mode, the wheels are laid flat, transforming the ankle joint into a locked support surface that provides stable gait support. This design enables efficient and stable load carriage over complex terrains, all while preserving the natural gait of the user. Next, we develop a unified control framework for human-robot collaborative locomotion across different terrains, which includes velocity control based on an admittance model for the wheeled mode, gait control using a Bézier trajectory for the legged mode, and the transition between the two modes. The preliminary experiments include wheeled-mode, legged-mode, mode transition and obstacle crossing under human-robot collaborative locomotion, validating the proposed robot's adaptability to different terrains while assisting with human load carriage.

Keywords: Human Performance Augmentation, Prosthetics and Exoskeletons, Marine Robotics
Abstract: Underwater assistance is crucial for individuals who depend on diving for their livelihood. In this paper, we propose a novel underwater exosuit actuator designed to assist with flutter kicking during diving, thereby decreasing the effort the diver has to exert. The actuator can provide bidirectional assistance to the up and downbeats when the diver kicks underwater, and has no restriction on leg movements when it is deactivated. Both the benchtop experiment and human subject tests were conducted to verify its performance. The benchtop experiment verified its kinematic features, while tests with five participants validated its assistive performance. The results indicate that the actuator delivers a peak torque of 0.0947 Nm/kg and a peak force of 100 N in both directions, while allowing free leg movement during walking or kicking when not powered, thus ensuring safety during diving.

Keywords: Reinforcement Learning, Human Factors and Human-in-the-Loop, Machine Learning for Robot Control
Abstract: Preference-based Reinforcement Learning (PbRL) methods provide a solution to avoid reward engineering by learning reward models based on human preferences. However, poor feedback- and sample- efficiency still remain the problems that hinder the application of PbRL. In this paper, we present a novel efficient query selection and preference-guided exploration method, called SENIOR, which could select the meaningful and easy-to-comparison behavior segment pairs to improve human feedback-efficiency and accelerate policy learning with the designed preference-guided intrinsic rewards. Our key idea is twofold: (1) We designed a Motion-Distinction-based Selection scheme (MDS). It selects segment pairs with apparent motion and different directions through kernel density estimation of states, which is more task-related and easy for human preference labeling; (2) We proposed a novel preference-guided exploration method (PGE). It encourages the exploration towards the states with high preference and low visits and continuously guides the agent achieving the valuable samples. The synergy between the two mechanisms could significantly accelerate the progress of reward and policy learning. Our experiments show that SENIOR outperforms other five existing methods in both human feedback-efficiency and policy convergence speed on six complex robot manipulation tasks from simulation and four real-worlds.Videos can be found on our project website:https://2025senior.github.io/

Keywords: Human Performance Augmentation, Haptics and Haptic Interfaces, Human-Robot Collaboration
Abstract: This study investigates the role of haptic feedback in a car-following scenario, where information about the motion of the front vehicle is provided through a virtual elastic connection with it. Using a robotic interface in a simulated driving environment, we examined the impact of varying levels of such haptic feedback on the driver's ability to follow the road while avoiding obstacles. The results of an experiment with 15 subjects indicate that haptic feedback from the front car's motion can significantly improve driving control (i.e., reduce motion jerk and deviation from the road) and reduce mental load (evaluated via questionnaire). This suggests that haptic communication, as observed between physically interacting humans, can be leveraged to improve safety and efficiency in automated driving systems, warranting further testing in real driving scenarios.

Keywords: Human Performance Augmentation, Physical Human-Robot Interaction, Human-Robot Collaboration
Abstract: Through the use of robotic supernumerary limbs, it has been proposed that a single user could perform tasks like surgery or industrial assembly that currently require a team. Although validation studies, often conducted in virtual reality, have demonstrated that individuals can learn to command supernumerary limbs, comparisons typically suggest that a team initially outperforms a supernumerary limb operating individual. In this study, we examined (i) the impact of using a commercially available physical robot setup instead of a virtual reality system and (ii) the effect of differences between limb couplings on user performance during a series of trimanual operations. Contrary to previous findings, our results indicate no clear difference in user performance when working as a trimanual user, in the pick and place of three objects, compared to when working as a team. Additionally, for this task we observe that while users prefer working with a partner when they control the majority of the limbs, we find no clear difference in their preference between solo trimanual operation and when they work with a partner and control the third limb. These findings indicate that factors typically not present in virtual reality such as visual occlusion and haptic feedback may be vital to consider for the effective operation of supernumerary limbs, and provide initial evidence to support the viability of supernumerary limbs for a range of physical tasks.

Keywords: Human Performance Augmentation, Prosthetics and Exoskeletons, Mechanism Design
Abstract: Wearable assistive devices (WADs) show potential for future applications in industrial and medical fields. However, wearing discomfort, caused by factors such as additional weight penalties, inappropriate straps, and joint misalignment, may disrupt the natural balance of human motion, leading to increased metabolic expenditure, muscle fatigue, and even physical injury. These challenges severely restrict the practical application of WADs. To address these challenges, we have developed a reactionless, lightweight WAD that utilizes a single device attached to the shoe, eliminating the need for additional rigid links or straps on the shank or thigh. This design provides assistive torque directly to the target limbs during human locomotion. The proposed device weighs 0.67 kg and can provide an assistive torque of 12 Nm and mechanical power of 10 W. Furthermore, we introduced a human-in-the-loop multi-objective optimization approach considering the assistance effectiveness, power consumption, system size, service safety, and actuator constraints to achieve relative optimality in performance of the proposed device. Cardiopulmonary exercise test and surface electromyography test results from seven participants indicated that the device can reduce the average gross metabolic rate by 12.62%, and the muscle activation of the rectus femoris and soleus by 35.12% and 4.10%, respectively.

Keywords: Human-Aware Motion Planning, Safety in HRI, Reinforcement Learning
Abstract: Autonomous mobile robots are increasingly used in pedestrian-rich environments where safe navigation and appropriate human interaction are crucial. While Deep Reinforcement Learning (DRL) enables socially integrated robot behavior, challenges persist in novel or perturbed scenarios to indicate when and why the policy is uncertain. Unknown uncertainty in decision-making can lead to collisions or human discomfort and is one reason why safe and risk-aware navigation is still an open problem. This work introduces a novel approach that integrates aleatoric, epistemic} and predictive uncertainty estimation into a DRL navigation framework for policy distribution uncertainty estimates. We, therefore, incorporate Observation-Dependent Variance (ODV) and dropout into the Proximal Policy Optimization (PPO) algorithm. For different types of perturbations, we compare the ability of deep ensembles and Monte-Carlo Dropout (MC-Dropout) to estimate the uncertainties of the policy. In uncertain decision-making situations, we propose to change the robot's social behavior to conservative collision avoidance. The results show improved training performance with ODV and dropout in PPO and reveal that the training scenario has an impact on the generalization. In addition, MC-Dropout is more sensitive to perturbations and correlates the uncertainty type to the perturbation better. With the safe action selection, the robot can navigate in perturbed environments with fewer collisions.

Keywords: Human-Robot Collaboration, Grasping, Human-Centered Robotics
Abstract: The existing language-driven grasping methods struggle to fully handle ambiguous instructions containing implicit intents. To tackle this challenge, we propose LangGrasp, a novel language-interactive robotic grasping framework. The framework integrates fine-tuned large language models (LLMs) to leverage their robust commonsense understanding and environmental perception capabilities, thereby deducing implicit intents from linguistic instructions and clarifying task requirements along with target manipulation objects. Furthermore, our designed point cloud localization module, guided by 2D part segmentation, enables partial point cloud localization in scenes, thereby extending grasping operations from coarse-grained object-level to fine-grained part-level manipulation. Experimental results show that the LangGrasp framework accurately resolves implicit intents in ambiguous instructions, identifying critical operations and target information that are unstated yet essential for task completion. Additionally, it dynamically selects optimal grasping poses by integrating environmental information. This enables high-precision grasping from object-level to part-level manipulation, significantly enhancing the adaptability and task execution efficiency of robots in unstructured environments. More information and code are available here: https://github.com/wu467/LangGrasp.

Keywords: Human-Robot Collaboration, Human Factors and Human-in-the-Loop, Optimization and Optimal Control
Abstract: This paper introduces an upper limb postural optimization method for enhancing physical ergonomics and force manipulability during bimanual human-robot co-carrying tasks. Existing research typically emphasizes human safety or manipulative efficiency, whereas our proposed method uniquely integrates both aspects to strengthen collaboration across diverse conditions (e.g., different grasping postures of humans, and different shapes of objects). Specifically, the joint angles of a simplified human skeleton model are optimized by minimizing the cost function to prioritize safety and manipulative capability. To guide humans towards the optimized posture, the reference end-effector poses of the robot are generated through a transformation module. A bimanual model predictive impedance controller (MPIC) is proposed for our human-like robot, CURI, to recalibrate the end effector poses through planned trajectories. The proposed method has been validated through various subjects and objects during human-human collaboration (HHC) and human-robot collaboration (HRC). The experimental results demonstrate significant improvement in muscle conditions by comparing the activation of target muscles before and after optimization.

Keywords: Human-Robot Collaboration, Whole-Body Motion Planning and Control, Mobile Manipulation
Abstract: Human-robot collaborative manipulation with mobile, multiple minipulators is crucial for expanding robotic applications, requiring precise handling of coupled force position constraints between partners. Current systems, however, exhibit end-effector oscillations and instability during dynamic interactions. To overcome these limitations, this work develops a collaborative framework integrating a collaborative controller and a whole-body controller. The collaborative controller employs the object’s center-of-mass dynamics model with real-time contact forces and motion states to predict trajectories while coordinating with an attitude stabilization controller to adjust the desired end-effector poses. The whole-body controller utilizes model predictive control to generate coordinated motions that strictly follow pose commands from the collaborative controller, ensuring stable transportation. Simulation and physical experiments validate the proposed framework’s effectiveness in real-world scenarios.

Keywords: Human-Robot Collaboration, Multi-Modal Perception for HRI, Human-Centered Automation
Abstract: In recent years, there has been a significant amount of research on algorithms and control methods for distributed collaborative robots. However, the emergence of collective behavior in a swarm is still difficult to predict and control. Nevertheless, human interaction with the swarm helps render the swarm more predictable and controllable, as human operators can utilize intuition or knowledge that is not always available to the swarm. Therefore, this paper designs the Dynamic Visualization Research Platform for Multimodal Human-Swarm Interaction (DVRP-MHSI), which is an innovative open system that can perform real-time dynamic visualization and is specifically designed to accommodate a multitude of interaction modalities (such as brain-computer, eye-tracking, electromyographic, and touch-based interfaces), thereby expediting progress in human-swarm interaction research. Specifically, the platform consists of custom-made low-cost omnidirectional wheeled mobile robots, multitouch screens and two workstations. In particular, the mutitouch screens can recognize human gestures and the shapes of objects placed on them, and they can also dynamically render diverse scenes. One of the workstations processes communication information within robots and the other one implements human-robot interaction methods. The development of DVRP-MHSI frees researchers from hardware or software details and allows them to focus on versatile swarm algorithms and human-swarm interaction methods without being limited to predefined and static scenarios, tasks, and interfaces. The effectiveness and potential of the platform for human-swarm interaction studies are validated by several demonstrative experiments.

Keywords: Human-Robot Collaboration, Physical Human-Robot Interaction, Robot Safety
Abstract: By mapping the performances of the task requirement and the inherent physical characteristics of the robotic systems to the hybrid constraints, this paper proposes a task-oriented adaptive position/force control (TOAPFC) scheme for the robotic systems to ensure the execution of the predefined tasks and the safety of robotic manipulators and humans in the task workspace. In the proposed scheme, a reference trajectory generation strategy and admittance model are regarded as the outer-loop of TOAPFC to obtain and shape the robotic system's task trajectory that guarantees the safety of the interaction system. An admittance-based adaptive position/force control scheme unifying the position and force into a control law is used as the inner-loop of TOAPFC to track the shaped task trajectory, where a barrier Lyapunov function is utilized to constrain the tracking errors within permitted ranges. Moreover, the system uncertainties and lumped disturbances are compensated by the radial basis function neural network and robust compensator, respectively. Meanwhile, the stability of the proposed admittance-based adaptive position/force control scheme is analyzed by using the Lyapunov stability theory. Finally, the effectiveness of the proposed scheme is verified by experiments on a real robotic system.

Keywords: Human-Robot Collaboration, Reactive and Sensor-Based Planning, Planning, Scheduling and Coordination
Abstract: The recent emergence of Industry 5.0 underscores the need for increased autonomy in human–robot interaction (HRI), presenting both motivation and challenges in achieving resilient and energy-efficient production systems. To address this, in this article, we introduce a strategy for seamless collaboration between humans and robots in manufacturing and maintenance tasks. Our method enables smooth switching between temporary HRI (human-aware mode) and long-horizon automated manufacturing (fully automatic mode), effectively solving the human–robot coexistence problem. We develop a task progress monitor that decomposes complex tasks into robot-centric action sequences, further divided into three-phase subtasks. A trigger signal orchestrates mode switches based on detected human actions and their contribution to the task. In addition, we introduce a human agent coefficient matrix, computed using selected environmental features, to determine cut-points for reactive execution by each robot. To validate our approach, we conducted extensive experiments involving robotic manipulators performing representative manufacturing tasks in collaboration with humans. The results show promise for advancing HRI, offering pathways to enhancing sustainability within Industry 5.0. Our work lays the foundation for intelligent manufacturing processes in future societies, marking a pivotal step toward realizing the full potential of human–robot collaboration.

Keywords: Human-Robot Collaboration, Imitation Learning, Learning from Demonstration
Abstract: Learning grinding skills from human craftsmen by imitation learning has emerged as a prominent research topic in the field of robotic machining. Given their robust trajectory generalization ability and resilience to various external disturbances and environmental changes, Dynamical Movement Primitives (DMPs) provide a promising skills learning solution for the robotic grinding. However, challenges arise when directly applying DMPs to grinding tasks, including low orientation accuracy, inaccurate synchronization of position, orientation, and force, and the inability to generalize surface trajectories. To address these issues, this paper proposes a robotic grinding skills learning method based on geodesic length DMPs (Geo-DMPs). First, a normalized two-dimensional weighted Gaussian kernel function and intrinsic mean clustering algorithm are proposed to extract surface geometric features from multiple demonstration trajectories. Then, an orientation manifold distance metric is introduced to exclude the time factor from the classical orientation DMPs, thereby constructing Geo-DMPs for the orientation learning to improve the orientation trajectory generation accuracy. On this basis, a synchronization encoding framework for position, orientation, and force skills is established, using a phase function related to geodesic length. This framework enables the generation of robotic grinding actions between any two points on the surface. Finally, experiments on robotic chamfer grinding and free-form surface grinding demonstrate that the proposed method exhibits high geometric accuracy and good generalization capabilities in encoding and generating grinding skills. This method holds significant implications for learning and promoting robotic grinding skills. To the best of our knowledge, this may be the first attempt to use DMPs to generate grinding skills for position, orientation, and force on model-free surfaces, thereby presenting a novel approach to robotic grinding skills learning.

Keywords: Human-Robot Collaboration, Human Factors and Human-in-the-Loop, Gesture, Posture and Facial Expressions
Abstract: This work aims to interpret human behavior to anticipate potential user confusion when a robot provides explanations for failure, allowing the robot to adapt its explanations for more natural and efficient collaboration. Using a dataset [1] that included facial emotion detection, eye gaze estimation, and gestures from 55 participants in a user study [2], we analyzed how human behavior changed in response to different types of failures and varying explanation levels. Our goal is to assess whether human collaborators are ready to accept less detailed explanations without inducing confusion. We formulate a data-driven predictor to predict human confusion during robot failure explanations. We also propose and evaluate a mechanism, based on the predictor, to adapt the explanation level according to observed human behavior. The promising results from this evaluation indicate the potential of this research in adapting a robot’s explanations for failures to enhance the collaborative experience.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception, Industrial Robots
Abstract: Regular temperature measurement of critical parts of an annealing furnace has always been a difficult task. Due to the harsh environment of high temperature, high noise, and darkness in the annealing furnace operation area, unmanned vehicles equipped with the RGB-T semantic segmentation model are usually adopted in most factories for inspection. However, existing RGB-T semantic segmentation models usually rely on good lighting or thermal conditions, which are generally difficult to fulfill in annealing furnace operation areas. In this paper, we propose a new hierarchical fusion-based semantic enhancement network, HFSENet. We first adopt the two-stream structure and the siamese structure to extract the low-level and high-level features of unimodal modalities, respectively. Then, considering the differences between the features in different hierarchical levels, we introduce a novel low-level feature spatial fusion module and a high-level feature channel fusion module to perform the multi-modal feature hierarchical fusion. On this basis, we also propose the semantic feature complementary enhancement module, which utilizes the appearance information set and object information set extracted from RGB and thermal infrared (TIR) branches to enhance the fused features and give them more semantic information. Finally, segmentation results with refined edges are obtained by an edge refinement decoder that includes a local search extraction module. The unmanned inspection vehicle we built with the proposed HFSENet has successfully passed the test, and the recognition performance of the four targets exceeds the current state-of-the-art (SOTA) method on our homemade annealing furnace operation area dataset.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception, Autonomous Vehicle Navigation
Abstract: Autonomous systems rely on accurate 3D object detection from LiDAR data, yet most detectors are limited to a predefined set of known classes, making them vulnerable to unexpected out-of-distribution (OOD) objects. In this work, we present HD-OOD3D, a novel two-stage method for detecting unknown objects. We demonstrate the superiority of two-stage approaches over single-stage methods, achieving more robust detection of unknown objects. Furthermore, we conduct an in-depth analysis of the standard evaluation protocol for OOD detection, revealing the critical impact of hyperparameter choices. To address the challenge of scaling the learning of unknown objects, we explore unsupervised training strategies to generate pseudo-labels for unknowns. Among the different approaches evaluated, our experiments show that top-K auto-labelling offers more promising performance compared to simple resizing techniques.

Keywords: Omnidirectional Vision, Deep Learning for Visual Perception, Computer Vision for Transportation
Abstract: Omnidirectional depth perception is essential for mobile robotics applications that require scene understanding across a full 360° field of view. Camera-based setups offer a cost-effective option by using stereo depth estimation to generate dense, high-resolution depth maps without relying on expensive active sensing. However, existing omnidirectional stereo matching approaches achieve only limited depth accuracy across diverse environments, depth ranges, and lighting conditions, due to the scarcity of real-world data. We present DFI-OmniStereo, a novel omnidirectional stereo matching method that leverages a large-scale pre-trained foundation model for relative monocular depth estimation within an iterative optimization-based stereo matching architecture. We introduce a dedicated two-stage training strategy to utilize the relative monocular depth features for our omnidirectional stereo matching before scale-invariant fine-tuning. DFI-OmniStereo achieves state-of-the-art results on the real-world Helvipad dataset, reducing disparity MAE by approximately 16% compared to the previous best omnidirectional stereo method. More details, code, and model weights are available at https://vita-epfl.github.io/DFI-OmniStereo-website/.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception, Transfer Learning
Abstract: Mobile robots rely on object detectors for perception and object localization in indoor environments. However, standard closed-set methods struggle to handle the diverse objects and dynamic conditions encountered in real homes and labs. Open-vocabulary object detection (OVOD), driven by Vision Language Models (VLMs), extends beyond fixed labels but still struggles with domain shifts in indoor environments. We introduce a Source-Free Domain Adaptation (SFDA) approach that adapts a pre-trained model without accessing source data. We refine pseudo labels via temporal clustering, employ multi-scale threshold fusion, and apply a Mean Teacher framework with contrastive learning. Our Embodied Domain Adaptation for Object Detection (EDAOD) benchmark evaluates adaptation under sequential changes in lighting, layout, and object diversity. Our experiments show significant gains in zero-shot detection performance and flexible adaptation to dynamic indoor conditions.

Keywords: Object Detection, Segmentation and Categorization, Computer Vision for Automation, Recognition
Abstract: Underwater Object Detection (UOD) techniques are critical for Autonomous Underwater Vehicle (AUV), which must operate in harsh underwater environments characterized by low visibility while satisfying the lightweight and real-time constraints required for vehicle-mounted systems. Current methods typically rely on underwater image enhancement combined with object detection to adapt to underwater conditions. However, these approaches mainly focus on the spatial domain, often overlooking the frequency-domain characteristics of the underwater environment. This oversight limits the removal of noise factors, such as scattering, blurring, distortion, and uneven illumination, and diminishes the focus of object edges and textures. Additionally, the increased parameter size and higher computational cost render them less suitable for real-time detection. To address these, the Wavelet-based Frequency Decomposition and Aggregation Network (WFDA) was proposed, which leverages the Wavelet Transform (WT) to decompose features into high- and low-frequency components for effective feature modeling and fusion-based downsampling. Specifically, the Wavelet-Based Feature Decomposition Modeling (WDM) module utilized multi-level wavelet decomposition to hierarchically model features across different frequency bands, while the Wavelet-Based Feature Aggregation Downsampling (WAD) module refined and extracted core features through single-level wavelet decomposition combined with channel aggregation. Evaluations on four public datasets demonstrate that WFDA achieved state-of-the-art (SOTA) performance and efficiency, making it well-suited for real-time, high-accuracy detection on robotic platforms. Code is available at https://github.com/Mariiiiooooo/WFDA.

Keywords: Object Detection, Segmentation and Categorization, Incremental Learning, Human Factors and Human-in-the-Loop
Abstract: Few-shot object detection is especially interesting for applications with mobile robots and becomes even more challenging when task-related classes are very similar. This work focuses on such a scenario: detecting different types of household and industrial tools. Such tools can be rare and specific and are usually not covered by existing large datasets, except for common ones such as screwdrivers. Additionally, the target classes might change frequently depending on the robot’s missions. Therefore, we propose DE-fine-ViT, a fine-grained few-shot object detection model that does not require fine-tuning. We build our architecture on top of the elaborate DE-ViT model, extending it with specialized components to improve the fine-grained detection capabilities. The user can construct class and part prototypes tailored to the task in an interactive preparation phase. During inference, our proposed re-evaluation module leverages the multi-granularity of prototypes for fine-grained class differentiation. We evaluate our model in multiple realistic experiments, including a specifically created fine-grained dataset, demonstrating its efficacy and suitability for scenarios with little data and low inter-class variance.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception, AI-Based Methods
Abstract: Unmanned aerial vehicle object detection (UAV-OD) has been widely used in various scenarios. However, most existing UAV-OD algorithms rely on manually designed components, which require extensive tuning. End-to-end models that do not depend on such manually designed components are mainly designed for natural images, which are less effective for UAV imagery. To address such challenges, this paper proposes an efficient detection transformer (DETR) framework tailored for UAV imagery, i.e., UAV-DETR. The framework includes a multi-scale feature fusion with frequency enhancement module, which captures both spatial and frequency information at different scales. In addition, a frequency-focused downsampling module is presented to retain critical spatial details during downsampling. A semantic alignment and calibration module is developed to align and fuse features from different fusion paths. Experimental results demonstrate the effectiveness and generalization of our approach across various UAV imagery datasets. On the VisDrone dataset, our method improves AP by 3.1% and AP50 by 4.2% over the baseline. Similar enhancements are observed on the UAVVaste dataset. The project page is available at https://github.com/ValiantDiligent/UAV-DETR.

Keywords: Visual-Inertial SLAM, SLAM, Sensor Fusion
Abstract: Visual-inertial odometry (VIO) has made significant progress in various applications. However, one of the key challenges in VIO is the efficient and robust fusion of visual and inertial measurements, particularly while mitigating the impact of sensor failures. To address this challenge, we propose a new learning-based VIO system, i.e., DW-VIO, which is able to integrate multiple sensors and provide robust state estimations. To this end, we design a novel deep learning-based data-fusion approach that dynamically associates information from multiple sensors to predict sensor weights for optimization. Moreover, in order to improve the efficiency, we present several real-time optimization techniques including a fast patch graph constructor and efficient GPU-accelerated multi-factor bundle adjustment layer. Experimental results show that DW-VIO outperforms most state-of-the-art (SOTA) methods on the EuRoC MAV, ETH3D-SLAM, and KITTI-360 benchmarks across various challenging sequences. Additionally, it maintains a minimum of 20 frames per second (FPS) on a single RTX 3060 GPU with high-resolution input, highlighting its efficiency.

Keywords: Visual-Inertial SLAM, SLAM
Abstract: In this paper, we introduce a novel estimator for vision-aided inertial navigation systems (VINS), the Preconditioned Cholesky-based Square Root Information Filter (PC-SRIF). When solving linear systems, employing Cholesky decomposition offers superior efficiency but can compromise numerical stability. Due to this, existing VINS utilizing (Square Root) Information Filters often opt for QR decomposition on platforms where single precision is preferred, avoiding the numerical challenges associated with Cholesky decomposition. While these issues are often attributed to the ill-conditioned information matrix in VINS, our analysis reveals that this is not an inherent property of VINS but rather a consequence of specific parameterizations. We identify several factors that contribute to an ill-conditioned information matrix and propose a preconditioning technique to mitigate these conditioning issues. Building on this analysis, we present PC-SRIF, which exhibits remarkable stability in performing Cholesky decomposition in single precision when solving linear systems in VINS. Consequently, PC-SRIF achieves superior theoretical efficiency compared to alternative estimators. To validate the efficiency advantages and numerical stability of PC-SRIF based VINS, we have conducted well controlled experiments, which provide empirical evidence in support of our theoretical findings. Remarkably, in our VINS implementation, PC-SRIF's runtime is 41% faster than QR-based SRIF.

Keywords: Visual-Inertial SLAM, Visual Tracking, Performance Evaluation and Benchmarking
Abstract: The tracking module of a visual-inertial SLAM system processes incoming image frames and IMU data to estimate the position of the frame in relation to the map. It is important for the tracking to complete in a timely manner for each frame to avoid poor localization or tracking loss. We therefore present a new approach which leverages GPU computing power to accelerate time-consuming components of tracking in order to improve its performance. These components include stereo feature matching and local map tracking. We implement our design inside the ORB-SLAM3 tracking process using CUDA. Our evaluation demonstrates an overall improvement in tracking performance of up to 2.8x on a desktop and Jetson Xavier NX board in stereo-inertial mode, using the well-known SLAM datasets EuRoC and TUM-VI.

Keywords: Visual-Inertial SLAM, Vision-Based Navigation, Sensor Fusion
Abstract: The Visual-Inertial Odometry has been widely deployed on autonomous robots traveling in open outdoor scenarios. However, the visual measurements will be influenced heavily by the observation distances, perspectives, lighting and texture conditions, with distinct and time-varying noise distributions of measurements. Existing methods for handling time-varying noise in Visual-Inertial Odometry regard all measurement noise as identically distributed, unable to effectively deal with the distinct noise in open outdoor scenarios, which degrades the localization accuracy. In this paper, a Multi-State Constraint Kalman Filter with Adaptive multivariate noise parameters Clustering and estimation for visual-inertial odometry (CAMSCKF) is proposed to address the issue, which can separately track the measurement noise covariance matrix (MNCM) of different measurement clusters and adjust the MNCM in real-time. Firstly, the joint distribution of the state and the MNCM coefficients for each cluster is modeled as an Gaussian-Multivariate Generalized Inverse Gaussian distribution. Subsequently, an Expectation Maximization algorithm-based stepwise adaptive measurement clustering method is designed, which clusters measurements according to their corresponding innovations. Finally, an analytical update method for the joint posterior distribution without fixed-point iteration is implemented, achieving adaptive adjustment of the MNCM, thereby enabling accurate and robust Visual-Inertial Odometry localization. The superiority of the proposed method is demonstrated by simulations and dataset experiments, especially under the aggressive motion. In the experiments of the challenging outdoor dataset UZH-FPV, the proposed method has improved the average position and attitude estimation accuracy by 35.69% and 32.88%, respectively, compared with the state-of-the-art ANGIG-KF.

Keywords: Visual-Inertial SLAM, Legged Robots, Sensor Fusion
Abstract: This paper presents GeoFlow-SLAM, a robust and effective Tightly-Coupled RGBD-Inertial and Legged Odometry Fusion SLAM for legged robotics undergoing aggressive and high-frequency motions. By integrating geometric consistency, legged odometry constraints, and dual-stream optical flow (GeoFlow), our method addresses three critical challenges: feature matching and pose initialization failures during fast locomotion and visual feature scarcity in texture-less scenes. Specifically, in rapid motion scenarios， feature matching is notably enhanced by leveraging dual-stream optical flow, which combines prior map points and poses. Additionally, we propose a robust pose initialization method for fast locomotion and IMU error in legged robots, integrating IMU/Legged odometry, inter-frame Perspective-n-Point (PnP), and Generalized Iterative Closest Point (GICP). Furthermore, a novel optimization framework that tightly couples depth-to-map and GICP geometric constraints is first introduced to improve the robustness and accuracy in long-duration, visually texture-less environments. The proposed algorithms achieve state-of-the-art (SOTA) on collected legged robots and open-source datasets. To further promote research and development, the open-source datasets and code will be made publicly available at https://github.com/HorizonRobotics/GeoFlowSlam.

Keywords: Visual-Inertial SLAM, SLAM, Deep Learning for Visual Perception
Abstract: The performance of a vision simultaneous localization and mapping (SLAM) system based on hand-crafted features degrades significantly in harsh environments due to unstable feature tracking. With the breakthrough of convolutional neural networks in deep feature extraction tasks, many researchers have tried to incorporate them into SLAM systems. However, it's challenging to guarantee the real-time performance of the entire SLAM system, and the erroneous usage scenarios limit the superior performance of deep feature extraction methods. To overcome these problems, we propose a visual-inertial SLAM system with multi-level detector and descriptor, called VINS-MLD2. In our framework, we first design an efficient deep feature extraction network that has the same performance as R2D2 by concatenating multi-level features, but runs 3 times faster under the image resolution commonly used in SLAM. Then, based on the camera baseline, we introduce the Matching Fusion, a matching method that fuses deep descriptor matching and optical flow matching results to improve matching accuracy for both short and wide baselines. In addition, an adaptive matching strategy is proposed to balance the running time and accuracy by adaptively adjusting the matching method. Experimental results in unmanned aerial vehicle (UAV) deployments and real-world environments demonstrate that the proposed method tracks features more stably and accurately.

Keywords: Visual-Inertial SLAM
Abstract: Online extrinsic calibration is crucial for building "power-on-and-go" moving platforms, like robots and AR devices. However, blindly performing online calibration for unobservable parameter may lead to unpredictable results. In the literature, extensive studies have been conducted on the extrinsic calibration between IMU and camera, from theory to practice. It is well-known that the observability of extrinsic parameter can be guaranteed under sufficient motion excitation. Furthermore, the impacts of degenerate motions are also investigated. Despite these successful analyses, we identify an issue with respect to the existing observability conclusion. This paper focuses on the observability investigation for straight line motion, which is a common-seen and fundamental degenerate motion in applications. We analytically prove that pure translational straight line motion can lead to the unobservability of the rotational extrinsic parameter between IMU and camera (at least one degree of freedom). By correcting the existing observability conclusion, our novel theoretical finding disseminates more precise principle to the research community and provides explainable calibration guideline for practitioners. Our analysis is validated by rigorous theory and experiments.

Keywords: Visual-Inertial SLAM, Sensor Fusion, Intelligent Transportation Systems
Abstract: This paper presents a novel tightly coupled Filter-based monocular visual inertial-wheel odometry (VIWO) system for ground robots, designed to deliver accurate and robust localization in long-term complex outdoor navigation scenarios. As an external sensor, the camera enhances localization performance by introducing visual constraints. However, obtaining a sufficient number of effective visual features is often challenging, particularly in dynamic or low texture environments. To address this issue, we incorporate the line features for additional geometric constraints. Unlike traditional approaches that treat point and line features independently, our method exploits the geometric relationships between points and lines in 2D images, enabling fast and robust line matching and triangulation. Additionally, we introduce Motion Consistency Check (MCC) to filter out potential dynamic points, ensuring the effectiveness of point feature updates. The proposed system was evaluated on publicly available datasets and benchmarked against state-of-the-art methods. Experimental results demonstrate superior performance in terms of accuracy, robustness, and efficiency. The source code is publicly available at: https://github.com/Happy-ZZX/PL-VIWO.

Keywords: Collision Avoidance, Autonomous Vehicle Navigation, Reinforcement Learning
Abstract: Mapless navigation for Automated Guided Vehicles (AGV) via Deep Reinforcement Learning (DRL) algorithms has attracted significantly rising attention in recent years. Collision avoidance from dynamic obstacles in unstructured environments, such as pedestrians and other vehicles, is one of the key challenges for mapless navigation. Autonomous navigation requires a policy to make decisions to optimize the path distance towards the goal but also to reduce the probability of collisions with obstacles. Mostly, the reward for AGV navigation is calculated by combining multiple reward functions for different purposes, such as encouraging the robot to move towards the goal or avoiding collisions, as a state-conditioned function. The combined reward, however, may lead to biased behaviours due to the empirically chosen weights when multiple rewards are combined and dangerous situations are misjudged. Therefore, this paper proposes a learning-based method with multiple unrelated rewards, which represent the evaluation of different behaviours respectively. The policy network, named Multi-Feature Policy Gradients (MFPG), is conducted by two separate Q networks that are constructed by two individual rewards, corresponding to goal distance shortening and collision avoidance, respectively. In addition, we also propose an auto-tuning method, named Ada-MFPG, that allows the MFPG algorithm to automatically adjust the weights for the two separate policy gradients. For collision avoidance, we present a new social norm-oriented continuous biased reward for performing specific social norm so as to reduce the probabilities of AGV collisions. By adding an offset gain to one of the reward functions, vehicles conducted by the proposed algorithm exhibited the predetermined features. The work was tested in different simulation environments under multiple scenarios with a single robot or multiple robots. The proposed MFPG method is compared with standard Deep Deterministic Policy Gradient (DDPG), the modified DDPG, SAC and TD3 with a social norm mechanism. MFPG significantly increases the success rate in robot navigation tasks compared with the DDPG. Besides, among all the benchmarking algorithms, the MFPG-based algorithms have the optimal task completion duration and lower variance compared with the baselines. The work has also been tested on real robots. Experiments on the real robots demonstrate the viability of the trained model for the real world scenarios. The learned model can be used for multi-robot mapless navigation in complex environments, such as a warehouse, that need multi-robot cooperation.

Keywords: AI-Enabled Robotics, Reinforcement Learning, Machine Learning for Robot Control
Abstract: Reinforcement learning (RL) is a fundamental and pivotal algorithm in the advancement of Embodied Intelligence. The performance of RL directly influences the quality and efficiency of a robot's decision-making and execution during interactions with its environment. However, the robustness of RL remains a critical challenge that needs to be addressed. A promising approach to enhancing robustness is adversarial reinforcement learning. Existing methods primarily focus on perturbations in the state space, while perturbations in the action space have been relatively underexplored. Given that action space plays an equally crucial role as state space in Embodied Intelligence, action-space perturbations provide a more comprehensive evaluation of RL robustness. Therefore, investigating RL robustness under action-space perturbations is both necessary and valuable for the development of Embodied Intelligence.To this end, we propose an adversarial learning framework that employs momentum-based gradient descent to model perturbations in the action space, such as actuator disturbances. Furthermore, we introduce an improved optimization method that integrates historical gradient information into conventional Stochastic Gradient Descent (SGD). This approach enhances training stability and improves perturbation efficiency. The proposed method is evaluated through simulations in the MuJoCo environment and UAV control experiments in GymFC, demonstrating significant improvements in robustness and adaptability under action-space perturbations. Additionally, real-world UAV flight tests were conducted to further validate the effectiveness of the proposed framework. The results confirm that the Sim-to-Real transfer is successful, providing empirical evidence for the applicability of our method in real-world scenarios.This study establishes that enhancing RL robustness through action-space perturbations is both feasible and effective. More importantly, our findings contribute to the future development of Embodied Intelligence, particularly in improving its resilience to uncertainties and dynamic environments.

Keywords: Biologically-Inspired Robots, Legged Robots, Reinforcement Learning
Abstract: Improving the adaptability of small-sized quadruped robots has been a longstanding challenge in robotics.However, the weak whole-body coordination in existing small-sized quadruped robots limits their locomotion in many environments.In this work, we propose a teacher-student online learning framework for agile whole-body control of small-sized quadruped robots with a flexible spine. We first select a simple and effective gait pattern, the diagonal symmetrical sequence, using a dynamics model.Based on the reference motions provided by the gait pattern and combined with privileged information, we train a teacher policy to generate high-quality motion data. After setting the state space to match the actual robot's state space, we initialize the robot's initial state using the teacher data and train a student policy. Finally, we deploy the student policy on the SQuRo-Lite, a small-sized quadruped robot with a flexible spine, demonstrating that our approach can achieve stable yet dynamic locomotion for walking and turning. In the variable-spacing slalom experiment, the robot is able to flexibly adjust the motion patterns of its spine and legs based on commands, enabling dynamic changes in its turning radius. This further validates that our approach can achieve agile whole-body control for small-sized quadruped robots. This work helps broaden the application scenarios of small quadruped robots.

Keywords: Biologically-Inspired Robots, Reinforcement Learning, Collision Avoidance
Abstract: Achieving autonomous navigation for biologically inspired jumping robots remains a long-standing challenge, due to the inherent instability of jumping motions and the limitations in onboard sensor capabilities. This paper proposes a skill-based hierarchical framework with dangerous action masking (SH-DAM) for autonomous navigation of jumping robot. The framework, based on hierarchical reinforcement learning, includes a low-level controller that learns locomotion skills (crawling, turning and jumping) to overcome various obstacles. A high-level controller selects and coordinates these skills, while also incorporating curriculum learning to enhance the performance of navigation tasks. For safe navigation, we utilize dangerous action masking to suppress the probability of selecting jump motions in dangerous regions. We improved the locust-inspired jumping robot platform JumpBot-S, by integrating a lightweight time-of-flight (ToF) sensor, and constructed a range of complex environments for experiments. Simulation results demonstrate that SH-DAM enables the robot to autonomously complete challenging navigation tasks. Compared to baseline algorithms, our method achieves a 12.57% increase in success rate, a 55.88% reduction in stuck rate, and a 57.89% reduction in rollover rate. Finally, we deployed our framework in real-world environments and conducted experiments in both normal lit and dimly lit conditions. This framework provides a new paradigm for jumping robot navigation in complex environments.

Keywords: Autonomous Agents, Reinforcement Learning, Deep Learning Methods
Abstract: To avoid dangerous situations, such as collisions in dynamic environments, autonomous vehicles must predict the risks of the current scene to take safe actions. Traditional rule-based risk prediction methods and existing reinforcement learning (RL) approaches, which typically rely on manually designed driving decision rules or heuristic reward functions, often fail to capture the complexity of real-world dangerous scenarios, leading to suboptimal and unsafe driving decisions. To address this limitation, we develop a novel RL method, called Group Opinion Risk-Aware Reinforcement Learning (GORA-RL), for safer driving decisions that align with real-world conditions. Specifically, we first introduce surveys of human drivers to assess risk in real-world driving situations. Using these real group opinions as training data, we train a risk prediction model, referred to as the risk prediction model with a Transformer (RPT), that captures the crucial characteristics of these scenarios, resulting in more realistic and reliable risk predictions. This model is then integrated as a reward function to train an RL algorithm for making driving decisions in various scenarios. The experiments validate that our approach outperforms two state-of-the-art (SOTA) methods in challenging congested scenarios, such as merging and intersections, in terms of reward and several other metrics. Project site: url{https://github.com/naiyisiji/RPT}.

Keywords: Autonomous Vehicle Navigation, Machine Learning for Robot Control, Reinforcement Learning
Abstract: Robotics Reinforcement Learning (RL) often relies on carefully engineered auxiliary rewards to supplement sparse primary learning objectives to compensate for the lack of large-scale, real-world, trial-and-error data. While these auxiliary rewards accelerate learning, they require significant engineering effort, may introduce human biases, and cannot adapt to the robot's evolving capabilities during training. In this paper, we introduce Reward Training Wheels (RTW), a teacher-student framework that automates auxiliary reward adaptation for robotics RL. To be specific, the RTW teacher dynamically adjusts auxiliary reward weights based on the student's evolving capabilities to determine which auxiliary reward aspects require more or less emphasis to improve the primary objective. We demonstrate RTW on two challenging robot tasks: navigation in highly constrained spaces and off-road vehicle mobility on vertically challenging terrain. In simulation, RTW outperforms expert-designed rewards by 2.35% in navigation success rate and improves off-road mobility performance by 122.62%, while achieving 35% and 3X faster training efficiency, respectively. Physical robot experiments further validate RTW's effectiveness, achieving a perfect success rate (5/5 trials vs. 2/5 for expert-designed rewards) and improving vehicle stability with up to 47.4% reduction in orientation angles.

Keywords: Autonomous Vehicle Navigation, Reinforcement Learning, Intelligent Transportation Systems
Abstract: Reinforcement Learning (RL) offers a promising solution to enable evolutionary automated driving. However, conventional RL methods often struggle with risk performance, as updated policies may fail to enhance performance or even lead to deterioration. To address this challenge, this research introduces a High Confidence Policy Improvement Reinforcement Learning-based (HCPI-RL) planner, designed to achieve the monotonic evolution of automated driving. The HCPI-RL planner features a novel RL policy update paradigm, ensuring that each newly learned policy outperforms previous policies, achieving monotonic performance enhancement. Hence, the proposed HCPI-RL planner has the following features: i) Evolutionary automated driving with guaranteed monotonic performance enhancement; ii) Capability of handling scenarios with emergency; iii) Enhanced decision-making optimality. Experimental results demonstrate that the proposed HCPI-RL planner enhances policy return by at least 20.1% and driving efficiency by at least 15.6%, compared to the conventional RL-based planners.

Keywords: AI-Based Methods, Multi-Robot Systems, Reinforcement Learning
Abstract: Transformer-based sequence models have proven effective in offline reinforcement learning for modeling agent trajectories using large-scale datasets. However, applying these models directly to multi-agent offline reinforcement learning introduces additional challenges, especially in managing complex inter-agent dynamics that arise as multiple agents interact with both their environment and each other. To overcome these issues, we propose the context-aware multi-agent trajectory transformer (COMAT), a novel model designed for offline multi-agent reinforcement learning tasks which predicts the future trajectory of each agent by incorporating the history of adjacent agents—referred to as context—into its sequence modeling. COMAT consists of three key modules: the transformer module to process input trajectories, the context encoder to extract relevant information from adjacent agents’ histories, and the context aggregator to integrate this information into the agent’s trajectory prediction process. Built upon these modules, COMAT predicts the agents’ future trajectories and actively leverages this capability as a tool for planning, enabling the search for optimal actions in multi-agent environments. We evaluate COMAT on multi-agent MuJoCo and StarCraft Multi-Agent Challenge tasks, on which it demonstrates superior performance compared to existing baselines.

Keywords: Vision-Based Navigation, Deep Learning Methods
Abstract: In this paper, we introduce a novel image-goal navigation approach, named RFSG. Our focus lies in leveraging the fine-grained connections between goals, observations, and the environment within limited image data, all the while keeping the navigation architecture simple and lightweight. To this end, we propose the spatial-channel attention mechanism, enabling the network to learn the importance of multi-dimensional features to fuse the goal and observation features. In addition, a self-distillation mechanism is incorporated to further enhance the feature representation capabilities. Given that the navigation task needs surrounding environmental information for more efficient navigation, we propose an image scene graph to establish feature associations at both the image and object levels, effectively encoding the surrounding scene information. Cross-scene performance validation was conducted on the Gibson and HM3D datasets, and the proposed method achieved state-of-the-art results among mainstream methods, with a speed of up to 53.5 frames per second on an RTX3080. This contributes to the realization of end-to-end image-goal navigation in real-world scenarios. The implementation and model of our method have been released at: https://github.com/nubot-nudt/RFSG.

Keywords: Vision-Based Navigation, Deep Learning for Visual Perception, Probability and Statistical Methods
Abstract: Perception-based navigation systems are useful for unmanned ground vehicle (UGV) navigation in complex terrains, where traditional depth-based navigation schemes are insufficient. However, these data-driven methods are highly dependent on their training data and can fail in surprising and dramatic ways with little warning. To ensure the safety of the vehicle and the surrounding environment, it is imperative that the navigation system is able to recognize the predictive uncertainty of the perception model and respond safely and effectively in the face of uncertainty. In an effort to enable safe navigation under perception uncertainty, we develop a probabilistic and reconstruction-based competency estimation (PaRCE) method to estimate the model's level of familiarity with an input image as a whole and with specific regions in the image. We find that the overall competency score can accurately predict correctly classified, misclassified, and out-of-distribution (OOD) samples. We also confirm that the regional competency maps can accurately distinguish between familiar and unfamiliar regions across images. We then use this competency information to develop a planning and control scheme that enables effective navigation while maintaining a low probability of error. We find that the competency-aware scheme greatly reduces the number of collisions with unfamiliar obstacles, compared to a baseline controller with no competency awareness. Furthermore, the regional competency information is particularly valuable in enabling efficient navigation.

Keywords: Vision-Based Navigation, AI-Enabled Robotics, AI-Based Methods
Abstract: Vision-and-Language Navigation in Continuous Environments (VLN-CE) presents challenges due to environmental variations and domain shifts, making it difficult for agents to generalize beyond seen environments. Most existing methods rely on learning correlations between observations and actions from training data, which leads to spurious dependencies on environmental biases. To address this, we propose CVLN-Think (CVT), a novel navigation model that incorporates causal inference to enhance robustness and adaptability. Specifically, Style Causal Adjuster (SCA) generates counterfactual style observations, enabling agents to learn invariant spatial structures rather than overfitting to dataset-specific visual patterns. Furthermore, Thinking Cause Navigation Engine (TCNE) applies causal intervention to adjust navigation decisions by identifying and mitigating biases from prior experience. Unlike conventional approaches that passively learn from data distributions, our model actively thinks along the ``observation-action'' chain to make more reliable navigation predictions. Experimental results demonstrate that our approach achieves satisfactory performance on VLN-CE tasks. Further analysis indicates that our method possesses stronger generalization capabilities, highlighting the superiority of our proposed approach.

Keywords: Vision-Based Navigation, Deep Learning Methods, Visual Learning
Abstract: Vision and Language Navigation (VLN) requires an agent to navigate through environments following natural language instructions. However, existing methods often struggle with effectively integrating visual observations and instruction details during navigation, leading to suboptimal path planning and limited success rates. In this paper, we propose OIKG (Observation-graph Interaction and Key-detail Guidance), a novel framework that addresses these limitations through two key components: (1) an observation-graph interaction module that decouples angular and visual information while strengthening edge representations in the navigation space, and (2) a key-detail guidance module that dynamically extracts and utilizes fine-grained location and object information from instructions. By enabling more precise cross-modal alignment and dynamic instruction interpretation, our approach significantly improves the agent's ability to follow complex navigation instructions. Extensive experiments on the R2R and RxR datasets demonstrate that OIKG achieves state-of-the-art performance across multiple evaluation metrics, validating the effectiveness of our method in enhancing navigation precision through better observation-instruction alignment.

Keywords: Vision-Based Navigation, Deep Learning Methods
Abstract: Robotic navigation plays a pivotal role in a wide range of real-world applications. While traditional navigation systems focus on efficiency and obstacle avoidance, their inability to model complex human behaviors in shared spaces has underscored the growing need for socially aware navigation. In this work, we explore a novel paradigm of socially aware robot navigation empowered by large language models (LLMs), and propose HSAC-LLM, a hybrid framework that seamlessly integrates deep reinforcement learning with the reasoning and communication capabilities of LLMs. Unlike prior approaches that passively predict pedestrian trajectories or issue pre-scripted alerts, HSAC-LLM enables bidirectional natural language interaction, allowing robots to proactively engage in dialogue with pedestrians to resolve potential conflicts and negotiate path decisions. Extensive evaluations across 2D simulations, Gazebo environments, and real-world deployments demonstrate that HSAC-LLM consistently outperforms state-of-the-art DRL baselines under our proposed socially aware navigation metric, which covers safety, efficiency, and human comfort. By bridging linguistic reasoning and interactive motion planning, our results highlight the potential of LLM-augmented agents for robust, adaptive, and human-aligned navigation in real-world settings. Project page: https://hsacllm.github.io/.

Keywords: Vision-Based Navigation, Incremental Learning
Abstract: Vision-language navigation (VLN) is a pivotal area within embodied intelligence, where agents must navigate based on natural language instructions. While traditional VLN research has focused on enhancing environmental comprehension and decision-making policy, these methods often reveal substantial performance gaps when agents are deployed in novel environments. This issue primarily arises from the lack of diverse training data. Expanding datasets to encompass a broader range of environments is impractical and costly. To address this challenge, we propose Vision-Language Navigation with Continuous Learning (VLNCL), a framework that allows agents to learn from new environments while preserving previous knowledge incrementally. We introduce a novel dual-loop scenario replay method (Dual-SR) inspired by brain memory mechanisms integrated with VLN agents. This approach helps consolidate past experiences and improves generalization across novel tasks. As a result, the agent exhibits enhanced adaptability to new environments and mitigates catastrophic forgetting. Our experiment demonstrates that VLN agents with Dual-SR effectively resist forgetting and adapt to unfamiliar environments. Combining VLN with continual learning significantly boosts the performance of otherwise average models, achieving SOTA results.

Keywords: Vision-Based Navigation, Deep Learning Methods
Abstract: Visual Language Navigation (VLN) is a fundamental task within the field of Embodied AI, focusing on the ability of agents to navigate complex environments based on natural language instructions. Despite the progress made by existing methods, these methods often present some common challenges. First, they rely on pre-trained backbone models for visual perception, which struggle with the dynamic viewpoints in VLN scenarios. Second, the performance is limited when using pre-trained LLMs or VLMs without fine-tuning, due to the absence of VLN domain knowledge. Third, while fine-tuning LLMs and VLMs can improve results, their computational costs are higher than those without fine-tuning. To address these limitations, we propose Weakly-supervised Partial Contrastive Learning (WPCL), a method that enhances an agent's ability to identify objects from dynamic viewpoints in VLN scenarios by effectively integrating pre-trained VLM knowledge into the perception process, without requiring VLM fine-tuning. Our method enhances the agent's ability to interpret and respond to environmental cues while ensuring computational efficiency. Experimental results have shown that our method outperforms the baseline methods on multiple benchmarks, which validates the effectiveness, robustness, and generalizability of our method.

Keywords: Vision-Based Navigation, Computer Vision for Automation
Abstract: Object Goal Navigation (ObjectNav) refers to an agent navigating to an object in an unseen environment, which is an ability often required in the accomplishment of complex tasks. Though it has drawn increasing attention from researchers in the Embodied AI community, there has not been a contemporary and comprehensive survey of ObjectNav. In this survey, we give an overview of this field by summarizing more than 70 recent papers. First, we give the preliminaries of the ObjectNav: the definition, the simulator, and the metrics. Then, we group the existing works into three categories: 1) end-to-end methods that directly map the observations to actions, 2) modular methods that consist of a mapping module, a policy module, and a path planning module, and 3) zero-shot methods that use zero-shot learning to do navigation. Finally, we summarize the performance of existing works and the main failure modes and discuss the challenges of ObjectNav. This survey would provide comprehensive information for researchers in this field to have a better understanding of ObjectNav.

Keywords: Deep Learning for Visual Perception, RGB-D Perception, AI-Based Methods
Abstract: The soft-argmax operation is widely adopted in neural network-based stereo matching methods to enable differentiable regression of disparity. However, networks trained with soft-argmax tend to predict multimodal probability distributions due to the absence of explicit constraints on the shape of the distribution. Previous methods leveraged Laplacian distributions and cross-entropy for training but failed to effectively improve accuracy and even increased the network's processing time. In this paper, we propose a novel method called Sampling-Gaussian as a substitute for soft-argmax. It improves accuracy without increasing inference time. We innovatively interpret the training process as minimizing the distance in vector space and propose a combined loss of L1 loss and cosine similarity loss. We leveraged the normalized discrete Gaussian distribution for supervision. Moreover, we identified two issues in previous methods and proposed extending the disparity range and employing bilinear interpolation as solutions. We have conducted comprehensive experiments to demonstrate the superior performance of our textit{Sampling-Gaussian} method. The experimental results prove that we have achieved better accuracy on five baseline methods across four datasets. Moreover, we have achieved significant improvements on small datasets and models with weaker generalization capabilities. Our method is easy to implement, and the code is available online.

Keywords: Deep Learning for Visual Perception
Abstract: Retinex theory, which treats an image as a composition of illuminance and reflectance, has made significant progress in low-light image enhancement. Previous methods attempt to refine the impractical Retinex theory by introducing deviations in estimated illumination and reflectance to develop more practical and robust enhancement techniques. However, the fact that state-of-the-art approaches still produce inferior results suggests that some form of corruption may be overlooked. In this paper, we propose a novel Complete Corruption-Aware Retinex Framework (CCRF), which not only considers corruption in low-light imaging—such as high ISO or long exposure settings—but, more importantly, also accounts for corruption induced by the enhancement method itself. Guided by this framework, we propose a Robust Corruption-Aware Loss (RCL) that enables the model to be robust under extreme darkness and complex light-object interactions. Additionally, we propose a Light-Up Map Denoising (LMD) module, which further eliminates model-induced perturbations. With these two plug-and-play modules, downstream tasks (e.g., low-light object detection) can benefit significantly. Extensive experiments demonstrate that our methods can be seamlessly integrated into state-of-the-art approaches, resulting in significant performance improvements over these methods. Code will be available at github.com/eafi/ccrf

Keywords: Deep Learning for Visual Perception, Semantic Scene Understanding, Sensor Fusion
Abstract: Vision-based 3D occupancy prediction has made significant advancements, but its reliance on cameras alone struggles in challenging environments. This limitation has driven the adoption of sensor fusion, among which camera-radar fusion stands out as a promising solution due to their complementary strengths. However, the sparsity and noise of the radar data limits its effectiveness, leading to suboptimal fusion performance. In this paper, we propose REOcc, a novel camera-radar fusion network designed to enrich radar feature representations for 3D occupancy prediction. Our approach introduces two main components, a Radar Densifier and a Radar Amplifier, which refine radar features by integrating spatial and contextual information, effectively enhancing spatial density and quality. Extensive experiments on the Occ3D-nuScenes benchmark demonstrate that REOcc achieves significant performance gains over the camera-only baseline model, particularly in dynamic object classes. These results underscore REOcc's capability to mitigate the sparsity and noise of the radar data. Consequently, radar complements camera data more effectively, unlocking the full potential of camera-radar fusion for robust and reliable 3D occupancy prediction.

Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Visual Learning
Abstract: Accurate 6D pose estimation is key for robotic manipulation, enabling precise object localization for tasks like grasping. We present RAG-6DPose, a retrieval-augmented approach that leverages 3D CAD models as a knowledge base by integrating both visual and geometric cues. Our RAG-6DPose roughly contains three stages: 1) Building a Multi-Modal CAD Knowledge Base by extracting 2D visual features from multi-view CAD rendered images and also attaching 3D points; 2) Retrieving relevant CAD features from the knowledge base based on the current query image via our ReSPC module; and 3) Incorporating retrieved CAD information to refine pose predictions via retrieval-augmented decoding. Experimental results on standard benchmarks and real-world robotic tasks demonstrate the effectiveness and robustness of our approach, particularly in handling occlusions and novel viewpoints. Supplementary material is available on our project website: https://sressers.github.io/RAG-6DPose. We will release codes and models upon acceptance.

Keywords: Deep Learning for Visual Perception, Semantic Scene Understanding, Data Sets for Robotic Vision
Abstract: Visual Terrain Classification (VTC) plays a vital role in enabling unmanned ground vehicles to understand complex environments. Existing research relies on image-label pairs annotated by static label sets, where semantic ambiguity and high annotation costs constrain fine-grained terrain characterization. These limitations hinder the model’s adaptation to real-world terrain diversity and restrict its applicability. To address these issues, we propose TerraX, a vision-language learning framework that integrates multi-modal image-label-text data, unifying structured annotations with fine-grained natural language descriptions. The framework introduces a composite dataset TerraData, an evaluation benchmark suite TerraBench, and a CLIP-based visual terrain classification model TerraCLIP. TerraData aggregates multi-source terrain images from public and self-collected datasets, annotated through a VLM-based vision-language data annotation pipeline. TerraBench defines three evaluation benchmarks to systematically assess model robustness and adaptability in real-world terrain classification scenarios. Built on the CLIP model, TerraCLIP utilizes multi-granularity contrastive loss and LoRA fine-tuning to enhance understanding for terrain categories and attributes, and incorporates confidence-weighted inference for accurate predictions. Extensive experiments across benchmarks and real-world platforms demonstrate that our approach significantly enhances VTC performance, highlighting its potential for deployment in complex environments.

Keywords: Deep Learning for Visual Perception, Object Detection, Segmentation and Categorization
Abstract: Vision-based BEV (Bird's-Eye-View) 3D object detection has recently become popular in autonomous driving. However, objects with a high similarity to the background from a camera perspective cannot be detected well by existing methods. In this paper, we propose a BEV-based 3D Object Detection Network with 2D Region-Oriented Attention (ROA-BEV), which enables the backbone to focus more on feature learning of the regions where objects exist. Moreover, our method further enhances the information feature learning ability of ROA through multi-scale structures. Each block of ROA utilizes a large kernel to ensure that the receptive field is large enough to catch information about large objects. Experiments on nuScenes show that ROA-BEV improves the performance based on BEVDepth. The source codes of this work will be available at https://github.com/DFLyan/ROA-BEV.

Keywords: Deep Learning for Visual Perception, Object Detection, Segmentation and Categorization, Computer Vision for Automation
Abstract: Compact and efficient 6DoF object pose estimation is crucial in applications such as robotics, augmented reality, and space autonomous navigation systems, where lightweight models are critical for real-time accurate performance. This paper introduces a novel uncertainty-aware end-to-end Knowledge Distillation (KD) framework focused on keypoint-based 6DoF pose estimation. Keypoints predicted by a large teacher model exhibit varying levels of uncertainty that can be exploited within the distillation process to enhance the accuracy of the student model while ensuring its compactness. To this end, we propose a distillation strategy that aligns the student and teacher predictions by adjusting the knowledge transfer based on the uncertainty associated with each teacher keypoint prediction. Additionally, the proposed KD leverages this uncertainty-aware alignment of keypoints to transfer the knowledge at key locations of their respective feature maps. Experiments on the widely-used LINEMOD benchmark demonstrate the effectiveness of our method, achieving superior 6DoF object pose estimation with lightweight models compared to state-of-the-art approaches. Further validation on the SPEED+ dataset for spacecraft pose estimation highlights the robustness of our approach under diverse 6DoF pose estimation scenarios.

Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Semantic Scene Understanding
Abstract: Deep learning-based multi-view stereo (MVS) methods enable dense point cloud reconstruction in texture-rich areas. However, existing methods incur significant computational costs to capture pixel dependencies for complete reconstruction in low-texture regions. Additionally, discrete depth layers in occluded environments hinder the cost volume's ability to model object information effectively. To address these issues, we propose a spatial-aware multi-view stereo network with attention cost volume, termed SA-MVSNet. The network introduces the pixel-driven spatial interaction (PDSI) module, which integrates the hierarchical spatial location enhancement mechanism (HSLE) and the spatial context aggregation mechanism (SCA). Leveraging an efficient parallel architecture, the PDSI module captures pixel-level spatial dependencies with the HSLE and strengthens global contextual information through the SCA. This design improves the network's ability to represent features in low-texture regions while maintaining high inference efficiency. Furthermore, SA-MVSNet incorporates an attention weight generation branch that refines the cost volume by aggregating multi-scale depth cues, effectively mitigating the impact of occlusion. Experiments on the DTU dataset and the Tanks and Temples dataset show that our method outperforms other learning-based methods, achieving superior performance and strong generalization ability.

Keywords: Localization, Deep Learning Methods, Multi-Robot Systems
Abstract: Ultra-wideband (UWB)-vision fusion localization has achieved extensive applications in the domain of multi-agent relative localization. The challenging matching problem between robots and visual detection renders existing methods highly dependent on identity-encoded hardware or delicate tuning algorithms. Overconfident yet erroneous matches may bring about irreversible damage to the localization system. To address this issue, we introduce Mr. Virgil, an end-to-end learning multi-robot visual-range relative localization framework, consisting of a graph neural network for data association between UWB rangings and visual detections, and a differentiable pose graph optimization (PGO) back-end. The graph-based front-end supplies robust matching results, accurate initial position predictions, and credible uncertainty estimates, which are subsequently integrated into the PGO back-end to elevate the accuracy of the final pose estimation. Additionally, a decentralized system is implemented for real-world applications. Experiments spanning varying robot numbers, simulation and real-world, occlusion and non-occlusion conditions showcase the stability and exactitude under various scenes compared to conventional methods. Our code is available at: https://github.com/HiOnes/Mr-Virgil.

Keywords: Deep Learning Methods, Motion and Path Planning, Intelligent Transportation Systems
Abstract: A thorough understanding of the interaction between the target agent and surrounding agents is a prerequisite for accurate trajectory prediction. Although many methods have been explored, they assign correlation coefficients to surrounding agents in a purely learning-based manner. In this study, we present ASPILin, which manually selects interacting agents and replaces the attention scores in Transformer with a newly computed physical correlation coefficient, enhancing the interpretability of interaction modeling. Surprisingly, these simple modifications can significantly improve prediction performance and substantially reduce computational costs. We intentionally simplified our model in other aspects, such as map encoding. Remarkably, experiments conducted on the INTERACTION, highD, and CitySim datasets demonstrate that our method is efficient and straightforward, outperforming other state-of-the-art methods.

Keywords: Deep Learning Methods, Motion and Path Planning, Reinforcement Learning
Abstract: Path planning is usually solved by addressing either the (high-level) route planning problem (waypoint sequencing to achieve the final goal) or the (low-level) path planning problem (trajectory prediction between two waypoints avoiding collisions). However, real-world problems usually require simultaneous solutions to the route and path planning subproblems with a holistic and efficient approach. In this paper, we introduce NaviFormer, a deep reinforcement learning model based on a Transformer architecture that solves the global navigation problem by predicting both high-level routes and low-level trajectories. To evaluate NaviFormer, several experiments have been conducted, including comparisons with other algorithms. Results show competitive accuracy from NaviFormer since it can understand the constraints and difficulties of each subproblem and act consequently to improve performance. Moreover, its superior computation speed proves its suitability for real-time missions.

Keywords: Localization, Deep Learning Methods, Visual Learning
Abstract: This paper presents a novel feature alignment strategy for cross-view geo-localization to address the large viewpoint differences between ground and satellite views. Existing methods for cross-view geo-localization often overlook factors such as occlusion and distortion errors caused by viewpoint transformation. These issues lead to reduced accuracy in complex scenes. To address this issue, we propose a framework comprising two novel components: a perspective-driven attention fusion (PDAF) module that aligns ground and satellite features through cross-view semantic correlation, effectively preserving structural consistency during view transformation; and a projection-stable patch-guided pose optimizer (PSPG) that enhances geometric reliability by selectively focusing on projection-stable patch to refine pose estimation. The PDAF module mitigates information loss through attention fusion between ground and bird's-eye-view (BEV) feature maps representations, while the PSPG refines pose estimation by dynamically suppressing unstable features through geometrically unstable token merging. Comprehensive evaluations on KITTI and Ford Multi-AV datasets demonstrate our method's superiority in orientation estimation and competitive location accuracy compared to state-of-the-art approaches. Qualitative results further confirm the framework's robustness in complex localization scenarios.

Keywords: Legged Robots, Localization, Deep Learning Methods
Abstract: This paper presents an algorithm to improve state estimation for legged robots. Among existing model-based state estimation methods for legged robots, the contact-aided invariant extended Kalman filter defines the state on a Lie group to preserve invariance, thereby significantly accelerating convergence. It achieves more accurate state estimation by leveraging contact information as measurements for the update step. However, when the model exhibits strong nonlinearity, the estimation accuracy decreases. Such nonlinearities can cause initial errors to accumulate and lead to large drifts over time. To address this issue, we propose compensating for errors by augmenting the Kalman filter with an artificial neural network serving as a nonlinear function approximator. Furthermore, we design this neural network to respect the Lie group structure to ensure invariance, resulting in our proposed Invariant Neural-Augmented Kalman Filter (InNKF). The proposed algorithm offers improved state estimation performance by combining the strengths of model-based and learning-based approaches. Project webpage: https://seokju-lee.github.io/innkf_webpage

Keywords: Localization, Deep Learning Methods, Vision-Based Navigation
Abstract: We present a novel machine learning framework and synthetic dataset for performing absolute localization on planetary surfaces where satellite navigation systems are unavailable. Current approaches involve manual surface-to-satellite image matching by human rover operators, limiting the rate of planetary exploration and scientific utilization. Our framework leverages deep neural networks to perform image similarity matching between a rover’s onboard cameras and corresponding ground-view images from a digital twin environment created from extracted satellite and elevation maps. The rover views, satellite, and elevation maps are taken from a photorealistic lunar environment simulated in a 3D graphics engine (Unreal Engine 4). The synthetic ground-view re-projections are generated using an open-source 3D graphics software (Blender). In total, we generate a dataset of 1.68 million images at 210,000 locations. The images and corresponding metadata are then used to train a DINOv2 vision transformer image similarity model through supervised fine-tuning to determine matching locations between the rover views and candidate re-projections. Through this method, our model is able to determine the ground truth location within 5 m using just 2.5% of the search space, outperforming other deep learning and classical image comparison benchmarks.

Keywords: Deep Learning Methods, Localization, Sensor Fusion
Abstract: This paper presents a learning-based online IMU compensation method (AGCNet) that can compensate for run time errors of the accelerometer and gyroscope to improve inertial odometry. AGCNet employs U-Net architecture with hybrid dilated convolutions to extract multiscale features. It also adopts skip connections and patch-based processing strategy to aggregate local and global information. The network is trained to minimize absolute errors between integration results derived from compensated IMU data and ground truth motion states. The network utilizes IMU measurements from the current time window to correct errors in the subsequent time window, enabling sparser computations. Experiments on two public visual inertial datasets show that AGCNet can accurately estimate the orientation from IMU measurements, outperforming existing learning-based methods. When applied to Open-VINS, AGCNet improves the accuracy of orientation estimation by an average of 29.8% and position estimation by an average of 37.3%.

Keywords: Localization, Deep Learning for Visual Perception
Abstract: Visual place recognition (VPR) is crucial for robots to identify previously visited locations, playing an important role in autonomous navigation in both indoor and outdoor environments. However, most existing VPR datasets are limited to single-viewpoint scenarios, leading to reduced recognition accuracy, particularly in multi-directional driving or feature-sparse scenes. Moreover, obtaining

additional data to mitigate these limitations is often expensive. This paper introduces a novel training paradigm to improve the performance of existing VPR networks by enhancing multi-view diversity within current datasets through uncertainty estimation and NeRF-based data augmentation. Specifically, we initially train NeRF using the existing VPR dataset. Then, our devised self-supervised uncertainty estimation network identifies places with high uncertainty. The poses of these uncertain places are input into NeRF to generate new synthetic observations for further training of VPR networks. Additionally, we propose an improved storage method for efficient organization of augmented and original training data. We conducted extensive experiments on three datasets and tested three different VPR backbone networks. The results demonstrate that our proposed training paradigm significantly improves VPR performance by fully utilizing existing data, outperforming other training approaches. We further validated the

effectiveness of our approach on self-recorded indoor and outdoor datasets, consistently demonstrating superior results. Our dataset and code have been released at https://anonymous.4open.science/r/UGNA-VPR-DDF0.

Keywords: Telerobotics and Teleoperation, AI-Enabled Robotics, Learning from Demonstration
Abstract: Teleoperation is crucial for autonomous robot learning, especially in manipulation tasks involving data collection for Imitation Learning (IL) and Reinforcement Learning (RL), such as Vision-Language-Action (VLA) and Human-In-The-Loop RL (HITL RL). Existing teleoperation systems typically provide unilateral control to the robot, lacking the ability to synchronize status from the robot. In this work, we introduce HACTS (Human-As-Copilot Teleoperation System), a novel system that enables bilateral, real-time joint synchronization between the robot arm and the teleoperation hardware. This simple yet effective feedback mechanism, akin to a steering wheel in an autonomous vehicle, allows the human copilot to intervene when necessary while simultaneously collecting action-correction data for future learning The HACTS hardware is implemented using 3D-printed components and low-cost, off-the-shelf motors, demonstrating both accessibility and scalability. Our experiments show that HACTS significantly enhances the performance of representative IL and RL methods for robot manipulation. Specifically, action-correction data improves the recovery capabilities of VLA models and facilitates human-in-the-loop reinforcement learning. HACTS paves the way for more effective and interactive human-robot collaboration and data-collection, advancing the capabilities of robot learning.

Keywords: Telerobotics and Teleoperation, Human Performance Augmentation, Aerial Systems: Applications
Abstract: Omnidirectional aerial robots offer full 6-DoF independent control over position and orientation, making them popular for aerial manipulation. Although advancements in robotic autonomy, human operation remains essential in complex aerial environments. Existing teleoperation approaches for multirotors fail to fully leverage the additional DoFs provided by omnidirectional rotation. Additionally, the dexterity of human fingers should be exploited for more engaged interaction. In this work, we propose an aerial teleoperation system that brings the rotational flexibility of human hands into the unbounded aerial workspace. Our system includes two motion-tracking marker sets--one on the shoulder and one on the hand--along with a data glove to capture hand gestures. Using these inputs, we design four interaction modes for different tasks, including Spherical Mode and Cartesian Mode for long-range moving, Operation Mode for precise manipulation, as well as Locking Mode for temporary pauses, where the hand gestures are utilized for seamless mode switching. We evaluate our system on a vertically mounted valve-turning task in the real world, demonstrating how each mode contributes to effective aerial manipulation. This interaction framework bridges human dexterity with aerial robotics, paving the way for enhanced aerial teleoperation in unstructured environments.

Keywords: Telerobotics and Teleoperation, Human Factors and Human-in-the-Loop, Physical Human-Robot Interaction
Abstract: In teleoperated surgery, the motion scaling factor directly influences both the operator's control precision of surgical instruments and operational comfort. Previous studies have revealed that the master manipulator state and operator's gaze information can reflect the complexity of surgical operations and the operator's intention to some extent. Although enabling real-time adjustment of scaling factors, they were limited by the narrow range of core parameters and the results were significantly influenced by subjective factors. To tackle these challenges, this paper presents a multi-dimensional adaptive motion scaling strategy based on the Bayesian optimization. The prediction of operator's intention and attention is achieved by integrating multiple dimensional parameters, including master-slave manipulator states, gaze information, as well as pupillary data, all of which have been experimentally validated. Specifically, there exists a significant temporal synchronization between the Index of Pupillary Activity (IPA) and teleoperation tasks, which aligns with research on the correlation between IPA and attention levels. Furthermore, to evaluate the proposed adaptive scaling strategy, we combine subjective questionnaire surveys with objective metric assessments, effectively reducing the excessive influence of operators' personal conditions and proficiency levels on optimization results.

Keywords: Telerobotics and Teleoperation, Learning from Demonstration, Haptics and Haptic Interfaces
Abstract: In this paper, we propose a concept of Teleoperated Teaching of Task and Impedance (TTTI) with a novel multi-modal interface that enables online teleoperated teaching of combined low-level impedance-regulation skills and high-level task decision-making skills using a single hand-held haptic device. To this end, we interactively switch the functionality of the haptic device for two modes of operation. To teach impedance-regulation low-level skills, we developed a novel stiffness command interface where the human operator uses the haptic device to manipulate the stiffness ellipsoid of the remote robotic arm endpoint in 3D space. For teaching high-level skills of how and when to employ low-level actions, we developed a GUI that enables a haptic device to remotely modify Behaviour Trees used to encode the robot's task decision-making process. The interface connects both teaching modes, where a newly demonstrated low-level skill appears in the Behaviour Tree at an operator-specified index. To demonstrate the main features of the proposed interface, we performed several proof-of-concept experiments on a teleoperation setup operating a remote shelf-stocker robot in a simulated supermarket environment. We examined the task of placing a product on a shelf that consists of several sub-tasks, where each involves different stiffness strategies, while the Behaviour Tree has to encode the task sequencing and decision-making process.

Keywords: Telerobotics and Teleoperation, Haptics and Haptic Interfaces, Medical Robots and Systems
Abstract: Tele-ultrasound has the potential greatly to improve health equity for countless remote communities. However, practical scenarios involve potentially large time delays which cause current implementations of telerobotic ultrasound (US) to fail. Using a local model of the remote environment to provide haptics to the expert operator can decrease teleoperation instability, but the delayed visual feedback remains problematic. This paper introduces a robotic tele-US system in which the local model is not only haptic, but also visual, by re-slicing and rendering a pre-acquired US sweep in real time to provide the operator with a preview of what the delayed image will resemble. A prototype system is presented and tested with 15 volunteer operators. It is found that visual-haptic model-mediated teleoperation (MMT) compensates completely for time delays up to 1000 ms round trip in terms of operator effort and completion time while conventional MMT does not. Visual-haptic MMT also significantly outperforms MMT for longer time delays in terms of motion accuracy and force control. This proof-of-concept study suggests that visual-haptic MMT may facilitate remote robotic tele-US.

Keywords: Telerobotics and Teleoperation, Imitation Learning, Human-Robot Collaboration
Abstract: Teleoperated robotic manipulators enable the collection of demonstration data, which can be used to train control policies through imitation learning. However, such methods can require significant amounts of training data to develop robust policies or adapt them to new and unseen tasks. While expert feedback can significantly enhance policy performance, providing continuous feedback can be cognitively demanding and time-consuming for experts. To address this challenge, we propose to use a cable-driven teleoperation system which can provide spatial corrections with 6 degree of freedom to the trajectories generated by a policy model. Specifically, we propose a correction method termed Decaying Relative Correction (DRC) which is based upon the spatial offset vector provided by the expert and exists temporarily, and which reduces the intervention steps required by an expert. Our results demonstrate that DRC reduces the required expert intervention rate by 30% compared to a standard absolute corrective method. Furthermore, we show that integrating DRC within an online imitation learning framework rapidly increases the success rate of manipulation tasks such as raspberry harvesting and cloth wiping.

Keywords: Telerobotics and Teleoperation, Human Detection and Tracking, Imitation Learning
Abstract: Imitation-based teleoperation enables intuitive robot control in hazardous or hard-to-reach environments. Existing methods, however, lack an effective and quickly-deployable system that uses simple visual sensors to achieve end-effector control and human-like arm joint configuration imitation across various robotic arm structures. This paper therefore presents a teleoperation system that utilizes a single RGB camera and advanced computer vision techniques to capture human motion, coupled with a kinematic mapping method to transfer movements from human to robotic arms. The system generates robot motion that ensures both end-effector tracking and human-like joint configuration imitation, adaptable to diverse structures, including those with multiple offset links. Experiments demonstrate that the system produces robot arm poses more closely aligned with human configurations compared to traditional methods that overlook human pose. The performance of the end-effector tracking control and human arm shape imitation is evaluated, with no noticeable error observed when the robot completes its motion and a maximum position error of 17.03% and a maximum orientation error of 0.0925 rad are observed during motion, which are likely attributed to delays cased by filters and communications. Additionally, the system's ability to actively avoid obstacles via arm configuration imitation in specific scenarios is confirmed. Supplementary video is available.

Keywords: Telerobotics and Teleoperation, Haptics and Haptic Interfaces, Force and Tactile Sensing
Abstract: Vision-based teleoperation systems are widely used due to their cost-effectiveness and intuitive operation. However, these systems often suffer from challenges such as hand occlusions, environmental variability, and the lack of tactile feedback, limiting their precision and applicability in complex tasks. To address these limitations, we present AirTouch, a novel, low-cost visuotactile teleoperation system that integrates air pressure-based tactile feedback with lightweight hand pose estimation. AirTouch features an inflatable tactile bubble that provides adjustable feedback through closed-loop pneumatic control, enhancing the operator's sense of interaction with remote environments. The system's robust hand-tracking algorithm ensures accurate control even under dynamic and occlusion-prone conditions, while its hardware design eliminates the need for wearable devices, enabling intuitive operation. AirTouch supports a wide range of robotic end-effectors, including dexterous hands, parallel grippers, and suction cups, demonstrating versatility across multiple platforms. Extensive experiments validate AirTouch's performance, achieving high precision in hand pose estimation and a 91% success rate in complex teleoperation tasks, all with a hardware cost as low as 39. These results highlight AirTouch as a scalable and practical solution for enhancing robotic teleoperation across industrial, medical, and hazardous scenarios.

Keywords: Task and Motion Planning, Task Planning, Representation Learning
Abstract: Large Language Models (LLMs) have demonstrated exceptional reasoning capabilities. However, in real-world robotics tasks, LLMs face grounding issues and lack precise feedback, resulting in generated solutions that run counter to reality. In this paper, we present double feedback, a method that enhances LLM reasoning through knowledge graphs (KGs). KG plays three key roles in Double-Feedback: prompting LLMs to generate solutions, representing task scenarios, and validating solutions to provide feedback. We design structured knowledge prompts that convey the knowledge background of the task, example solutions, revision principles, and robotic tasks to the LLM. We also introduce distributed representations to quantify task scenarios with interpretability. Based on structured knowledge prompts and distributed representations, we utilize KG to evaluate the feasibility of each step before execution and validate the effectiveness of the solution after the task is completed. LLMs can adapt and reprogram solutions based on KG feedback. Extensive experiments have shown that dual feedback is superior to previous work in ALFRED benchmarks. Additionally, ablation studies have shown that dual feedback guides LLMs in generating solutions that align with real-world robotic tasks.

Keywords: Task and Motion Planning, Reinforcement Learning, Multi-Robot Systems
Abstract: In swarm robotics, confrontation including the pursuit-evasion game is a key scenario. High uncertainty caused by unknown opponents' strategies, dynamic obstacles, and insufficient training complicates the action space into a hybrid decision process. Although the deep reinforcement learning method is significant for swarm confrontation since it can handle various sizes, as an end-to-end implementation, it cannot deal with the hybrid process. Here, we propose a novel hierarchical reinforcement learning approach consisting of a target allocation layer, a path planning layer, and the underlying dynamic interaction mechanism between the two layers, which indicates the quantified uncertainty. It decouples the hybrid process into discrete allocation and continuous planning layers, with a probabilistic ensemble model to quantify the uncertainty and regulate the interaction frequency adaptively. Furthermore, to overcome the unstable training process introduced by the two layers, we design an integration training method including pre-training and cross-training, which enhances the training efficiency and stability. Experiment results in both comparison, ablation, and real-robot studies validate the effectiveness and generalization performance of our proposed approach. In our defined experiments with twenty to forty agents, the win rate of the proposed method reaches around ninety percent, outperforming other traditional methods.

Keywords: Task and Motion Planning, Motion and Path Planning, Simulation and Animation
Abstract: The local extremum is a crucial factor that affects the efficiency of online coverage path planning (CPP). Most online CPP methods generate coverage motions point-by-point in unknown environments. However, these solutions ignore efficient global coverage and probably result in local extremum. This paper presents a hierarchy coverage path planning approach (HCPP) with proactive extremum prevention. HCPP incrementally generates coverage tasks and produces coverage motions in a global-to-local planning manner. Global planning generates a traversal sequence of all coverage tasks, and local planning provides a route from one task to the next. By maintaining the connectivity of the uncovered area from both a global and local perspective, HCPP avoids the local extremums caused by separate areas. The effectiveness of HCPP was confirmed by multiple simulations and physical experiments in a laboratory setting on an Akerman robot. Experimental results indicate that HCPP reduces coverage times by preventing local extremum while achieving complete coverage.

Keywords: Task and Motion Planning, Probabilistic Inference
Abstract: Planning for sequential robotics tasks often requires integrated symbolic and geometric reasoning. Task and Motion Planning (TAMP) algorithms typically solve these problems by performing a tree search over high-level task sequences while checking for kinematic and dynamic feasibility. This can be inefficient because, typically, candidate task plans resulting from the tree search ignore geometric information. This often leads to motion planning failures that require expensive backtracking steps to find alternative task plans. We propose a novel approach to TAMP called Stein Task and Motion Planning (STAMP) that relaxes the hybrid optimization problem into a continuous domain. This allows us to leverage gradients from differentiable physics simulation to fully optimize discrete and continuous plan parameters for TAMP. In particular, we solve the optimization problem using a gradient-based variational inference algorithm called Stein Variational Gradient Descent. This allows us to find a distribution of solutions within a single optimization run. Furthermore, we use an off-the-shelf differentiable physics simulator that is parallelized on the GPU to run parallelized inference over diverse plan parameters. We demonstrate our method on a variety of problems and show that it can find multiple diverse plans in a single optimization run while also being significantly faster than existing approaches. https://rvl.cs.toronto.edu/stamp

Keywords: Task Planning, Manipulation Planning, Deep Learning Methods
Abstract: Deformable object manipulation stands as one of the most captivating yet formidable challenges in robotics. While previous techniques have predominantly relied on learning latent dynamics through demonstrations, typically represented as either particles or images, there exists a pertinent limitation: acquiring suitable demonstrations,

especially for long-horizon tasks, can be elusive. Moreover, basing learning entirely on demonstrations can hamper the model's ability to generalize beyond the demonstrated tasks. In this work, we introduce a demonstration-free hierarchical planning approach capable of tackling intricate long-horizon deformable manipulation tasks without necessitating any training. We employ large language models (LLMs) to articulate a high-level, stage-by-stage plan corresponding to a specified task. For every individual stage, the LLM provides both the tool's name and the Python code to craft intermediate subgoal point clouds. With the tool and subgoal for a particular stage at our disposal, we present a granular closed-loop model predictive control strategy. This leverages Differentiable Physics with Point-to-Point correspondence (DiffPhysics-P2P) loss in the earth mover distance (EMD)

space, applied iteratively. Experimental findings affirm that our technique surpasses multiple benchmarks in dough manipulation, spanning

both short and long horizons. Remarkably, our model demonstrates robust

generalization capabilities to novel and previously unencountered complex tasks without any preliminary demonstrations. We further substantiate our approach with experimental trials on real-world robotic platforms. Our project page: https://qq456cvb.github.io/projects/donut.

Keywords: Task Planning, Human-Robot Collaboration
Abstract: Large language models (LLMs) have gained increasing popularity in robotic task planning due to their exceptional abilities in text analytics and generation, as well as their broad knowledge of the world. However,they fall short in decoding visual cues. LLMs have limited direct perception of the world, which leads to a deficient grasp of the current state of the world. By contrast, the emergence of visual language models (VLMs) fills this gap by integrating visual perception modules, which can enhance the autonomy of robotic task planning. Despite these advancements, VLMs still face challenges, such as the potential for task execution errors, even when provided with accurate instructions. To address such issues, this letter proposes a ReplanVLM framework for robotic task planning. In this study, we focus on error correction interventions. An internal error correction mechanism and an external error correction mechanism are presented to correct errors under corresponding phases. A replan strategy is developed to replan tasks or correct error codes when task execution fails. Experimental results on real robots and in simulation environments have demonstrated the superiority of the proposed framework, with higher success rates and robust error correction capabilities in open-world tasks.

Keywords: Social HRI, Human-Aware Motion Planning, Collision Avoidance
Abstract: Robots are becoming essential in human environments, requiring them to behave in a socially compliant manner. Although previous learning-based methods have shown potential in social navigation, most have treated pedestrians as individuals, failing to account for group level interactions. Additionally, these methods have modeled pairwise interactions only in the spatial domain, overlooking the temporal evolution of relations among agents. In this letter, the above limitations are addressed by proposing a novel spatio-temporal graph attention network that explicitly models group level interactions in both spatial and temporal domains. Specifically, a novel group-awareness mechanism is designed to model group-aware behaviors, and a new network is proposed to capture spatio-temporal features of relations among agents while leveraging the model-free deep reinforcement learning to optimize the group-aware navigation policy. The test results show that our approach outperforms the baselines in all metrics in both simulation and real-world experiments. Furthermore, quantitative analysis of questionnaire responses further verifies the benefits of our method in group awareness and social compliance.

Keywords: Constrained Motion Planning, Dynamics, Underactuated Robots
Abstract: The wheeled bipedal robot (WBR) is a type of robot with strong athletic ability. It combines the advantages of wheeled and legged robots while possessing efficient motion performance and good terrain adaptability. However, research on the jumping motion of WBR is currently limited, mainly focusing on relatively simple jumping actions. This paper explores the WBR's somersaulting jump, expanding the degrees of freedom in jumping actions and further enhancing its mobility. A comprehensive planning and control strategy is proposed. Specifically, the paper first designs the entire process of somersaulting jump, including preparation, flight, and recovery, based on human motion. Building on this, the dynamic models for both phases are derived. Critical states at take-off and landing are carefully designed. Moreover, trajectory planning considers constraints such as system dynamics, joint limits, and torque restrictions, ensuring the rationality of the trajectory. Subsequently, effective controllers are constructed to ensure the stability of the somersaulting jump motion. Finally, the effectiveness and adaptability of the overall planning and control strategy are verified through physical simulation.

Keywords: Climbing Robots, Computer Vision for Automation, Field Robots
Abstract: This paper presents an innovative robotic inspection system designed to enhance the detection and analysis of structural defects in concrete infrastructure. The proposed inspection system is comprised of three modules: a robotic data collection module, a visual inspection module, and a subsurface mapping module. The robotic data collection module features an omnidirectional robotic platform, designed to move sideways without spinning. It is equipped with Ground Penetrating Radar (GPR) and RGB-D cameras, facilitating systematic data collection across construction sites. The visual inspection module employs a learning-based method, InspectionNet++, to analyze the frames for surface defects such as cracks, spalls, and stains, providing high accuracy and metric measurements of the defects. The subsurface mapping module processes the GPR data to detect and visualize hidden defects, creating a comprehensive map that correlates these with visible surface anomalies. Field tests demonstrate the system’s ability to automate construction structural inspection with improved efficiency and precision. Additionally, the customized visualization software is introduced to enable intuitive and interactive exploration of the detected defects within a unified interface. By automating data collection and enhancing defect detection through learning algorithms, the system not only speeds up the inspection process but also increases the reliability of infrastructure evaluations, supporting more informed maintenance decisions.

Keywords: Field Robots, Collision Avoidance, Reactive and Sensor-Based Planning
Abstract: This paper describes methodologies to estimate the collision probability, Euclidean distance and gradient between a robot and a surface, without explicitly constructing a free space representation. The robot is assumed to be an ellipsoid, which provides a tighter approximation for navigation in cluttered and narrow spaces compared to the commonly used spherical model. Instead of computing distances over point clouds or high-resolution occupancy grids, which is expensive, the environment is modeled using compact Gaussian mixture models and approximated via a set of ellipsoids. A parallelizable strategy to accelerate an existing ellipsoid-ellipsoid distance computation method is presented. Evaluation in 3D environments demonstrates improved performance over state-of-the-art methods. Execution times for the approach are within a few microseconds per ellipsoid pair using a single-thread on low-power embedded computers.

Keywords: Field Robots, Autonomous Vehicle Navigation, Motion and Path Planning
Abstract: Navigating unknown underwater environments is a significant challenge for autonomous underwater vehicles (AUVs), especially those with torpedo-like shapes. Lacking a prior map, these vehicles rely on real-time sensor data for perception. Although online motion planning addresses this challenge, many existing methods are primarily tested on more maneuverable robots, such as multicopters and ground vehicles, and do not account for the unique kinematics of torpedo-shaped AUVs, such as limited lateral movement, or the need for 3D motion planning. In this paper, we propose an online motion planning system specifically designed for torpedo-shaped AUVs to navigate 3D underwater terrain without prior environmental knowledge. The system employs a receding horizon planning framework to ensure safe navigation by replanning the trajectory when collisions are detected or the planning horizon is reached. For trajectory generation, a search-based method is used and utilizes a 3D Dubins curve heuristic to guide the generation of an optimal 3D trajectory that adheres to the AUV's kinematic constraints. To further enhance safety and smoothness, gradient-based optimization is applied to refine the trajectory. Experiments in simulated environments validate the proposed method, demonstrating its ability to generate safe trajectories for AUVs in complex and unknown environments. We release our code as an open-source package.

Keywords: Field Robots
Abstract: Unmanned aerial vehicles (UAVs) are critical in the automated inspection of wind turbine blades. Nevertheless, several issues persist in this domain. Firstly, existing inspection platforms encounter challenges in meeting the demands of automated inspection tasks and scenarios. Moreover, current blade stop angle estimation methods are vulnerable to environmental factors, restricting their robustness. Additionally, there is an absence of real-time blade detail prioritized exposure adjustment during capture, where lost details cannot be restored through post-optimization. To address these challenges, we introduce a platform and two approaches. Initially, a UAV inspection platform is presented to meet the automated inspection requirements. Subsequently, a Fermat point based blade stop angle estimation approach is introduced, achieving higher precision and success rates. Finally, we propose a blade detail prioritized exposure adjustment approach to ensure appropriate brightness and preserve details during image capture. Extensive tests, comprising over 120 flights across 10 wind turbine models in 5 operational wind farms, validate the effectiveness of the proposed approaches in enhancing inspection autonomy.

Keywords: Service Robotics, Field Robots, Robotics in Hazardous Fields
Abstract: In the oil and gas industry, scale accumulation on radiant coils within furnaces significantly reduces heat-transfer efficiency, leading to increased energy consumption. This paper introduces the REFINE-bot, a robotic system developed to improve the descaling process and operational efficiency in fired heaters. Unlike existing solutions which are mainly designed for specific tube sizes and positions and focused on inspection, the REFINE-bot integrates an adaptable clamping mechanism that adapts to both vertical and horizontal tubes of varying diameters (3"–8"), even in complex environments with narrow tube-to-tube and wall-to-tube gaps. We evaluate three different cleaning tools—a Knot End Brush, Wire Cup Brush, and Sandpaper—under simulated hard scale conditions in a lab environment, revealing cleaning tool limitations and optimal safety parameters to prevent tube damage. An adaptive force control is also developed to online adjust the position of the cleaning relative to the tube surface to address uneven scale heights. Additionally, the robot is successfully deployed in a real furnace setting to test its clamping and cleaning mechanisms on the actual scale. The results demonstrate superior cleaning performance at radiant coils of a furnace compared to traditional manual descaling methods, as evaluated by measured reductions in scale thickness and infrared thermal imaging.

Keywords: Field Robots, Motion and Path Planning, Autonomous Vehicle Navigation
Abstract: Path planning in uneven terrain scenarios is one of the core capabilities of intelligent off-road robots and vehicles. The complex terrain undulations often cause bumpy motion and sharp turns along the planned path, making smooth and safe path planning challenging. Most path planning methods in this community rely on dense point cloud maps as direct inputs, which inevitably incur high computational overhead for map representation and terrain assessment. To address these problems, we propose a novel path planning method toward uneven terrains, SEM-RRT*, which balances both planning quality and computational efficiency. First, we propose a map representation namely Statistical Elevation Map (SEM), which is lightweight to store and compute. Then, to enable fast terrain risk assessment, a terrain risk filter with omnidirectional and multi-scale characteristics is designed. Finally, we incorporate multi-objective cost evaluation, backward search, and rolling optimization strategies into the Informed RRT* framework, leveraging its path optimality on large scale map. Extensive experiments in challenging terrain scenarios, such as hills, canyons, and volcanic landscapes, show that SEM-RRT* outperforms existing methods in both path quality and computational time.

Keywords: Field Robots, Marine Robotics, Aerial Systems: Applications
Abstract: Hybrid aerial-underwater vehicles (HAUVs) are attracting significant interest for their unique capability to operate across both air and water. However, achieving a lightweight design coupled with efficient cross-domain performance remains a formidable challenge. This paper introduces Nezha-T, a novel, ultra-lightweight HAUV featuring a dual-stable floating state capability. This is achieved through an innovative center-of-gravity (CoG) arrangement method, which enables seamless transitions between upright and horizontal floating postures. This ability to stably floating in either orientation is crucial for stable water entry and exit, mitigating impact forces on the vehicle and its payload. Furthermore, to counteract the residual buoyancy inherent in this design, a zero-lift pitch angle is incorporated into the control system, improving depth-keeping and pitch control performance. The proposed design were rigorously validated through computational fluid dynamics (CFD) simulations, pool experiments, and open-water field tests. The results confirm the feasibility of the dual-stable floating states, emonstrate stable cross-domain traversal, verify the effectiveness of the depth-keeping control system, and validate the vehicle's fixed-wing flight capability.

Keywords: Field Robots, Robotics in Hazardous Fields, Mining Robotics
Abstract: A LiDAR-IMU fusion system utilizing adaptive scanning is developed for high-resolution deformation monitoring of underground coal mine infrastructure, such as sealed walls. The system integrates data from a LiDAR scanner and an IMU, employing a penalty function-based scanning strategy to optimize point cloud quality. Following feature extraction and state estimation, a 3D point cloud model of the sealed wall is constructed. Deformation monitoring is achieved through point cloud segmentation, registration, and error analysis across multiple time intervals. A methodology for optimizing equipment placement on walls of varying dimensions is proposed to efficiently capture deformation details. Two metrics, PATD and PARE, are introduced to evaluate system performance. Calibration experiments using standardized boards and blocks are designed to determine optimal monitoring parameters, including distance, height, and sampling frequency. Simulated deformation experiments under real-world conditions validate the system’s rationality and accuracy.

Keywords: Mapping, Localization, Autonomous Vehicle Navigation
Abstract: Accurate positioning is essential for autonomous driving, but localization using 2D maps is challenging due to the domain gap between perspective view and 2D map. While GNSS accuracy is often limited by atmospheric effects, multipath, and signal blockages. We propose a novel positioning method that combines perspective view images with satellite images retrieved based on rough GNSS positions to achieve precise three-degree-of-freedom (3-DoF) pose estimation. Our method leverages the Swin Transformer for satellite image processing and semantic completion for monocular image analysis. By extracting depth and semantic information from monocular images, we convert these to overhead projections, effectively bridging the gap between different viewpoints. This cross-view transformation allows for precise alignment of features from monocular images onto semantically enriched satellite images. Additionally, we integrate a robust global position estimator using the semantic information from satellite images to further enhance accuracy and robustness. The experimental results demonstrate that our method excels in various complex scenarios; we successfully improved the positioning accuracy within 1 m to 80.67% and the heading in 1◦ to 33.78%. However, longitudinal localization remains more challenging, with higher errors than lateral positioning.

Keywords: Mapping, Sensor Fusion, Deep Learning for Visual Perception
Abstract: The construction of high-definition (HD) maps at intersections is crucial for autonomous driving and vehicle-to-infrastructure (V2I) collaboration. However, the semantic complexity of intersections poses significant challenges for HD mapping. Previous research has predominantly relied on traditional algorithms to process LiDAR or camera data, which often struggle with occlusion and inherent sensor limitations. To address these challenges, we propose a novel method, called ESFusion for Effective BEV Feature Selection and Fusion. To the best of our knowledge, this is the first work to leverage multi-modal data from intelligent roadside infrastructure, particularly LiDAR and cameras, for generating HD maps at intersections. To enhance multi-modal feature representation in Bird's Eye View (BEV), we design a Cross-modal Channel Exchange (CCE) module that creates multi-scale spatial features and facilitates LiDAR-camera information exchange across channels. Additionally, we introduce a Dynamic Feature Selection (DFS) module to adaptively select the most valuable information between modalities. Comprehensive evaluations on the DAIR-V2X dataset demonstrate that our method outperforms single-modal approaches and existing state-of-the-art fusion methods for vehicle-side applications. Moreover, experiments on the nuScenes dataset further highlight the high flexibility of our proposed module, showcasing its ability to be seamlessly integrated into existing multi-modal fusion workflows.

Keywords: Mapping, SLAM
Abstract: This paper presents a novel collaborative online dense mapping system for multiple Unmanned Aerial Vehicles (UAVs). The system confers two primary benefits: it facilitates simultaneous UAVs co-localization and real-time dense map reconstruction, and it recovers the metric scale even in GNSS-denied conditions. To achieve these advantages, Ultra-wideband (UWB) measurements, monocular Visual Odometry (VO), and co-visibility observations are jointly employed to recover both relative positions and global UAV poses, thereby ensuring optimality at both local and global scales. In the proposed methodology, a two-stage optimization strategy is proposed to reduce optimization burden. Initially, relative Sim3 transformations among UAVs are swiftly estimated, with UWB measurements facilitating metric scale recovery in the absence of GNSS. Subsequently, a global pose optimization is performed to effectively mitigate cumulative drift. By integrating UWB, VO, and co-visibility data within this framework, both local geometric consistency and global pose accuracy are robustly maintained. Through comprehensive simulation and empirical real-world testing, we demonstrate that our system not only improves UAV positioning accuracy in challenging scenarios but also facilitates the high-quality, online integration of dense point clouds in large-scale areas. This research offers valuable contributions and practical techniques for precise, real-time map reconstruction using an autonomous UAV fleet, particularly in GNSS-denied environments.

Keywords: Mapping, SLAM, Sensor Fusion
Abstract: Recently, 3D Gaussian Splatting (3DGS) has garnered significant attention for its remarkable capacity to efficiently synthesize novel views with high fidelity. Nevertheless, 3DGS encounters challenges in accurately representing the geometry of real-world scenes. To address this issue, previous methods commonly utilize a depth-normal consistency term on 2D images to regulate the geometry of 3D Gaussians. However, these methods degrade in performance when dealing with low-texture surfaces or limited training views. In contrast, we present GPGS, a novel approach that directly regulates Gaussians in 3D space using Geometric Priors (GP). Given posed LiDAR scans and images, we organize the point clouds into a hierarchical voxel map. Each voxel contains occupancy information and explicitly reveals the internal planar or non-planar structure. We propose a novel divide-and-conquer strategy to separately regulate Gaussians in planar and non-planar voxels. For planar voxels, we design positional and rotational constraints to align Gaussians with the estimated plane. Considering the noisy ranging measurements of complex structures, we use depth-normal consistency to regularize Gaussians in non-planar voxels. Additionally, an occupancy-aware density control strategy is introduced to confine the densification process within occupied voxels, thus reducing artifacts. Extensive experiments on real-world datasets show that our proposed approach outperforms existing state-of-the-art methods in both geometric accuracy and visual quality.

Keywords: Mapping
Abstract: Recent developments in 3D Gaussian Splatting have made significant advances in surface reconstruction. However, scaling these methods to large-scale scenes remains challenging due to high computational demands and the complex dynamic appearances typical of outdoor environments. These challenges hinder the application in aerial surveying and autonomous driving. This paper proposes a novel solution to reconstruct large-scale surfaces with fine details, supervised by full-sized images. Firstly, we introduce a coarse-to-fine strategy to reconstruct a coarse model efficiently, followed by adaptive scene partitioning and sub-scene refining from image segments. Additionally, we integrate a decoupling appearance model to capture global appearance variations and a transient mask model to mitigate interference from moving objects. Finally, we expand the multi-view constraint and introduce a single-view regularization for texture-less areas. Our experiments were conducted on the publicly available dataset GauU-Scene V2, which was captured using unmanned aerial vehicles. To the best of our knowledge, our method outperforms existing NeRF-based and Gaussian-based methods, achieving high-fidelity visual results and accurate surface from full-size image optimization. Open-source code will be available on GitHub.

Keywords: Mapping, SLAM, Localization
Abstract: Fixed-Lag smoothing is widely employed as a backend in localization tasks. Generally, increasing the window length leads to better accuracy, but demands more computational resources. Therefore, determining an appropriate window length and whether a fixed length should be maintained throughout the localization process are worth studying. Assuming independent and identically distributed noise based on the distance-independent characteristic of LiDAR ranging errors, we propose an uncertainty-based adaptive sliding window (ASW) strategy. Through mathematical derivation, the reference uncertainty is affected by the LiDAR feature distribution of each frame. Consequently, we develop a multi-modal LiDAR inertial odometry and mapping framework based on ASW, which integrates mechanical and solid-state LiDAR to enhance odometry accuracy and mapping density. By designing a joint matching module, our approach leverages the strengths of distinct scanning patterns. Additionally, we incorporate loop closure detection in the mapping process to minimize cumulative drift. Extensive experiments conducted on both public and self-collected datasets demonstrate the effectiveness of our method. Compared to the state-of-the-art method, our approach improves the average accuracy by 10.3%. We also provide an open-source implementation for further studies. https://github.com/wowhhhhgd/ASW-LIOM.

Keywords: Mapping, Localization
Abstract: Loop closure detection and pose estimation play a significant role in correcting odometry trajectories and generating globally consistent point cloud maps. Geometric feature descriptor methods neglect object-level spatial topology features, resulting in inadequate performance in loop closure detection. Semantic graph-based loop closing methods improve upon this, however, they still follow the paradigm of "first generating descriptors, then comparing similarity, and finally achieving alignment (6D pose)". Specifically, they compare two semantic graphs that are not spatially aligned, which makes direct node correspondences impossible and necessitates extensive descriptor extraction and comparison. This decouples similarity comparison from 6D pose estimation, resulting in a cumbersome process that limits practicality and scalability. This paper proposes SLOOP, a novel descriptor-free semantic graph matching method that "aligns two graphs first, followed by efficient similarity comparison". Specifically, we first design a dedicated neighborhood semantic feature module to extract high-quality matched node pairs. Next, we seek the aligned coordinate systems for candidate loops based on the robust ground normal vectors and two suitable node pairs examined by the two-stage global geometric consistency metrics. Finally, the aligned coordinate systems enable efficient extraction and comparison of node spatial distributions. We conducted extensive outdoor loop detection experiments and compared with various loop closure detection approaches, demonstrating the improved performance of SLOOP in loop closure detection and its practicality. The code and related materials are available at https://github.com/BIT-TYJ/SLOOP_c.

Keywords: Mapping, SLAM
Abstract: Building an online 3D LiDAR mapping system that produces a detailed surface reconstruction while remaining computationally efficient is a challenging task. In this paper, we present PlanarMesh, a novel incremental, mesh-based LiDAR reconstruction system that adaptively adjusts mesh resolution to achieve compact, detailed reconstructions in real-time. It introduces a new representation, planar-mesh, which combines plane modeling and meshing to capture both large surfaces and detailed geometry. The planar-mesh can be incrementally updated considering both local surface curvature and free-space information from sensor measurements. We employ a multi-threaded architecture with a Bounding Volume Hierarchy (BVH) for efficient data storage and fast search operations, enabling real-time performance. Experimental results show that our method achieves reconstruction accuracy on par with, or exceeding, state-of-the-art techniques—including truncated signed distance functions, occupancy mapping, and voxel-based meshing—while producing smaller output file sizes (10 times smaller than raw input and more than 5 times smaller than mesh-based methods) and maintaining real-time performance (around 2 Hz for a 64-beam sensor).

Keywords: Biologically-Inspired Robots, Biomimetics, Aerial Systems: Mechanics and Control
Abstract: Large flapping-wing aerial vehicles (FWAVs) face dual challenges in aerodynamic and structural design, with long-standing technical bottlenecks, particularly in roll maneuvers. In this study, by reverse-engineering the biomechanical mechanisms of raptor flight, we propose a bio-inspired wing-shoulder torsional mechanism and successfully developed an eagle-inspired flapping-wing aerial vehicle with a wingspan of 1.87m and a takeoff weight of 1,260g. A nonlinear explicit dynamics-lattice Boltzmann fluid-structure interaction (FSI) numerical model was innovatively established, comprehensively revealing the interaction mechanism between unsteady flapping flow fields and flexible wing deformations. Numerical simulations demonstrate that at a cruising speed of 8 m/s, the proposed mechanism generates a high-purity roll torque of 3.3 N·m (with a residual yaw torque of 0.2 N·m, torque purity ratio 16.5:1), while lift and thrust losses are below 1.5%. Flight experiments validate the exceptional performance of this mechanism in 3D maneuvers: a 360° barrel roll is completed in 2.6 seconds (average roll rate 136°/s). This study provides a theoretical framework and technological prototype for next-generation bio-inspired aerial vehicles that integrate efficient cruising with high maneuverability, marking the first instance where FWAVs surpass traditional aircraft in specific 3D maneuverability metrics.

Keywords: Biologically-Inspired Robots, Hydraulic/Pneumatic Actuators, Motion Control
Abstract: In this paper, we propose a design of an underwater soft snake-like robot prototype that uses two actuators made of 3D-printed soft materials to build the robot body. Control signals with appropriate displacement phases and different voltages are used to control the water pump to drive the soft actuator to bend to generate a sine wave with increasing amplitude from the head to the tail of the robot body. We test customized tail materials, phase shifts, and voltage growth rate signals to observe the effects of different parameters on the movement of the snake robot in water. Experiments show that the movement speed is positively correlated with the swing amplitude of the snake robot's motion module. In addition, measured data show that swimming efficiency and movement speed are also affected by tail flexibility and movement gait. When the phase offset is 2/3π, the tail is made of harder PLA material, and the voltage growth rate is 1.2, the maximum underwater movement speed achieved by the snake robot is 4.464 cm/s (0.076 BL/s). We also found that when the phase offset increases, the snake motion speed and motion efficiency first increase and then decrease. The results obtained in this study will aid in the advancement of soft, slender swimming robots and improve the understanding of the swimming capabilities of both robots and sea snakes.

Keywords: Biologically-Inspired Robots, Actuation and Joint Mechanisms, Soft Robot Applications
Abstract: In recent years, the development of robots capable of operating in both aerial and aquatic environments has gained significant attention. This study presents the design and fabrication of a novel aerial-aquatic locomotion robot (AALR). Inspired by the diving beetle, the AALR incorporates a biomimetic propulsion mechanism with power and recovery strokes. The variable stiffness propulsion module (VSPM) uses low melting point alloy (LMPA) and variable stiffness joints (VSJ) to achieve efficient aquatic locomotion while reducing harm to marine life. The AALR's innovative design integrates the VSPM into the arms of a traditional quadrotor, allowing for effective aerial-aquatic locomotion. The VSPM adjusts joint stiffness through temperature control, meeting locomotion requirements in both aerial and aquatic modes. A dynamic model for the VSPM was developed, with optimized dimensional parameters to increase propulsion force. Experiments focused on aquatic mode analysis and demonstrated the AALR's swimming capability, achieving a maximum swimming speed of 77 mm/s underwater. The results confirm the AALR's effective performance in water environment, highlighting its potential for versatile, eco-friendly operations.

Keywords: Biologically-Inspired Robots, Multifingered Hands
Abstract: The development of bionic hands is a crucial area robotics research, aiming to achieve human-like dexterity, adaptability, and efficiency in manipulation tasks. This study presents the Skeleton Bionic Hand (SkB-Hand), which features a dual elastic-tendon mechanism for extensor and volar plate in a single finger. The skeleton structure ensures the SkB-Hand is lightweight (< 600g) and low-cost (< 150USD) while maintaining performance in various grasping tasks. We evaluated the SkB-Hand through experiments such as the Kapandji test, GRASP Taxonomy, and dynamic scenarios. Results showed the SkB-Hand scored 7/10 in the Kapandji test and 33/33 in the GRASP Taxonomy test. The hand demonstrated resistance to deformation under external forces while ensuring flexibility. Integrated into the Techman Robot (TM-Robot), the SkB-Hand provide its potential for general robotic grasping tasks in daily life.

Keywords: Biologically-Inspired Robots, Aerial Systems: Mechanics and Control, Micro/Nano Robots
Abstract: Insects and hummingbirds exhibit remarkable agility, including full body flip maneuvers. Achieving similar performance in bio-inspired tailless flapping-wing robots (FWRs) is challenging due to the complex dynamics, inherent nonlinearities and control issues. This paper presents an nonlinear model predictive control (NMPC) algorithm designed to enable 360-degree flip maneuvers for the developed X-wing tailless FWR, which weighs 30.8 g and has a wingspan of 14.5 cm. We first introduce a high-fidelity model of the FWR, which incorporates the aerodynamics of the wings, dynamics of the motors and servos, body kinodynamic model, and the model of thrust and torques generation. This high-fidelity model allows for testing the FWR in a simulation, thus reducing the cost of flip maneuver of real-world experiments. Based on this high-fidelity model, we propose an NMPC controller to offline compute state trajectories and corresponding control inputs, which are then used as feedforward control for the FWR during its 360-degree flip maneuvers. Next, we present an online basic feedback controller that integrates the feedforward control for the FWR's flip control. Experimental results demonstrate the successful execution of the flip maneuvers without any mechanical modifications, highlighting the effectiveness of the proposed control strategy.

Keywords: Biologically-Inspired Robots, Biomimetics, Soft Sensors and Actuators
Abstract: Bioinspired magnetic cilia attract the tremendous attention of researchers due to flexible nature and remotely controlled manipulations of droplets and fluids. Nonetheless, controlling catalytic process by magnetic cilia remains underexplored. Here, we present magnetic soft cilia carpets with different magnetizations to control enzyme-like chemical reactions. We demonstrate a methodology to optimize material, geometric, and magnetic field parameters to achieve high-frequency oscillations of the cilia under an alternating magnetic field at 15 Hz. We also show stable oscillation of water-based droplets on the cilia surface at a magnetic field frequency of 10 Hz without droplet detachment. Furthermore, we demonstrate enhanced droplet catalysis resulting from droplet oscillation on the cilia surface. As a concept for automated lab analysis, we demonstrate droplet transport by magnetic cilia onto a flexible pH sensor. Finally, as a proof-of-concept, we show control of nanozyme reaction rates by varying cilia magnetization angles, achieving a fourfold enhancement of reaction rate under a rotating magnetic field compared to unmagnetized cilia. These findings offer a promising approach to remotely control enzyme-like reactions and droplet catalysis using magnetic cilia.

Keywords: Aerial Systems: Applications, Bioinspired Robot Learning, Machine Learning for Robot Control
Abstract: This paper develops a real-time brain-inspired learning control (RBiLC) strategy for adaptive tracking of quadcopters subject to nonlinear configuration uncertainties and time-varying uncertain dynamics. An online learning-evaluation-optimization mechanism addresses flight control law reconfiguration during unforeseen structural changes (e.g., propeller/motor faults). The RBiLC-based adaptive controller reduces design time and human intervention. Lyapunov-Krasovskii functions ensure stability under parametric uncertainties, while a signed sinusoidal perturbation estimate guides online learning direction and magnitude. Theoretical stability analysis under dynamic uncertainties proves the system converges to a compact set within finite time. Simulations and flight tests confirm the scheme achieves enhanced tracking performance and accelerated adaptation compared to existing methods.

Keywords: Brain-Machine Interfaces, Cognitive Control Architectures, Intention Recognition
Abstract: Multi-tasks decoding from electroencephalogram (EEG) signals is of great value for brain-computer interaction (BCI) applications in natural scenes. Although most existing studies have concentrated on decoding significantly different multi-tasks, a few studies explored the various cognitive processes that individuals may exhibit when elicited by the same stimulus. However, in practice, the diversity and complexity of individuals’ cognitive responses when faced with the same stimulus cannot be ignored. In this paper, we aimed to construct a paradigm of the various cognitive processes elicited by the same stimulus, explore the neural signatures, and decode the multiple cognitive processes from EEG signals. Experimental results show that the regularized linear discriminant analysis (RLDA) classifier with event-related spectral perturbation (ERSP) features yielded a decoding accuracy of 96.30% ± 3.40% for the multi-cognitions. In-depth research on signatures and decoding of various cognitive processes elicited by the same stimulus is of great significance for improving the naturalness and intelligence of BCI.

Keywords: Perception for Grasping and Manipulation, Force and Tactile Sensing, Contact Modeling
Abstract: Peg-hole-insertion processes of diverse shapes are typical contact-rich tasks, which need the accurate representation of object’s shape, pose, and peg-hole contact states. The visual-tactile sensor can perceive the relative moving trend between the gripper and the grasped object, which could be applied in the perception of the peg-hole contact states. In order to complete peg-hole insertion tasks, this manuscript proposes a method of using the visual-tactile sensor to estimate the relative position of peg and hole. Furthermore, it introduces the theory of topological and geometric reasoning to characterize the insertion process, which could be used for various polygon shaped pegs and holes. In five different shapes of peg and hole experiments, errors of peg-hole relative position estimation using the method proposed in this manuscript are almost within 5 degrees, which can meet the needs of insertion tasks. What’s more, insertion processes become more smooth by adopting the topological and geometric reasoning, indicating the effectiveness of the reasoning process.

Keywords: Perception for Grasping and Manipulation, Collision Avoidance, Deep Learning for Visual Perception
Abstract: Robust robotic grasping in cluttered environments presents a significant challenge, as existing methods often neglect the complex interactions between the gripper, objects, and obstacles, leading to collisions and grasping failures. To address this, we propose a framework that integrates collision avoidance as a core constraint within the grasp pose optimization process. Central to this framework is a Neural Collision Detection (NCD) network that takes scene configurations and grasp poses as inputs, producing a collision score that approximates traditional collision detection functions. The NCD network provides critical feedback for refining grasp predictions and demonstrates strong generalization across diverse environments, facilitating efficient collision detection and constrained grasp pose optimization. Additionally, we incorporate frictional force closure, geometric symmetry, and surface alignment as regularization terms within the optimization function, enhancing the physical stability and geometric plausibility of the generated grasps. Extensive experiments conducted in real-world environments show a significant improvement in grasp success rates, with robust generalization to previously unseen objects and scenarios. These results validate the efficacy of our framework, highlighting its potential for enabling reliable robotic manipulation in complex and cluttered environments.

Keywords: Perception for Grasping and Manipulation, Force and Tactile Sensing, Calibration and Identification
Abstract: Fine dexterous manipulation requires reactive control based on rich sensing of manipulator-object interactions. Tactile sensing arrays provide rich contact information across the manipulator's surface. However their implementation faces two main challenges: accurate force estimation across complex surfaces like robotic hands, and integration of these estimates into reactive control loops. We present a data-efficient calibration method that enables rapid, full-array force estimation across varying geometries, providing online feedback that accounts for non-linearities and deformation effects. Our force estimation model serves as feedback in an online closed-loop control system for interaction force tracking. The accuracy of our estimates is independently validated against measurements from a calibrated force-torque sensor. Using the Allegro Hand equipped with Xela uSkin sensors, we demonstrate precise force application through an admittance control loop running at 100Hz, achieving up to 0.12pm0.08 [N] error margin—results that show promising potential for dexterous manipulation.

Keywords: Perception for Grasping and Manipulation, Grasping, Computer Vision for Automation
Abstract: Grasp detection is a fundamental robotic task critical to the success of many industrial applications. However, current language-driven models for this task often struggle with cluttered images, lengthy textual descriptions, or slow inference speed. We introduce GraspMamba, a new language-driven grasp detection method that employs hierarchical feature fusion with Mamba vision to tackle these challenges. By leveraging rich visual features of the Mamba-based backbone alongside textual information, our approach effectively enhances the fusion of multimodal features. GraspMamba represents the first Mamba-based grasp detection model to extract vision and language features at multiple scales, delivering robust performance and rapid inference time. Intensive experiments show that GraspMamba outperforms recent methods by a clear margin. We validate our approach through real-world robotic experiments, highlighting its fast inference speed.

Keywords: Perception for Grasping and Manipulation, Deep Learning in Grasping and Manipulation, Grasping
Abstract: Recent advancements in 3D robotic manipulation have improved grasping of everyday objects, but transparent and specular materials remain challenging due to depth sensing limitations. While several 3D reconstruction and depth completion approaches address these challenges, they suffer from setup complexity or limited observation information utilization. To address this, leveraging the power of single-view 3D object reconstruction approaches, we propose a training-free framework SR3D that enables robotic grasping of transparent and specular objects from a single-view observation. Specifically, given single-view RGB and depth images, SR3D first uses the external visual models to generate 3D reconstructed object mesh based on RGB image. Then, the key idea is to determine the 3D object’s pose and scale to accurately localize the reconstructed object back into its original depth corrupted 3D scene. Therefore, we propose view matching and keypoint matching mechanisms, which leverage both the 2D and 3D's inherent semantic and geometric information in the observation to determine the object's 3D state within the scene, thereby reconstructing an accurate 3D depth map for effective grasp detection. Experiments in both simulation and real-world show the reconstruction effectiveness of SR3D.

Keywords: Perception for Grasping and Manipulation, Data Sets for Robotic Vision, Robotics and Automation in Construction
Abstract: Vision-based robotic object grasping is typically investigated in the context of isolated objects or unstructured object sets in bin picking scenarios. However, there are several settings, such as construction or warehouse automation, where a robot needs to interact with a structured object formation such as a stack. In this context, we define the problem of selecting suitable objects for grasping along with estimating an accurate 6DoF pose of these objects. To address this problem, we propose a camera-IMU based approach that prioritizes unobstructed objects on the higher layers of stacks and introduce a dataset for benchmarking and evaluation, along with a suitable evaluation metric that combines object selection with pose accuracy. Experimental results show that although our method can perform quite well, this is a challenging problem if an error-free solution is needed. Finally, we show results from the deployment of our method for a brick-picking application in a construction scenario.

Keywords: Perception for Grasping and Manipulation, Multifingered Hands, AI-Based Methods
Abstract: Sequentially grasping multiple objects with multi-fingered hands is common in daily life, where humans can fully leverage the dexterity of their hands to enclose multiple objects. However, the diversity of object geometries and the complex contact interactions required for high-DOF hands to grasp one object while enclosing another make sequential multi-object grasping challenging for robots. In this paper, we propose SeqMultiGrasp, a system for sequentially grasping objects with a four-fingered Allegro Hand. We focus on sequentially grasping two objects, ensuring that the hand fully encloses one object before lifting it, and then grasps the second object without dropping the first. Our system first synthesizes single-object grasp candidates, where each grasp is constrained to use only a subset of the hand’s links. These grasps are then validated in a physics simulator to ensure stability and feasibility. Next, we merge the validated single-object grasp poses to construct multi-object grasp configurations. For deployment, we train a diffusion model conditioned on point clouds to propose grasp poses, followed by a heuristic-based execution strategy for real-world grasping. We test our system using 8*8 object combinations in simulation and 6*3 object combinations in real. Our diffusion based grasp model obtains an average success rate of 65.8% over 1,600 simulation trials and 56.7% over 90 real-world trials, suggesting that it is a promising approach for sequential multi-object grasping with multi-fingered hands. Supplementary material is available on our project website: https://hesic73.github.io/SeqMultiGrasp.

Keywords: Machine Learning for Robot Control, Reinforcement Learning, Visual Servoing
Abstract: In this paper, we present DL-Clip, an innovative online learning approach for nonlinear stabilizing control that operates without prior knowledge of system dynamics or reward signals, while significantly improving training efficiency. DL-Clip introduces a novel integration of stabilizing control with efficient Reinforcement Learning (RL) training mechanisms. The algorithm uses Lyapunov functions to ensure system stability and employs clipping operations to optimize policy updates, achieving faster convergence. We evaluate the effectiveness of DL-Clip through experiments, including simulations of the inverted pendulum and the Image-Based Visual Servoing (IBVS) for multicopter position stabilization. In addition, we validate the approach through a real flight experiment based on the IBVS problem, demonstrating its practical applicability.

Keywords: Machine Learning for Robot Control, AI-Based Methods, Deep Learning in Grasping and Manipulation
Abstract: Robotic soil manipulation is essential for automated farming, particularly in excavation and levelling tasks. However, the nonlinear dynamics of granular materials challenge traditional control methods, limiting stability and efficiency. We propose Celebi, a causality-enhanced optimisation method that integrates differentiable physics simulation with adaptive step-size adjustments based on causal inference. To enable gradient-based optimisation, we construct a differentiable simulation environment for granular material interactions. We further define skill parameters with a differentiable mapping to end-effector motions, facilitating efficient trajectory optimisation. By modelling causal effects between task-relevant features extracted from point cloud observations and skill parameters, Celebi selectively adjusts update step sizes to enhance optimisation stability and convergence efficiency. Experiments in both simulated and real-world environments validate Celebi’s effectiveness, demonstrating robust and reliable performance in robotic excavation and levelling tasks.

Keywords: Machine Learning for Robot Control, AI-Based Methods, Reinforcement Learning
Abstract: Sim-to-real transfer remains a significant challenge in robotics due to the discrepancies between simulated and real-world dynamics. Traditional methods like Domain Randomization often fail to capture fine-grained dynamics, limiting their effectiveness for precise control tasks. In this work, we propose a novel approach that dynamically adjusts simulation environment parameters online using in-context learning. By leveraging past interaction histories as context, our method adapts the simulation environment dynamics to real-world dynamics without requiring gradient updates, resulting in faster and more accurate alignment between simulated and real-world performance. We validate our approach across two tasks: object scooping and table air hockey. In the sim-to-sim evaluations, our method significantly outperforms the baselines on environment parameter estimation by 80% and 42% in the object scooping and table air hockey setups, respectively. Furthermore, our method achieves at least 70% success rate in sim-to-real transfer on object scooping across three different objects. By incorporating historical interaction data, our approach delivers efficient and smooth system identification, advancing the deployment of robots in dynamic real-world scenarios. Demos are available on our project page: https://sim2real-capture.github.io/

Keywords: Machine Learning for Robot Control, Compliance and Impedance Control, Learning from Demonstration
Abstract: In this paper, we present a controller that combines motion generation and control in one loop, to endow robots with reactivity and safety. In particular, we propose a control approach that enables to follow the motion plan of a first order Dynamical System (DS) with a variable stiffness profile, in a closed loop configuration where the controller is always aware of the current robot state. This allows the robot to follow a desired path with an interactive behavior dictated by the desired stiffness. We also present two solutions to enable a robot to follow the desired velocity profile, in a manner similar to trajectory tracking controllers, while maintaining the closed-loop configuration. Additionally, we exploit the concept of energy tanks in order to guarantee the passivity during interactions with the environment, as well as the asymptotic stability in free motion, of our closed-loop system. The developed approach is evaluated extensively in simulation, as well as in real robot experiments, in terms of performance and safety both in free motion and during the execution of physical interaction tasks. Note to Practitioners—The approach presented in this work allows for safe and reactive robot motions, as well as the capacity to shape the robot’s physical behavior during interactions. This becomes crucial for performing contact tasks that might require adaptability or for interactions with humans as in shared control or collaborative tasks. Furthermore, the reactive properties of our controller make it adequate for robots that operate in proximity to humans or in dynamic environments where potential collisions are likely to happen.

Keywords: Machine Learning for Robot Control, Deep Learning in Grasping and Manipulation, Reinforcement Learning
Abstract: Learning robotic manipulation skills from vision is a promising approach for developing robotics applications that can generalize broadly to real-world scenarios. As such, many approaches to enable this vision have been explored with fruitful results. Particularly, object-centric representation methods have been shown to provide better inductive biases for skill learning, leading to improved performance and generalization. Nonetheless, we show that object-centric methods can struggle to learn simple manipulation skills in multi-object environments.

Thus, we propose DOCIR, an object-centric framework that introduces a disentangled representation for objects of interest, obstacles, and robot embodiment. We show that this approach leads to state-of-the-art performance for learning pick and place skills from visual inputs in multi-object environments and generalizes at test time to changing objects of interest and distractors in the scene. Furthermore, we show its efficacy both in simulation and zero-shot transfer to the real world.

Keywords: Machine Learning for Robot Control, Evolutionary Robotics, Reinforcement Learning
Abstract: Deep reinforcement learning (DRL) typically requires reinitializing training for new tasks, limiting its generalization due to isolated knowledge transfer. Meta-reinforcement learning (Meta-RL) addresses this by enabling rapid adaptation through prior task experiences, yet existing gradient-based methods like MAML suffer from poor out-of-distribution performance due to overfitting narrow task distributions. To overcome this limitation, we propose Evolving Gradient Regularization MAML (ER-MAML). By integrating evolving gradient regularization into the MAML framework, ER-MAML optimizes meta-gradients while constraining adaptation directions via a regularization policy. This dual mechanism prevents overparameterization and enhances robustness across diverse task distributions. Experiments demonstrate ER-MAML outperforms state-of-the-art baselines by 14.6% in out-of-distribution success rates. It also achieves strong online adaptation performance in the MetaWorld benchmark. These results validate ER-MAML's effectiveness in improving meta-RL generalization under distribution shifts.

Keywords: Machine Learning for Robot Control, Legged Robots, AI-Based Methods
Abstract: Diffusion models demonstrate superior performance in capturing complex distributions from large-scale datasets, providing a promising solution for quadrupedal locomotion control. However, the robustness of the diffusion planner is inherently dependent on the diversity of the pre-collected datasets. To mitigate this issue, we propose a two-stage learning framework to enhance the capability of the diffusion planner under limited dataset (reward-agnostic). Through the offline stage, the diffusion planner learns the joint distribution of state-action sequences from expert datasets without using reward labels. Subsequently, we perform the online interaction in the simulation environment based on the trained offline planner, which significantly diversified the original behavior and thus improves the robustness. Specifically, we propose a novel weak preference labeling method without the ground-truth reward or human preferences. The proposed method exhibits superior stability and velocity tracking accuracy in pacing, trotting, and bounding gait under different speeds and can perform a zero-shot transfer to the real Unitree Go1 robots.

Keywords: Machine Learning for Robot Control, Legged Robots, Incremental Learning
Abstract: One of the largest challenges in the deployment of legged robots in the real world is deriving effective general gaits. In this paper, we present BeeTLe, which is a framework that enables terrain aware locomotion without the need for dedicated terrain sensors. BeeTLe is realised as a multi-expert policy Reinforcement Learning (RL) algorithm. This enables multiple gaits, applicable to different surface types, to be stored and shared in a single policy. Sensor free terrain awareness is incorporated using a Recurrent Neural Network (RNN) to infer surface type purely from actuator positions over time. The RNN achieves an accuracy of 94% in terrain identification out of 8 possible options. We demonstrate that BeeTLe achieves a greater performance than the baselines across a series of challenges including: the traversal of a flat plane, a tilted plane, a sequence of tilted planes and geometry modelling a natural hilly terrain. This is despite not seeing the sequence of tilted planes and the natural hilly terrain during training.

Keywords: Dual Arm Manipulation, Human-Robot Collaboration, Task and Motion Planning
Abstract: In robotic operation, asymmetric tasks requiring dual-arm cooperation are the highly challenging research direction. Autonomous operation generally has a low success rate or poor generalization because of its excessive dependence on the accuracy of sub-goals from asymmetric tasks. Although teleoperation can significantly improve the performances above, during operation, the operators are prone to neglect crucial intermediate sub-goals that are conducive to fine-grained dual-arm cooperation. Therefore, we propose a dual-arm shared control framework which firstly introduces the Sub-goal Generation module to sufficiently concentrate on the intermediate states, thus improving the ability of fine-grained dual-arm cooperation and reducing the adjustment quantity during robotic asymmetric task operation. Also, we integrate the Trajectory Prediction module that computes the future trajectory based on the historical movement information to enhance the robot motion smoothness. Finally, through dynamic combination of Sub-goal Generation module, Trajectory Prediction module and operator movement in the shared control framework, we effectively decrease the sensitivity to the accuracy of sub-goals, thus significantly improving the success rate. In simulation, we conduct the comparative experiments with autonomous operation and teleoperation on four common asymmetric tasks to validate the advantages of our shared control framework. The effect of each element in our framework is verified by ablation study. Certainly, our shared control framework can also be applied in real-world scenario.

Keywords: Dual Arm Manipulation, Compliant Assembly, Compliance and Impedance Control
Abstract: While modeling multi-contact manipulation as a quasi-static mechanical process transitioning between different contact equilibria, we propose formulating it as a planning and optimization problem, explicitly evaluating (i) contact stability and (ii) robustness to sensor noise. Specifically, we conduct a comprehensive study on multi-manipulator control strategies, focusing on dual-arm execution in a planar peg-in-hole task and extending it to the Multi-Manipulator Multiple Peg-in-Hole (MMPiH) problem to explore increased task complexity. Our framework employs Dynamic Movement Primitives (DMPs) to parameterize desired trajectories and Black-Box Optimization (BBO) with a comprehensive cost function incorporating friction cone constraints, squeeze forces, and stability considerations. By integrating parallel scenario training, we enhance the robustness of the learned policies. To evaluate the friction cone cost in experiments, we test the optimal trajectories computed for various contact surfaces, i.e., with different coefficients of friction. The stability cost is analytical explained and tested its necessity in simulation. The robustness performance is quantified through variations of hole pose and chamfer size in simulation and experiment. Results demonstrate that our approach achieves consistently high success rates in both the single peg-in-hole and multiple peg-in-hole tasks, confirming its effectiveness and generalizability. The video can be found at https://youtu.be/IU0pdnSd4tE.

Keywords: Dual Arm Manipulation, Manipulation Planning, Learning from Demonstration
Abstract: Robotic manipulation can involves multiple manipulators to complete a task. In those cases, the complexity of performing the task in a coordinated manner increases, requiring coordinated planning while avoiding collisions between robots and environmental elements. For these challenges, we propose a robotic arm control algorithm based on Learning from Demonstration to independently learn the tasks of each arm, followed by a graph-based communication method using Gaussian Belief Propagation. Our method enables the resolution of decoupled dual-arm tasks learned independently without requiring coordinated planning. The algorithm generates smooth, collision-free solutions between arms and environmental obstacles while ensuring efficient movements without the need for constant replanning. Its efficiency has been validated through experiments and comparisons against another multi-robot control method in simulation using PyBullet with two opposing IIWA robots, as well as a mobile robot with two UR3 arms, which has also been used for real-world testing.

Keywords: Dual Arm Manipulation, Grippers and Other End-Effectors, Grasping
Abstract: この研究では、次の方法を提案し、提案し、実装します。 積み重ねられた生地の最上層のみを デュアルアームロボットをターゲット位置に配置します。ザ 提案手法は、3本指のロボットハンドを採用 1本の粘着性フィンガーと2本の非粘着性フィンガーから構成 指をつかんで、最上層の生地層だけをつかみます。後 把握すると、特徴マッチングが実行され、 回転角度と移動ベクトルを整列させるために必要な ターゲット位置のファブリック。その後、生地が移動します 指で押されながら。提案された手法は、 ロボットに実装し、実験で検証 構造の異なる5種類の生地を使用し、 表面の材料、質量、厚さ、それによって その有効性を確認します。

Keywords: Dual Arm Manipulation, Human-Robot Collaboration, Motion and Path Planning
Abstract: Robotic manipulation in dynamic environments often requires seamless transitions between different grasp types to maintain stability and efficiency. However, achieving smooth and adaptive grasp transitions remains a challenge, particularly when dealing with external forces and complex motion constraints. Existing grasp transition strategies often fail to account for varying external forces and do not optimize motion performance effectively. In this work, we propose an Imitation-Guided Bimanual Planning Framework that integrates efficient grasp transition strategies and motion performance optimization to enhance stability and dexterity in robotic manipulation. Our approach introduces Strategies for Sampling Stable Intersections in Grasp Manifolds for seamless transitions between uni-manual and bi-manual grasps, reducing computational costs and regrasping inefficiencies. Additionally, a Hierarchical Dual-Stage Motion Architecture combines an Imitation Learning-based Global Path Generator with a Quadratic Programming-driven Local Planner to ensure real-time motion feasibility, obstacle avoidance, and superior manipulability. The proposed method is evaluated through a series of force-intensive tasks, demonstrating significant improvements in grasp transition efficiency and motion performance.

Keywords: Dual Arm Manipulation, Motion and Path Planning
Abstract: Constrained motion planning is a common but challenging problem in robotic manipulation. In recent years, data-driven constrained motion planning algorithms have shown impressive planning speed and success rate. Among them, the latent motion method based on manifold approximation is the most efficient planning algorithm. Due to errors in manifold approximation and the difficulty in accurately identifying collision conflicts within the latent space, time-consuming path validity checks and path replanning are required. In this paper, we propose a method that trains a neural network to predict the minimum distance between the robot and obstacles using latent vectors as inputs. The learned distance gradient is then used to calculate the direction of movement in the latent space to move the robot away from obstacles. Based on this, a local path optimization algorithm in the latent space is proposed, and it is integrated with the path validity checking process to reduce the time of replanning. The proposed method is compared with state-of-the-art algorithms in multiple planning scenarios, demonstrating the fastest planning speed.

Keywords: Dual Arm Manipulation, Robust/Adaptive Control, Motion Control
Abstract: Dual-arm robots possess exceptional collaborative capabilities and versatility, demonstrating broad application prospects across various fields. As a significant research area for dual-arm robots, the requirements for coordinated motion control are gradually increasing. In practical applications, robots inevitably encounter noise interference, which can lead to suboptimal performance in coordinated motion control. In this letter, cooperative motion control of dual-arm robots in the presence of harmonic noise is investigated. On the basis of the relative Jacobian method, an adaptive noise rejection strategy is proposed for cooperative motion control of dual-arm robots perturbed by harmonic noise. Such a strategy incorporates a compensator, which can simulate and suppress interference from harmonic noise. Theoretical analysis indicates that the Cartesian error generated by the proposed strategy exhibits convergence. Simulation and experiment results under a dual-arm system consisting of two Panda robot manipulators further verify the noise resistance and applicability of the proposed strategy with the existence of harmonic noise.

Keywords: Dual Arm Manipulation, Deep Learning in Grasping and Manipulation, Perception for Grasping and Manipulation
Abstract: Dual-arm manipulation of fabrics is important and complex in the field of embodied intelligence for robots. Previous work has mostly focused on extracting the geometric features of fabrics, but has neglected the knowledge and experience constraints on grasp postures like humans, which greatly affects the success rate of operations. Therefore, this paper proposes a dual-arm manipulation method for fabrics based on imitation learning, including the study of an action encoding learning algorithm based on dynamic motion primitives (DMPs), the establishment of a dual-arm motion primitive library to provide prior knowledge for operation planning, the creation of a segment-based dual-arm grasp method based on prior knowledge, and the realization of extracting the optimal grasp posture from the implicit geometric features of messy fabrics. Experiments show that the success rate of this method in the fabric unfolding task is 85.0%, with an average of 5.0 operations, and the success rate in the fabric folding task is 96.7%, both of which are significantly better than the two baseline methods, SpeedFolding and FlingBot.

Keywords: Force and Tactile Sensing, Sensor Fusion, Embedded Systems for Robotic and Automation
Abstract: Visuotactile sensors provide high-resolution tactile information but are incapable of perceiving the material features of objects. We present UltraTac, an integrated sensor that combines visuotactile imaging with ultrasound sensing through a coaxial optoacoustic architecture. The design shares structural components and achieves consistent sensing regions for both modalities. Additionally, we incorporate acoustic matching into the traditional visuotactile sensor structure, enabling the integration of the ultrasound sensing modality without compromising visuotactile performance. Through tactile feedback, we can dynamically adjust the operating state of the ultrasound module to achieve more flexible functional coordination. Systematic experiments demonstrate three key capabilities: proximity sensing in the 3–8 cm range (R² = 0.99), material classification (average accuracy: 99.20%), and texture-material dual-mode object recognition achieves 92.11% accuracy on a 15-class task. Finally, we integrate the sensor into a robotic manipulation system to concurrently detect container surface patterns and internal content, which verifies its promising potential for advanced human-machine interaction and precise robotic manipulation.

Keywords: Force and Tactile Sensing, Visual Servoing, Force Control
Abstract: This paper proposes a novel force feedback system based on visual processing and Moir´e patterns. The system uses a force sensor with a simple and efficient structure to eliminate the need for cables or other electronic components. A brass flexure plate was employed as the primary elastic element, leveraging the Moir´e fringe principle for force measurements. High-speed cameras are used to capture real-time images that are then processed using advanced algorithms to accurately extract force data. Force feedback experiments were conducted to evaluate the performance of the proposed system, and the results were compared with those taken from conventional force sensors. The experimental results indicated that the proposed device consistently delivered precise target force outputs with outcomes that closely matched those obtained using traditional sensors. In addition, the system demonstrated robust performance in high-frequency measurements at 500 Hz, achieving an average force- feedback convergence time of approximately 0.1s.

Keywords: Force and Tactile Sensing, Grasping
Abstract: Vision-based tactile sensing, an economical and widely utilized methodology, has the potential to offer crucial contact geometry information for localizing objectives even in cases of visual occlusion. However, this kind of fingertip sensor is limited. When a person picks up a relatively small object placed on a flat surface with two fingers, they may not only use the pads of their fingers depending on the size of the object but also use their fingernails for small or thin objects. Fingers with nail structures have been shown to be effective in picking up objects like this in robot hands as well. Moreover, in actual work, accidental contact between sensors and surrounding objects such as tables often occurs. Sensors with fingernails can avoid this situation in advance by having the fingernails touch the object before the fingertip touches the object. In this work, we present the NailTact, which can detect the force applied to both the fingertip part and the nail part from the same camera image using a single camera. Using the prototype robot finger, we will verify the sensor response characteristics to the load on the nail and the sensor response when grasping an object with the nail and the situation when the finger makes contact with a table. We also present a simple model that illustrates the relationship between the force applied to the nail and the movement of the marker. In the card-grasping experiment, we not only successfully grasped a very thin object but also measured the grasping force.

Keywords: Force and Tactile Sensing, Dexterous Manipulation, Imitation Learning
Abstract: Tactile perception is essential for real-world manipulation tasks, yet the high cost and fragility of tactile sensors can limit their practicality. In this work, we explore BeadSight (a low-cost, open-source tactile sensor) alongside a tactile pre-training approach, an alternative method to precise, pre-calibrated sensors. By pre-training with the tactile sensor and then disabling it during downstream tasks, we aim to enhance robustness and reduce costs in manipulation systems. We investigate whether tactile pre-training, even with a low-fidelity sensor like BeadSight, can improve the performance of an imitation learning agent on complex manipulation tasks. Through visuo-tactile pre-training on both similar and dissimilar tasks, we analyze its impact on a longer-horizon downstream task. Our experiments show that visuo-tactile pre-training improved performance on a USB cable plugging task by up to 65% with vision-only inference. Additionally, on a longer-horizon drawer pick-and-place task, pre-training — whether on a similar, dissimilar, or identical task — consistently improved performance, highlighting the potential for a large-scale visuo-tactile pre-trained encoder.

Keywords: Force and Tactile Sensing, Perception for Grasping and Manipulation, Lensless Imaging, Computer Vision for Other Robotic Applications
Abstract: Vision-based tactile sensors have drawn increasing interest in the robotics community. However, traditional lensbased designs impose minimum thickness constraints on these sensors, limiting their applicability in space-restricted settings. In this paper, we propose ThinTact, a novel lensless visionbased tactile sensor with a sensing field of over 200 mm2 and a thickness of less than 10 mm. ThinTact utilizes the mask-based lensless imaging technique to map the contact information to CMOS signals. To ensure real-time tactile sensing, we propose a real-time lensless reconstruction algorithm that leverages a frequency-spatial-domain joint filter based on discrete cosine transform (DCT). This algorithm achieves significantly faster computation than existing optimization-based methods. Additionally, to improve the sensing quality, we develop a mask optimization method based on the generic algorithm and the corresponding system matrix calibration algorithm. We evaluate the performance of our proposed lensless reconstruction and tactile sensing through qualitative and quantitative experiments. Furthermore, we demonstrate ThinTact’s practical applicability in diverse applic