Keywords: Big Data in Robotics and Automation, Data Sets for Robot Learning, Machine Learning for Robot Control
Abstract: The current paradigm for motion planning generates solutions from scratch for every new problem, which consumes significant amounts of time and computational resources. For complex, cluttered scenes, motion planning approaches can often take minutes to produce a solution, while humans are able to accurately and safely reach any goal in seconds by leveraging their prior experience. We seek to do the same by applying data-driven learning at scale to the problem of motion planning. Our approach builds a large number of complex scenes in simulation, collects expert data from a motion planner, then distills it into a reactive neural policy. We then combine this with lightweight optimization to obtain a safe path for real world deployment. We perform a thorough evaluation of our method on 64 motion planning tasks across four diverse environments with randomized poses, scenes and obstacles, in the real world, demonstrating an improvement of 23%, 17% and 79% motion planning success rate over state of the art sampling, optimization and learning based planning methods. All code, models and datasets will be released on acceptance. Video results available at mihdalal.github.io/neuralmotionplanner.

Keywords: Mobile Manipulation, Autonomous Vehicle Navigation, Collision Avoidance
Abstract: Interactive navigation is crucial in scenarios where proactively interacting with objects can yield shorter paths, thus significantly improving traversal efficiency. While existing methods primarily rely on body-velocity-based pushing for relocation of large obstacles (which could be comparable to the size of a robot), they prove ineffective in narrow or constrained spaces where the robot's dimensions restrict its manipulation capabilities, thus compromising clearance and overall navigation performance. This paper introduces a novel interactive navigation framework for legged manipulators, featuring an active arm-pushing mechanism for effective obstacle clearance. The framework enables the robot to dynamically detect and reposition movable obstacles in space-constrained environments, leading to more feasible and efficient navigation paths. At the core of this framework, we develop a reinforcement learning-based arm-pushing controller with a two-stage reward strategy for large-object manipulation. Specifically, this strategy progressively refines the pushing behavior, first guiding the manipulator to a pre-pushing zone for kinematically feasible contact configuration, and then maintaining end-effector positioning at appropriate contact points for stable object displacement while preventing toppling. Through simulations and ablation studies, we validate the robustness of the arm-pushing controller, showing that the two-stage reward strategy improves policy convergence and long-term performance in large-object manipulation. Real-world experiments further underscore its effectiveness in confined environments with movable obstacles, achieving shorter paths and reduced traversal time. The open-source project can be found at url{https://github.com/Zhihaibi/Interactive-Navigation-for -legged-manipulator.git}.

Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Visual Servoing
Abstract: Creating an intelligent surgical environment requires not only advanced robotic systems but also optimized microscopic imaging. However, autofocus remains a fundamental challenge, with current methods suffering from slow iterative processes or directional ambiguity, which compromises real-time performance. This paper presents an implicit disparity-blur alignment approach for robotic microsurgical autofocus, integrating stereo geometry’s monotonic depth cues with defocus characteristics for rapid convergence. A novel physics-guided dual-stream network is developed to encode implicit depth representations through hierarchical cross-pathway feature fusion, enabling reliable focus prediction without explicit stereo matching in blur-degraded regions. An ROI-aware attention module is proposed to dynamically optimize focus-critical regions, coupled with learnable physics-guided kernel learning for precise Z-offset estimation. The approach achieves a top directional accuracy of 94.85% and a single-pass focus error of 0.20 mm with an inference time of 53 ms on a surgical dataset, which outperforms state-of-the-art methods in reducing iteration count by 22.8% and inference time by 51.8%. An intelligent robotic microscope prototype is developed, with validation through ex vivo tests demonstrating its ability to enable fast and precise multi-region focusing for microsurgeries.

Keywords: Additive Manufacturing, Intelligent and Flexible Manufacturing
Abstract: This paper presents a rapid 3D printing framework that enhances the efficiency of commercially available layered manufacturing-based 3D printers. Unlike traditional methods that simplify printing regions to a single point or rely on predefined entry and exit points, our approach utilizes an improved Traveling Salesman Problem (TSP) algorithm to autonomously generate an optimized, cyclic printing path, while automatically assigning entry and exit points for each region. This minimizes non-printing paths and improves efficiency. Additionally, we propose a principal axis calculation method for irregular shapes, aligning better with geometric orientation. This optimization enhances infill uniformity and surface smoothness. Simulations and experimental results demonstrate that the proposed framework improves printing efficiency while maintaining print quality, with promising applicability to large-scale and complex 3D printing models.

Keywords: Medical Robots and Systems, Sensor-based Control, Underactuated Robots
Abstract: Magnetic manipulation has been adopted as a method of actuation in both wireless capsule endoscopy and softtethered endoscopy, with the goal of improving gastrointestinal procedures. However, by nature of magnetic manipulation, these endoscopes are typically limited to a maximum of five degrees of freedom (DoF). With the need to introduce additional contactbased sensing modalities for subsurface investigation into these systems as well as to improve overall dexterity, it is both practically and clinically beneficial to recover the lost DoF i.e. the roll around the main axis. This paper presents a method of achieving the magnetic manipulation of an underactuated device by leveraging developable surfaces, specifically, the oloid shape. The design of a clinically relevant magnetic endoscope with all its ancillary elements, as well as contact sensors, is proposed and demonstrated in vivo. The contact sensor data from the in vivo experiments show that for sweeping motions over 100◦ of roll, contact between the endoscope’s sensor region and the colon wall can be maintained for 74% of the motion.

Keywords: Intelligent Transportation Systems, Deep Learning Methods, AI-Based Methods
Abstract: Pedestrian trajectory prediction ensures safe navigation in autonomous driving and intelligent robots. Existing methods, including physical models, machine learning, and deep learning, have shown promising results but still face challenges in handling dynamic environments, social interactions, and high-dimensional data. In this paper, we propose a novel PhysGCN-DL model within the itransformer framework to address these challenges. Our model incorporates physically-inspired dynamic interaction modeling by representing physical interactions between pedestrians as edge weights in graph convolution. This approach captures the heterogeneity of pedestrian movement and improves the interpretability of social interactions. Additionally, we introduce a new loss function that balances diversity and accuracy in trajectory prediction, ensuring robustness in dense and sparse environments. Experimental results demonstrate the superiority of our model in predicting diverse and accurate pedestrian trajectories compared to existing methods.

Keywords: Rehabilitation Robotics, Prosthetics and Exoskeletons, Wearable Robotics
Abstract: Self-selected walking speed is a key outcome for exercise-based rehabilitation programs following lower-extremity trauma. This work introduces a novel reinforcement learning-based assist-as-needed (RL-AAN) controller for ankle exoskeletons, aimed at gait speed training. Built on an actor–critic architecture, the RL-AAN controller integrates a control objective that balances the trade-off between expected stride velocity (SV) errors and exoskeleton assistance. This approach allows the exoskeleton to progressively reduce ankle plantar- and dorsiflexion (PDF) assistance as the user’s performance improves, promoting active participation. The desired assistive torque is computed as the product of the actor output and the wearer’s biomechanical ankle PDF moment, estimated by a subject-agnostic model, thereby ensuring personalized and biomechanically relevant assistance. In a proof-of-concept study with healthy individuals walking on a self-paced treadmill with ankle weights, the RL-AAN controller outperformed a conventional Fixed-K controller—achieving greater immediate speed increases during assisted walking (14.2% vs. 10.0% relative to unassisted perturbed walking) and inducing short-term gait speed adaptation post-training, not observed with the conventional controller. These findings highlight the potential of RL-AAN control for subject-tailored gait training, with promising clinical implications for exercise-based rehabilitation in individuals with neurological or musculoskeletal gait impairments.

Keywords: Field Robots, Vision-Based Navigation, Robotics and Automation in Agriculture and Forestry
Abstract: Natural environments pose significant challenges for autonomous robot navigation, particularly due to their unstructured and ever-changing nature. Hiking trails, with their dynamic conditions influenced by weather, vegetation, and human traffic, represent one of these challenges. This work introduces a novel approach to autonomous hiking trail navigation that balances trail adherence with the flexibility to adapt to off-trail routes when necessary. The solution is a Traversability Analysis module that integrates semantic data from camera images with geometric information from LiDAR to create a comprehensive understanding of the surrounding terrain. A planner uses this traversability map to navigate safely, adhering to trails while allowing off-trail movement when necessary to avoid on-trail hazards or for safe off-trail shortcuts. The method is evaluated through simulation to determine the balance between semantic and geometric information in traversability estimation. These simulations tested various weights to assess their impact on navigation performance across different trail scenarios. Weights were then validated through autonomous field tests at the West Virginia University Core Arboretum, demonstrating the method's effectiveness in a real-world environment.

Keywords: Mobile Manipulation, Bimanual Manipulation, Deep Learning in Grasping and Manipulation
Abstract: The ability to use random objects as tools in a generalizable manner is a missing piece in robots' intelligence today to boost their versatility and problem-solving capabilities. State-of-the-art robotic tool usage methods focused on procedurally generating or crowd-sourcing datasets of tools for a task to learn how to grasp and manipulate them for that task. However, these methods assume that only one object is provided and that it is possible, with the correct grasp, to perform the task; they are not capable of identifying, grasping, and using the best object for a task when many are available, especially when the optimal tool is absent. In this work, we propose GeT-USE, a two-step procedure that learns to perform real-robot generalized tool usage by learning first to extend the robot's embodiment in simulation and then transferring the learned strategies to real-robot visuomotor policies. Our key insight is that by exploring a robot's embodiment extensions (i.e., building new end-effectors) in simulation, the robot can identify the general tool geometries most beneficial for a task. This learned geometric knowledge can then be distilled to perform generalized tool usage tasks by selecting and using the best available real-world object as tool. On a real robot with 22 degrees of freedom (DOFs), GeT-USE outperforms state-of-the-art methods by 30-60% success rates across three vision-based bimanual mobile manipulation tool-usage tasks under unseen real-world objects, environments, and lighting conditions.

Keywords: Mobile Manipulation, Constrained Motion Planning, Collision Avoidance
Abstract: Mobile manipulators operating in dynamic environments shared with humans and robots must adapt in real time to environmental changes to complete their tasks effectively. While global planning methods are effective at considering the full task scope, they lack the computational efficiency required for reactive adaptation. In contrast, local planning approaches can be executed online but are limited by their inability to account for the full task’s duration. To tackle this, we propose Globally-Guided Geometric Fabrics (G3F), a framework for real-time motion generation along the full task horizon, by interleaving an optimization-based planner with a fast reactive geometric motion planner, called Geometric Fabrics (GF). The approach adapts the path and explores a multitude of acceptable target poses, while accounting for collision avoidance and the robot’s physical constraints. This results in a real-time adaptive framework considering whole-body motions, where a robot operates in close proximity to other robots and humans. We validate our approach through various simulations and real-world experiments on mobile manipulators in multi-agent settings, achieving improved success rates compared to vanilla GF, Prioritized Rollout Fabrics and Model Predictive Control.

Keywords: Mobile Manipulation, Deep Learning in Grasping and Manipulation, Task and Motion Planning
Abstract: In this work, we present PlaceNet: a deep learning framework for mobile manipulator base placement which provides solutions to the shortcomings common in the state-of-the-art. Our method addresses the lack of obstacle awareness of reachability methods and the limited generalization of learning methods. Using only the raw pointcloud and task pose data as input, PlaceNet learns the concepts of reachability and obstacle occlusions in an environment-independent manner, enabling its use in situations outside its training experiences. Tests comparing PlaceNet to inverse reachability and heuristic methods demonstrated state-of-the-art performance in both the In-Distribution and Out-Of-Distribution test sets, achieving as high as 98% success rate for problems with many solutions, and an 82% success rate overall. PlaceNet can be trained on grounded pointcloud data from any source without the need for dynamic simulation, marking it as an accessible alternative to similar frameworks which require expensive, high-performance GPUs for running simultaneous simulation and training or which depend on labor intensive data collection. PlaceNet is lightweight during deployment and can easily run with low latency on affordable hardware, including laptop GPUs and the NVIDIA Jetson line for embedded deployment.

Keywords: Mobile Manipulation, Deep Learning Methods, Visual Learning
Abstract: Long-horizon reasoning and task execution are crucial for complex mobile manipulation tasks in household environments. Existing benchmarks and methods primarily focus on single-room or single-object mobile manipulation scenarios, limiting the scope of long-horizon planning and scene-level understanding. To address this gap, we introduce a novel benchmark for long-horizon mobile manipulation in multi-room household environments. Our task requires agents to follow a sequence of language instructions, each directing the movement of specific objects across receptacles and rooms. In this task, we investigate the role of long-term memory by constructing a hierarchical scene graph that captures the relationships between objects, furniture, and rooms. This scene graph-based memory is dynamically updated as the agent explores the environment, which effectively aligns the scene information with the targets and environmental context specified in the language instructions. Additionally, we benchmark the proposed task in dynamic environments where objects can be relocated during task execution, simulating real-world scenarios. Our results demonstrate that the scene graph-based memory significantly improves the agent's performance in long-horizon mobile manipulation tasks. Moreover, dynamically updating the state of objects within the scene graph enables the agent to better adapt to dynamic conditions.

Keywords: Mobile Manipulation, Legged Robots, Reinforcement Learning
Abstract: Mobile manipulation in dynamic environments is challenging due to movable obstacles blocking the robot’s path. Traditional methods, which treat navigation and manipulation as separate tasks, often fail in such "manipulate-to-navigate" scenarios, as obstacles must be removed before navigation. In these cases, active interaction with the environment is required to clear obstacles while ensuring sufficient space for movement. To address the manipulate-to-navigate problem, we propose a reinforcement learning-based approach for learning manipulation actions that facilitate subsequent navigation. Our method combines manipulability priors to focus the robot on high manipulability body positions with affordance maps for selecting high-quality manipulation actions. By focusing on feasible and meaningful actions, our approach reduces unnecessary exploration and allows the robot to learn manipulation strategies more effectively.

We present two new manipulate-to-navigate simulation tasks called Reach and Door with the Boston Dynamics Spot robot. The first task tests whether the robot can select a good hand position in the target area such that the robot base can move effectively forward while keeping the end effector position fixed. The second task requires the robot to move a door aside in order to clear the navigation path. Both of these two tasks need first manipulation and then navigating the base forward. Results show that our method allows a robot to effectively interact with and traverse dynamic environments. Finally, we transfer the learned policy to a real Boston Dynamics Spot robot, which successfully performs the Reach task.

Keywords: Mobile Manipulation, Long term Interaction, Semantic Scene Understanding
Abstract: Enabling mobile robots to perform long-term tasks in dynamic real-world

environments is a formidable challenge, especially when the environment

changes frequently due to human-robot interactions or the robot's own actions. Traditional methods typically assume static scenes, which limits their applicability in the continuously changing real world. To overcome these limitations, we present DovSG, a novel mobile manipulation framework that leverages dynamic open-vocabulary 3D scene graphs and a language-guided task planning module for long-term task execution. DovSG takes RGB-D sequences as input and utilizes vision-language models (VLMs) for object detection to obtain high-level

object semantic features. Based on the segmented objects, a structured 3D scene graph is generated for low-level spatial relationships. Furthermore, an efficient mechanism for locally updating the scene graph, allows the robot to adjust parts of the graph dynamically during

interactions without the need for full scene reconstruction. This mechanism is particularly valuable in dynamic environments, enabling the robot to continually adapt to scene changes and effectively support

the execution of long-term tasks. We validated our system in real-world environments with varying degrees

of manual modifications, demonstrating its effectiveness and superior performance in long-term tasks. Our project page is available at https://bjhyzj.github.io/dovsg-web.

Keywords: Mobile Manipulation, Manipulation Planning, Learning from Demonstration
Abstract: 通过缓冲过程捕捉飞行物体是一个 人类通常执行的技能，但它仍然是一种 对机器人来说是重大挑战。在本文中，我们介绍 一个结合优化和学习的框架，以 在移动机械手 （CCMM） 上实现合规捕获。 首先，我们提出了一个用于移动的高级捕获规划器 计算最佳捕获点的机械手 （MM） 和关节配置。接下来，预捕（PRC） 规划器确保机器人到达目标关节 尽快配置。学习合规 捕捉策略，我们提出了一个利用 LSTM 在捕获时间依赖关系方面的优势 以及空间上下文的位置编码 （P-LSTM）。这 网络旨在有效学习合规 来自人类演示的策略。在此之后， 捕获后 （POC） 规划器跟踪合规序列 P-LSTM 输出，同时避免潜在的碰撞 由于人类和机器人之间的结构差异。我们 通过模拟和 现实世界的接球场景，取得成功 模拟率为98.70%，真实测试率为92.59%， 冲击扭矩降低 28.7%。开源 将发布代码供社区参考。

Keywords: Mobile Manipulation, Reinforcement Learning
Abstract: Mobile manipulators require coordinated control between navigation and manipulation to accomplish tasks. Typically, coordinated mobile manipulation behaviors have base navigation to approach the goal followed by arm manipulation to reach the desired pose. Selecting the embodiment between the base and arm can be determined based on reachability. Previous methods evaluate reachability by computing inverse kinematics and activate arm motions once solutions are identified. In this study, we introduce a new approach called predictive reachability that decides reachability based on predicted arm motions. Our model utilizes a hierarchical policy framework built upon a world model. The world model allows the prediction of future trajectories and the evaluation of reachability. The hierarchical policy selects the embodiment based on the predicted reachability and plans accordingly. Unlike methods that require prior knowledge about robots and environments for inverse kinematics, our method only relies on image-based observations. We evaluate our approach through basic reaching tasks across various environments. The results demonstrate that our method outperforms previous model-based approaches in both sample efficiency and performance, while enabling more reasonable embodiment selection based on predictive reachability.

Keywords: In-Hand Manipulation, Imitation Learning, Multifingered Hands
Abstract: We present a framework for learning dexterous in-hand manipulation with multifingered hands using visuomotor diffusion policies. Our system enables complex in-hand manipulation tasks, such as unscrewing a bottle lid with one hand, by leveraging a fast and responsive teleoperation setup for the four-fingered Allegro Hand. We collect high-quality expert demonstrations using an augmented reality (AR) interface that tracks hand movements and applies inverse kinematics and motion retargeting for precise control. The AR headset provides real-time visualization, while gesture controls streamline teleoperation. To enhance policy learning, we introduce a novel demonstration outlier removal approach based on HDBSCAN clustering and the Global-Local Outlier Score from Hierarchies (GLOSH) algorithm, effectively filtering out low-quality demonstrations that could degrade performance. We evaluate our approach extensively in real-world settings and provide all experimental videos on the project website: https://dex-manip.github.io/

Keywords: In-Hand Manipulation, Multifingered Hands
Abstract: In-hand manipulation is a crucial ability for reorienting and repositioning objects within grasps. The main challenges in this are not only the complexity of the computational models, but also the risks of grasp instability caused by active finger motions, such as rolling, sliding, breaking, and remaking contacts. This paper presents the development of the Roller Ring (RR), a modular robotic attachment with active surfaces that is wearable by both robot and human hands to manipulate without lifting a finger. By installing the angled RRs on hands, such that their spatial motions are not colinear, we derive a general differential motion model for manipulating objects. Our motion model shows that complete in-hand manipulation skill sets can be provided by as few as only 2 RRs through non-holonomic object motions, while more RRs can enable enhanced manipulation dexterity with fewer motion constraints. Through extensive experiments, we test the RRs on both a robot hand and a human hand to evaluate their manipulation capabilities. We show that the RRs can be employed to manipulate arbitrary object shapes to provide dexterous in-hand manipulation.

Keywords: In-Hand Manipulation, Modeling, Control, and Learning for Soft Robots, Reinforcement Learning
Abstract: In-hand manipulation tasks using hand-like robotic grippers offer a promising approach to accomplish various tasks in a human-centered environment. Due to the inherent safety of soft robots, in-hand manipulations performed by soft robots provide great opportunities for future human-robot collaboration, which is the scope of this paper. By modeling a new and innovative soft-robotic gripper known as the Anthropomorphic Soft Gripper and synthesis of an open-loop controller with deep reinforcement learning, it is demonstrated how the movement of objects by in-hand manipulations can be accomplished. Moreover, this work explores the application of deep reinforcement learning methods without the employment for domain randomization. As noted by Bhatt et al. the inherent soft properties of soft robotic grippers enable remarkably robust in-hand manipulation in open-loop control, giving the impetus for the approach that is being followed in this work. Motion sequences generated in simulation are successfully transferred to the real anthropomorphic soft gripper and validated in experiments.

Keywords: In-Hand Manipulation, Grippers and Other End-Effectors, Dexterous Manipulation
Abstract: Achieving controlled sliding of objects on finger surfaces is a significant challenge for robots, substantially constraining their ability to perform complex in-hand manipulation tasks. In this work, we investigate the role of surface vibration in modulating the effective friction at the object-finger contact locations to facilitate controlled sliding. We demonstrate that friction at contact points can be reduced by applying targeted vibrations at specific locations on a robotic finger, creating regions that are suitable for sliding. In this way, we create sticking/sliding regions on finger surfaces on demand and can easily switch between sliding and rolling contacts. To investigate this phenomenon, we embedded an array of vibration modules into robotic fingers. We first analyzed the velocity fields created by surface vibrations on a single finger. Then, we developed a method to select the appropriate activation states of the modules that achieve the desired velocity field at a given object location. Utilizing these fingers and the vibration selection method, we formed a two-finger robotic hand and demonstrated controlled sliding and rotation of a held object within the hand. To the best of our knowledge, this is the first work that utilizes vibration-induced friction modulation for in-hand manipulation that can achieve combinations of object sliding and rolling actions.

Keywords: In-Hand Manipulation, Dexterous Manipulation
Abstract: Traditional robot control relies on analytical methods that require precise system models, which are hard to apply in real-world settings and limit generalization to arbitrary tasks. However, systems like serial manipulators and passively adaptive hands feature inherently stable regions without control discontinuities like loss of contact or singularities. In these regions, approximate controllers focusing on the correct direction of motion enable successful coarse manipulation. When coupled with a rough estimation of the motion magnitude, precision manipulation is achieved. Leveraging this insight, we introduce a novel inverse Jacobian estimation method that independently estimates the primary motion direction and magnitude of the manipulator's actuators. Our method efficiently estimates the direct mapping from task to actuator space with no need for a priori system knowledge enabling the same framework to control both hands and arms without compromising task performance. We present a novel control method with no a priori knowledge for precision manipulation. Experiments on the Yale Model O hand, Yale Stewart Hand, and a UR5e arm demonstrate that the inverse Jacobians estimated via our approach enable real-time control with submillimeter precision in manipulation tasks. These results highlight that online self-ID data alone is sufficient for precise real-world manipulation.

Keywords: Robot Safety, AI-Enabled Robotics
Abstract: Ensuring safe interactions in human-centric environments requires robots to understand and adhere to constraints recognized by humans as "common sense" (e.g., "moving a cup of water above a laptop is unsafe as the water may spill" or "rotating a cup of water is unsafe as it can lead to pouring its content"). Recent advances in computer vision and machine learning have enabled robots to acquire a semantic understanding of and reason about their operating environments. While extensive literature on safe robot decision-making exists, semantic understanding is rarely integrated into these formulations. In this work, we propose a semantic safety filter framework to certify robot inputs with respect to semantically defined constraints (e.g., unsafe spatial relationships, behaviors, and poses) and geometrically defined constraints (e.g., environment-collision and self-collision constraints). In our proposed approach, given perception inputs, we build a semantic map of the 3D environment and leverage the contextual reasoning capabilities of large language models to infer semantically unsafe conditions. These semantically unsafe conditions are then mapped to safe actions through a control barrier certification formulation. We demonstrate the proposed semantic safety filter in teleoperated manipulation tasks and with learned diffusion policies applied in a real-world kitchen environment that further showcases its effectiveness in addressing practical semantic safety constraints. Together, these experiments highlight our approach's capability to integrate semantics into safety certification, enabling safe robot operation beyond traditional collision avoidance.

Keywords: Swarm Robotics, Multi-Robot Systems, Robot Safety
Abstract: Recent research has demonstrated that blockchain-enabled robot swarms—where robots coordinate using blockchain technology— can secure robot swarms by neutralizing malicious and malfunctioning robots. This security is achieved through blockchain technology’s consistency properties. However, prior work addressed malfunctions at the information level, that is, it studied how to neutralize robots that stored information in the blockchain that did not correspond to the real-world state (i.e., it studied the oracle problem). In contrast, this study focuses on inconsistencies at the blockchain protocol level. We analyze how network partitions, which may arise from robots’ local-only communication capabilities, malfunctioning hardware, or external attacks, can lead to inconsistent information in a robot swarm. In order to mitigate these disruptions, we propose a decentralized approach to detect partitions and a corresponding response. We study our approach in a swarm robotics simulator, where we demonstrate its effectiveness in reducing blockchain inconsistencies.

Keywords: Robot Safety, Collision Avoidance
Abstract: As the airspace becomes increasingly congested, decentralized conflict resolution methods for airplane encounters have become essential. While decentralized safety controllers can prevent dangerous midair collisions, they do not always ensure prompt conflict resolution. As a result, airplane progress may be blocked for extended periods in certain situations. To address this blocking phenomenon, this paper proposes integrating bio-inspired nonlinear opinion dynamics into the airplane safety control framework, thereby guaranteeing both safety and blocking-free resolution. In particular, opinion dynamics enable the safety controller to achieve collaborative decision-making for blocking resolution and facilitate rapid, safe coordination without relying on communication or preset rules. Extensive simulation results validate the improved flight efficiency and safety guarantees. This study provides practical insights into the design of autonomous controllers for airplanes.

Keywords: Robot Safety, Safety in HRI, Reinforcement Learning
Abstract: Goal-conditioned policies, such as those learned via imitation learning, provide an easy way for humans to influence what tasks robots accomplish. However, these robot policies are not guaranteed to execute safely or to succeed when faced with out-of-distribution goal requests. In this work, we enable robots to know when they can confidently execute a user's desired goal, and automatically suggest safe alternatives when they cannot. Our approach is inspired by control-theoretic safety filtering, wherein a safety filter minimally adjusts a robot's candidate action to be safe. Our key idea is to pose alternative suggestion as a safe control problem in goal space, rather than in action space. Offline, we use reachability analysis to compute a goal-parameterized reach-avoid value network which quantifies the safety and liveness of the robot’s pre-trained policy. Online, our robot uses the reach-avoid value network as a safety filter, monitoring the human's given goal and actively suggesting alternatives that are similar but meet the safety specification. We demonstrate our Safe ALTernatives (SALT) framework in simulation experiments with Franka Panda tabletop manipulation. We find that SALT is able to learn to predict successful and failed closed-loop executions, is a less pessimistic monitor than open-loop uncertainty quantification, and proposes alternatives that consistently align with those that people find acceptable.

Keywords: Robot Safety, Manipulation Planning, Optimization and Optimal Control
Abstract: Safe real-time control of robotic manipulators in unstructured environments requires handling numerous safety constraints without compromising task performance. Traditional approaches, such as artificial potential fields (APFs), suffer from local minima, oscillations, and limited scalability, while model predictive control (MPC) can be computationally expensive. Control barrier functions (CBFs) offer a promising alternative due to their high level of robustness and low computational cost, but these safety filters must be carefully designed to avoid significant reductions in the overall performance of the manipulator. In this work, we introduce an Operational Space Control Barrier Function (OSCBF) framework that integrates safety constraints while preserving task-consistent behavior. Our approach scales to hundreds of simultaneous constraints while retaining real-time control rates, ensuring collision avoidance, singularity prevention, and workspace containment even in highly cluttered settings or during dynamic motions. By explicitly accounting for the task hierarchy in the CBF objective, we prevent degraded performance across both joint-space and operational-space tasks, when at the limit of safety. We validate performance in both simulation and hardware, and release our open-source high-performance code and media on our project webpage, https://stanfordasl.github.io/oscbf/

Keywords: Robot Safety, Robust/Adaptive Control, Collision Avoidance
Abstract: Modern autopilot systems are prone to sensor attacks that can jeopardize flight safety. To mitigate this risk, we proposed a modular solution: the secure safety filter, which extends the well-established control barrier function (CBF)-based safety filter to account for, and mitigate, sensor attacks. This module consists of a secure state reconstructor (which generates plausible states) and a safety filter (which computes the safe control input that is closest to the nominal one). Differing from existing work focusing on linear, noise-free systems, the proposed secure safety filter handles bounded measurement noise and, by leveraging reduced-order model techniques, is applicable to the nonlinear dynamics of drones. Software-in-the-loop simulations and drone hardware experiments demonstrate the effectiveness of the secure safety filter in rendering the system safe in the presence of sensor attacks.

Keywords: Robot Safety, Machine Learning for Robot Control, Collision Avoidance
Abstract: Robot learning has produced remarkably effective “black-box” controllers for complex tasks such as dynamic locomotion on humanoids. Yet ensuring dynamic safety, i.e., constraint satisfaction, remains challenging for such policies. Reinforcement learning (RL) embeds constraints heuristically through reward engineering, and adding or modifying constraints requires retraining. Model-based approaches, like control barrier functions (CBFs), enable runtime constraint specification with formal guarantees but require accurate dynamics models. This paper presents SHIELD, a layered safety framework that bridges this gap by: (1) training a generative, stochastic dynamics residual model using real-world data from hardware rollouts of the nominal controller, capturing system behavior and uncertainties; and (2) adding a safety layer on top of the nominal (learned locomotion) controller that leverages this model via a stochastic discrete-time CBF formulation enforcing safety constraints in probability. The result is a minimally-invasive safety layer that can be added to the existing autonomy stack to give probabilistic guarantees of safety that balance risk and performance. In hardware experiments on an Unitree G1 humanoid, SHIELD enables safe navigation (obstacle avoidance) through varied indoor and outdoor environments using a nominal (unknown) RL controller and onboard perception.

Keywords: Robot Safety, Deep Learning Methods, Vision-Based Navigation
Abstract: The robustness certification of deep neural networks (DNNs) is crucial in many safety-critical domains. Randomized Smoothing (RS) has emerged as the current state-of-the-art method for DNN robustness verification that successfully scales on large DNNs used in practice, has achieved excellent results, and has been extended for a large variety of adversarial perturbation scenarios. However, an important cost in RS is during inference, since it requires passing tens or hundreds of thousands of perturbed samples through the DNN to perform the verification. In this work we aim to address this, and explore what happens as we decrease the number of samples by orders of magnitude, and the effect on the certified radius. Surprisingly, we find that emph{the performance reduction in terms of average certified radius is not too large, even if we decrease the number of samples by two orders of magnitude, or more}. Moreover, we find that the resulting certified radius reduction can be mitigated using off-the-self methods designed to improve RS performance. This can pave the way for dramatically faster robustness certification, unlocking the possibility of performing it in real-time, which we demonstrate. We perform a detailed analysis, both theoretically and experimentally, and show promising results on the standard CIFAR-10 and ImageNet datasets.

Keywords: Aerial Systems: Mechanics and Control, Motion Control
Abstract: This paper presents a novel, compact overactuated tricopter featuring a servo-driven twisting and tilting mechanism, preventing the adverse effects of internal force contradiction during flight. Each arm's vectored thrust is provided by a single motor, with the twisting and tilting angles controlled by two vertically mounted servos. These components are collectively mounted within a 3D-printed semi-ring structure, and are rigidly attached to the fuselage via carbon tubes at the twist end. To address the asymmetry inherent in the tricopter configuration, we conducted a qualitative analysis of the disturbances introduced by the actuators. Additionally, we emphasize the need to include gyroscopic torque effects caused by arm rotations. This issue is addressed using a control allocation method, with our proposed improved Force Decomposition (FD)-based iteration offering a low-cost computational solution. The dynamic models of the motion of rotational joints, identified and employed as virtual sensors, contribute to the estimation of the improved control effective matrix. This overactuated tricopter can operate like a conventional rotorcraft, with the added capability of achieving attitude adjustments through manual control inputs. Finally, we demonstrate the tricopter’s advantages by comparing simulations and flight experiments, both with and without the application of the improved method.

Keywords: Aerial Systems: Mechanics and Control, Intelligent Transportation Systems, Underactuated Robots
Abstract: Aerial robots have demonstrated significant potential in suspended cargo transportation, especially in industries such as logistics and food delivery. Due to the underactuated and nonlinear dynamics of the cable-suspended system, directly tracking a given trajectory with a multicopter without modifying its controller often leads to significant payload swing. This compromises the safety and stability of the cargo. To address the aforementioned issue, this paper proposes an online trajectory refinement method for a variable-length cable-suspended aerial transportation robot, independent from the control layer. By incorporating payload swing angle information, the reference trajectory is refined in real-time, effectively suppressing payload oscillations during transportation. Specially, Lyapunov techniques and LaSalle's invariance theorem are employed to rigorously guarantee the feasibility of the designed trajectory refinement scheme. Finally, hardware experiments are conducted to validate the effectiveness and superiority of the proposed method. The results demonstrate that the refined trajectory not only enables precise positioning of the multicopter, but also effectively suppresses payload oscillations during transportation, significantly enhancing the safety and reliability of the aerial cargo delivery.

Keywords: Autonomous Vehicle Navigation, Wheeled Robots, Motion Control
Abstract: In this paper, we propose a differential-flatness-based control~(DFBC) framework for precise tracking control of tractor-trailers, particularly during reversing maneuvers, which are challenging due to the unstable nonlinear kinematics. The proposed controller leverages the differential flatness property of tractor-trailers, equivalently transforming the nonlinear kinematics into a Brunovsky canonical from, thus allowing the incorporation of a linear feedback mechanism within the space defined by the flat outputs and their derivatives. Compared to the traditional linear quadratic regulator~(LQR) controllers based on Jacobian linearization, the proposed DFBC method achieves higher precision and stability in reversing maneuvers. We also implement the proposed DFBC method on a self-developed 1/10 scale autonomous tractor-trailer to showcase its effectiveness in real-world scenarios.

Keywords: Biologically-Inspired Robots, Redundant Robots, Motion Control
Abstract: Centipedes exhibit great maneuverability in diverse environments due to their many legs and body-driven control. By leveraging similar morphologies and control strategies, their robotic counterparts also demonstrate effective terrestrial locomotion. However, the success of these multi-legged robots is largely limited to forward locomotion; steering is substantially less studied, in part because of the difficulty in coordinating a high degree-of-freedom robot to follow predictable, planar trajectories. To resolve these challenges, we take inspiration from control schemes based on geometric mechanics(GM) in elongate systems' locomotion through highly damped environments. We model the elongate, multi-legged system as a ``terrestrial swimmer" in highly frictional environments and implement steering schemes derived from low-order templates. We identify an effective turning strategy by superimposing two traveling waves of lateral body undulation and further explore variations of the ``turning wave" to enable a spectrum of arc-following steering primitives. We test our hypothesized modulation scheme on a robophysical model and validate steering trajectories against theoretically predicted displacements producing steering radii between 0 and 0.6 body length. We then apply our control framework to Ground Control Robotics' elongate multi-legged robot, Major Tom, using these motion primitives to autonomously navigate around obstacles and corners on indoor and outdoor terrain. Our work creates a systematic framework for controlling these highly mobile devices in the plane using a low-order model based on sequences of body shape changes.

Keywords: Climbing Robots, Industrial Robots, Motion Control
Abstract: Wheeled climbing robots have great application prospects in the machining of large components with variable curvature. However, its accurate motion control on variable curvature surfaces faces two fatal challenges. The varied contact states between the robot’s wheels and the variable curvature surfaces make it difficult to establish an accurate kinematics model. Additionally, there exists a different degree of robot slippage when the robot moves in different attitudes due to the dragging effect of gravity. To overcome the above problems, we first present a kinematics modeling method with instantaneous plane constraints on a variable curvature surface. Subsequently, a linear extended state observer (LESO)-based nonlinear model predictive control (NMPC) scheme is designed, in which the NMPC is used to calculate the nominal control inputs and the LESO is used to estimate and compensate for lumped disturbance brought by the robot slippage and surface constraints. Experiments on a real wind turbine blade with variable curvature show that the proposed control scheme can well eliminate the influence of lumped disturbance, and the climbing robots can achieve unbiased (AVG < 0.1 mm) and high-precision (RMSE < 2 mm) trajectory tracking. provides a solution for automatic and high-precision trajectory tracking for climbing robot with localization system in factories, offering the potential for high-quality machining of large components.

Keywords: Robust/Adaptive Control, Motion Control
Abstract: This paper proposes a novel adaptive global sliding mode control strategy for a two-degree-of-freedom (2- DOF) piezoelectric nanopositioning system based on the integral extended state observer (IESO) technique. Its uniqueness is that a global robustness property is generated in the whole precision motion control process, which effectively circumvents sensitivity to the perturbations during the reaching phase. First, to generate an accurate disturbance estimation for compensation control, the IESO is constructed by incorporating an integral action into the observer design. Then, a global sliding mode control approach is developed for the nanopositioning system to ensure global robustness against the unknown hysteresis nonlinearity and crossaxis coupling motion. Moreover, an adaptive rule is established for the global sliding mode controller, which does not require a priori knowledge of the estimation error, hysteresis, and crosscoupling nonlinearity in the control design. Experimental studies are conducted to demonstrate the effectiveness and superiority of the proposed motion control scheme over existing ones.

Keywords: Micro/Nano Robots, Soft Sensors and Actuators, Soft Robot Materials and Design
Abstract: Biological machines that use biological cells and soft materials in combination to obtain a sense of the environment driven by bioenergy and generate driving force are called biohybrid actuators. With the development of tissue engineering and organoid technology, researchers have applied biohybrid actuators technology to the research of precision medicine and targeted drug delivery, but the research on feedback and evaluation of biohybrid actuation performance is limited to visual and simulation calculations. Therefore, we hope to develop an intelligent crawling skeleton for sensing function, which can be used to evaluate the actuation ability of muscle actuators, and eventually realize the high-precision control of biohybrid actuators. In this work, an intelligent crawling skeleton based on three-dimensional liquid metal is proposed to detect and feedback the crawling of C2C12 muscle actuators. Three-dimensional muscle tissue was composed of mixing hydrogels and cells, and the functionalization of muscle rings was promoted using static mechanical forces and external electric field stimulation. The composite crawling skeleton is fabricated by inverting mold and soft lithography technology. The skeleton can adapt to large deformations above 90 degrees and is more sensitive to deformations by adjusting materials with different elastic modulus. Inspired by the tendon-bone structure, the intelligent crawling skeleton can obtain the deformation degree of the biohybrid actuator in the crawling process according to the characteristics of the deformation from the muscle tissue, and put forward a good idea for the feedback and closed-loop control of the biohybrid actuators.

Keywords: Micro/Nano Robots, Nanomanufacturing, Automation at Micro-Nano Scales
Abstract: DNA nanorobots have emerged as a promising technology in the biomedical field, owing to their nanoscale dimensions, programmable structures, and minimal physiological toxicity. However, existing DNA-based carriers often exhibit limited stability in complex biological environments and lack efficient targeted drug delivery capabilities. Addressing these challenges requires the development of DNA nanorobots with highly controllable and multifunctional designs for precise biomedical applications. This study introduces a conformational transition design of tetrahedral DNA structures, leveraging a multi-resolution molecular dynamics simulation model validated by all-atom molecular modeling. Through this approach, we dynamically analyzed the stability of the nanomechanical structures using simulation-based methods. Utilizing this design strategy, we successfully constructed a deformable tetrahedral DNA nanorobot characterized by high stability and efficient drug delivery capabilities. Furthermore, we demonstrated the precise release of therapeutic agents with low physiological toxicity and high controllability by employing the DNA nanorobot for in vitro targeting of circulating tumor cells (CTCs). These findings highlight the potential of the proposed DNA origami-based nanorobot design method to advance the development of nanorobots in biomedicine and their application in targeted therapeutic interventions.

Keywords: Micro/Nano Robots, Medical Robots and Systems
Abstract: 自动表征机械性能 本研究提出了一种新的肿瘤和正常细胞 使用窄微通道诱导和测量的方法 细胞变形。动态力学表征 串行连接微通道的技术模拟 肿瘤细胞在生物学上的变形和迁移 相关环境，实现明确差异化 肿瘤和正常细胞之间通过连续 压缩。高速成像结合计算机 视觉和图像处理技术有助于快速 以及对肿瘤细胞的准确自动分析。使用三种肿瘤细胞系和三种正常细胞系的实验 细胞系验证了该方法。此外，这项研究 揭示了细胞核的机械性能 确定整体细胞力学，使用 肿瘤细胞和正常细胞之间的差异归因于 原子核力学的变化。这种方法表明 早期癌症诊断的前景。

Keywords: Micro/Nano Robots, Biological Cell Manipulation, Automation at Micro-Nano Scales
Abstract: Microfluidic technology is currently a popular approach in the field of single-cell research, which is used to reveal the heterogeneity among cells. However, most of the existing microfluidic technologies for single-cell research lack the ability to control the microenvironment of single cells after isolating them. In this work, a technology that combines lateral field optoelectronic tweezers (LOET) with electrowetting-on dielectric (EWOD) is used to separate cells into single cells and then encapsulate each single cell within an individual droplet, generating single-cell droplets. More importantly, it also enables the control of the microenvironment of the separated single cells. The driving control of the single- cell droplets is achieved through the EWOD, which has good application prospects in the field of single-cell research.

Keywords: Micro/Nano Robots, Biologically-Inspired Robots, Mechanism Design
Abstract: Insect-scale flapping-wing micro aerial vehicles (FWMAVs) employing piezoelectric direct-drive configurations eliminate traditional kinematic chains through direct coupling of the wing and actuator. While this design approach significantly reduces structural complexity and manufacturing costs compared to transmission-dependent systems, it inherently limits wing stroke amplitude and consequent lift generation. This paper presents a novel lift-enhancement strategy for piezoelectric direct-drive FWMAVs, effectively improving payload capacity through optimized aerodynamic performance. The redesigned X-configuration prototype demonstrates outstanding metrics: 68 mm wingspan with 212 mg total mass achieves 7.47 mN maximum lift (exceeding 3.5:1 lift-to-weight ratio) and 1.25 m/s takeoff speed. Experimental validation confirms 39% payload capacity improvement and 34% lift-to-weight ratio enhancement compared to baseline designs. This enhancement establishes our robot as the current state-of-the-art in piezoelectric direct-drive FWMAVs regarding lift-to-weight ratio.

Keywords: Micro/Nano Robots, Automation at Micro-Nano Scales, Soft Robot Materials and Design
Abstract: Miniature robots hold great promise for performing micromanipulation tasks within hard-to-reach confined spaces. However, effectively maneuvering across complex and unstructured terrain, achieving adaptive morphogenesis, and developing adaptive multimodal locomotion strategies remain challenges for these robotic systems. Here, we develop a multi-stimulus-responsive deformable miniature robot integrated with an adaptive multimodal motion control method. Sodium alginate hydrogel and graphene-coated magnetic elastomer are integrated into the sheet-shaped robot to enable responsiveness to temperature, humidity, and magnetic fields. A kinematic gait model is designed to control oscillatory motion in the semi-contracted state and rotational motion in the fully contracted state of the miniature robot. To automatically mitigate angular deviation between the robot's motion direction and the intended path, an adaptive orientation compensation control algorithm based on Support Vector Regression (SVR) is proposed. Experimental results demonstrate that the proposed robot exhibits capabilities for flexible and accurate navigation within unstructured environments (e.g., rock piles and stomach models), and is further shown to be capable of cargo transport. The proposed adaptive morphogenesis robots, enabled by dual-mode motion control, hold significant potential for targeted delivery and other micromanipulation applications in complex, unstructured, and confined environments.

Keywords: Micro/Nano Robots, Medical Robots and Systems, Dynamics
Abstract: Micro-nano robots must break the symmetry of the flow field to generate net displacement in the low Reynolds number environment. The spherical micro-robots utilize the frictional forces generated through interaction with the surface. We designed a magnetic microroller robot powered by the rotating AC magnetic field. Here, we employed dual measurements of laser ranging and computer vision to demonstrate that a single 100 μm microroller maintains a lubrication film of 1 to 15 μm with the surface during normal motion. We found that the translational velocity of the microroller is correlated with the lubrication film thickness. Based on the robot's gravity, we controlled an additional downward gradient magnetic field to effectively increase the load of robot and reduce the lubrication film thickness, thereby controllably increasing the translational velocity of the robot. For example, the gradient magnetic field generated by superimposing a 30 mA direct current input can reduce the lubrication film thickness from 8 μm to 4 μm in a 10 Hz rotating magnetic field, and increase the translational velocity from 230 μm/s to 460 μm/s. The enhancement of the robot's motion performance enables it to better control its movement in fluids. Finally, we validated the strategy for controllable acceleration of micro-scale particles rolling on surfaces, applied to control fluid motion in multiple arteries within blood vessels. These results offer deeper insights into the physical motion mechanism of surface robots and hold significant implications for future applications in biomedical engineering.

Keywords: Micro/Nano Robots, Automation at Micro-Nano Scales, Robotics and Automation in Life Sciences
Abstract: Magnetic helical microrobots have been widely applicated in environmental remediation, sensing, targeted medical applications, and so on. However, for locomotion and manipulation in unstructured liquid environments, the capabilities of distinguished motions and on-demand parking/starting over a team of microrobot are essential. Here, we propose a method for achieving on-demand motions conversion of helical microrobots by modulating surface wettability through surface chemical modification and surface microstructural modifications. An obvious difference shows that microrobot after chemical modification exhibit hydrophilicity and microrobot after microstructural modification exhibit hydrophobicity, where the latter possess higher moving step-out frequency and maximum forward velocity compared to the microrobot after surface chemical modification. The step-out frequencies and maximum velocities of the three types of microrobots (chemistry-modified, unmodified, and pimples-modified) are 13 Hz, 16 Hz, 22 Hz, and 385 μm/s, 511 μm/s, 649 μm/s. Furthermore, our method has demonstrated that can be employed to achieve effective on-demand targeted motions and modal conversion in liquid environment. We anticipate that the method can be potentially employed to achieve precise targeted drug delivery and surgery in biomedical applications

Keywords: Motion and Path Planning, Collision Avoidance, Robot Safety
Abstract: Deploying mobile robots safely among humans requires the motion planner to account for the uncertainty in the other agents' predicted trajectories. This remains challenging in traditional approaches, especially with arbitrarily shaped predictions and real-time constraints. To address these challenges, we propose a Dynamic Risk-Aware Model Predictive Path Integral control (DRA-MPPI), a motion planner that incorporates uncertain future motions modelled with potentially non-Gaussian stochastic predictions. By leveraging MPPI’s gradient-free nature, we propose a method that efficiently approximates the joint Collision Probability (CP) among multiple dynamic obstacles for several hundred sampled trajectories in real-time via a Monte Carlo (MC) approach. This enables the rejection of samples exceeding a predefined CP threshold or the integration of CP as a weighted objective within the navigation cost function. Consequently, DRA-MPPI mitigates the freezing robot problem while enhancing safety. Real-world and simulated experiments with multiple dynamic obstacles demonstrate DRA-MPPI’s superior performance compared to state-of-the-art approaches, including Scenario-based Model Predictive Control (S-MPC), Frenét planner, and vanilla MPPI. Videos of the experiments can be found at https://autonomousrobots.nl/paper_websites/dra-mppi.

Keywords: Motion and Path Planning, Aerial Systems: Perception and Autonomy, Reinforcement Learning
Abstract: Robots are frequently tasked to gather relevant sensor data in unknown terrains. A key challenge for classical path planning algorithms used for autonomous information gathering is adaptively replanning paths online as the terrain is explored given limited onboard compute resources. Recently, learning-based approaches emerged that train planning policies offline and enable computationally efficient online replanning performing policy inference. These approaches are designed and trained for terrain monitoring missions assuming a single specific map representation, which limits their applicability to different terrains. To address this limitation, we propose a novel formulation of the adaptive informative path planning problem unified across different map representations, enabling training and deploying planning policies in a larger variety of monitoring missions. Experimental results validate that our novel formulation easily integrates with classical non-learning-based planning approaches while maintaining their performance. Our trained planning policy performs similarly to state-of-the-art map-specifically trained policies. We validate our learned policy on unseen real-world terrain datasets.

Keywords: Motion and Path Planning, AI-Based Methods, Human-Robot Collaboration
Abstract: Recent advances in research have demonstrated that Vision-Language Models (VLMs) are a promising technology for robot task planning. This paper presents a novel approach that leverages visual prompts and VLMs to generate feasible robot action sequences for achieving shared tasks through human-robot collaboration while simultaneously estimating human intentions. Our method enhances VLMs’ understanding of the environment by utilizing annotations (bounding boxes and labels) and dynamically infers human intentions based on changing environmental conditions to generate optimal robot action sequences to achieve common goals. Additionally, the system incorporates a mechanism to regenerate new sequences through VLM analysis when action failures or external interference occur. Furthermore, by designing prompts as versatile modules for diverse tasks, our proposed technology offers a new approach to robot action planning that excels in both efficiency and adaptability.

Keywords: Motion and Path Planning, Autonomous Agents, Imitation Learning
Abstract: Autonomous vehicle trajectory planning faces significant challenges in dynamic traffic environments due to the complex and mixed causal relationships between critical scene elements (e.g., pedestrians, vehicles, road markings) and safe decision-making. To identify the causal factors influencing planning outcomes, we propose Causal-Planner, which disentangles the scene interaction graph into causal and confounding components via attention-based adversarial graph learning. Additionally, we introduce a long-short-term episodic memory gating (LSTEM) module that enhances causal interaction disentangling by adaptively capturing evolving causal relationships in dynamic scenarios through bidirectional gated memory fusion. Extensive experiments on the nuPlan dataset suggest that Causal-Planner achieves competitive performance, performing well in both Test-random and Test-hard scenarios under open-loop and closed-loop evaluations.

Keywords: Motion and Path Planning, Autonomous Agents, Planning, Scheduling and Coordination
Abstract: Autonomous motion planning under unknown nonlinear dynamics presents significant challenges. An agent needs to continuously explore the system dynamics to acquire its properties, such as reachability, in order to guide system navigation adaptively. In this paper, we propose a hybrid planning-control framework designed to compute a feasible trajectory toward a target. Our approach involves partitioning the state space and approximating the system by a piecewise affine (PWA) system with constrained control inputs. By abstracting the PWA system into a directed weighted graph, we incrementally update the existence of its edges via affine system identification and reach control theory, introducing a predictive reachability condition by exploiting prior information of the unknown dynamics. Heuristic weights are assigned to edges based on whether their existence is certain or remains indeterminate. Consequently, we propose a framework that adaptively collects and analyzes data during mission execution, continually updates the predictive graph, and synthesizes a controller online based on the graph search outcomes. We demonstrate the efficacy of our approach through simulation scenarios involving a mobile robot operating in unknown terrains, with its unknown dynamics abstracted as a single integrator model.

Keywords: Motion and Path Planning, Collision Avoidance, Computational Geometry
Abstract: We propose the BSSM: Point-Cloud based (B)iased (S)igned (S)mooth (M)etric, which is used to compute a distance metric between a manipulator and its environment. Unlike many methods that requires that the environment is modeled using simple geometric primitives such as spheres, boxes, and cylinders, our proposed metric directly utilizes point clouds. The proposed metric has properties of being smooth (infinitely differentiable), signed (yielding non-zero values upon overlap), and biased. The latter is a novel feature that, as demonstrated by our simulation results, offers advantages for motion planning. This metric is suitable for GPU parallelization and simulation studies and a real experiment are offered to investigate its benefits.

Keywords: Motion and Path Planning, Collision Avoidance, Motion Control
Abstract: This paper presents a novel method for modeling the shape of a continuum robot as a Neural Configuration Euclidean Distance Function (N-CEDF). By learning separate distance fields for each link and combining them through the kinematics chain, the learned N-CEDF provides an accurate and computationally efficient representation of the robot's shape. The key advantage of a distance function representation of a continuum robot is that it enables efficient collision checking for motion planning in dynamic and cluttered environments, even with point-cloud observations. We integrate the N-CEDF into a Model Predictive Path Integral (MPPI) controller to generate safe trajectories for multi-segment continuum robots. The proposed approach is validated for continuum robots with various links in several simulated environments with static and dynamic obstacles.

Keywords: Motion and Path Planning, Collision Avoidance, Optimization and Optimal Control
Abstract: This contribution presents a robot path-following framework via Reactive Model Predictive Contouring Control (RMPCC) that successfully avoids obstacles, singularities and self-collisions in dynamic environments at 100 Hz. Many path-following methods rely on the time parametrization, but struggle to handle collision and singularity avoidance while adhering kinematic limits or other constraints. Specifically, the error between the desired path and the actual position can become large when executing evasive maneuvers. Thus, this paper derives a method that parametrizes the reference path by a path parameter and performs the optimization via RMPCC. In particular, Control Barrier Functions (CBFs) are introduced to avoid collisions and singularities in dynamic environments. A Jacobian-based linearization and Gauss-Newton Hessian approximation enable solving the nonlinear RMPCC problem at 100 Hz, outperforming state-of-the-art methods by a factor of 10. Experiments confirm that the framework handles dynamic obstacles in real-world settings with low contouring error and low robot acceleration.

Keywords: Flexible Robotics, Medical Robots and Systems, Mechanism Design
Abstract: Natural Orifice Transluminal Endoscopic Surgery (NOTES) holds great promise due to its ability to eliminate external incisions, reduce trauma, and accelerate recovery. However, the adoption of NOTES is hindered by the limited capabilities of existing instruments, particularly in achieving the required balance between compact size, dexterity, and load capacity. This paper introduces a novel robotic manipulator designed for NOTES, featuring a 5 mm diameter and 7 degrees of freedom (DoF). The manipulator incorporates an innovative 3-PRS flexible parallel mechanism combined with a continuum parallel structure, achieving enhanced dexterity and variable stiffness functionality within a miniaturized design. A kinematic and variable stiffness analysis is performed, and experimental validation demonstrates its bending performance and stiffness modulation. Additionally, the feasibility and practicality of the robotic system are confirmed through a peg-transfer experiment, proving its potential for real-world surgical applications. This research offers a viable solution for enhancing the performance of NOTES instruments.

Keywords: Medical Robots and Systems, Flexible Robotics
Abstract: Ensuring both mobility and stability during colonoscopy is crucial for enhancing procedural efficiency and reducing the risk of complications. Traditional colonoscopy robots face challenges due to the fixed diameter of the colonoscope and the variability in colon anatomy. To address this, we propose a soft automatic anchoring system (SAAS) to enhance mobility and stability in colonoscopy robots. The SAAS features proximal and distal soft balloon anchors, employing origami principles to achieve a 42.2% higher maximum expansion capability compared to conventional flat surface anchors. With real-time pressure feedback, the SAAS automatically anchors upon contact detection, ensuring precise anchoring without over-inflation while adapting to colon diameter variations and robot posture changes. Experimental results show a tenfold reduction in displacement during the stability test, significantly enhancing the robot's performance under external loads. Performance comparison tests in phantom further demonstrated notable improvements in the efficiency of colonoscopy procedures using the SAAS. This system has the potential to greatly enhance the safety, precision, and overall efficiency of colonoscopy, offering substantial benefits for both medical practitioners and patient outcomes.

Keywords: Medical Robots and Systems, Virtual Reality and Interfaces
Abstract: Robotic needle positioning tasks in neurosurgery often face challenges due to insufficient perception of planar guidance images during surgery. In this work, we propose an Augmented Reality (AR) interface to help perform the robotic needle positioning tasks by learning from demonstration (LfD). Enhanced immersion in the workflow is achieved by displaying surgical scenes and calculated navigation information. The framework utilizes mixed interactive interfaces in virtual and real environments, enhancing demonstration efficiency and quality. A head-mounted display and an optical tracking system are utilized to perform the visualization and needle tracking. Gaussian Mixture Model (GMM) and Gaussian Mixture Re- gression (GMR) are employed to learn a robust and smooth trajectory policy from demonstrations. Experiments on robot reproduction of the needle positioning task achieved a final positioning error of 0.6 mm and an average trajectory error of 1.07 mm. Comparative user studies with haptic device-based teleoperation exhibit a low completion time of 62.76 s and reduced workload of the proposed system.

Keywords: Medical Robots and Systems, Tendon/Wire Mechanism, Soft Robot Materials and Design
Abstract: Minimally Invasive Surgery has advanced surgical practice, yet early-stage gastrointestinal cancer treatment remains challenging. Endoscopic Submucosal Dissection offers a solution but faces maneuverability constraints in complex anatomical environments. This paper presents a novel inflatable soft robotic manipulator with a biocompatible thin-film shell and tendon-driven antagonistic actuation. The robot remains compact at 6.5 mm of diameter, expanding to 11.7 mm to enhance stiffness for force exertion and precise manipulation. Featuring two decoupled wrists with four degrees of freedom, it enables dexterous motion for advanced endoscopic procedures. The study details design, fabrication, actuation modeling, workspace evaluation, and simulated retraction experiments in a constrained environment. Results demonstrate high repeatability, high master-slave control accuracy, effective workspace utilization, and feasibility for endoluminal applications, enhancing robotic-assisted endoscopic procedures with improved dexterity and adaptability under soft actuation constraints.

Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles
Abstract: To address the screw loosening and pullout limitations of rigid pedicle screws in spinal fixation procedures, and to leverage our recently developed Concentric Tube Steerable Drilling Robot (CT-SDR) and Flexible Pedicle Screw (FPS), in this paper, we introduce the concept of Augmented Bridge Spinal Fixation (AB-SF). In this concept, two connecting J- shape tunnels are first drilled through pedicles of vertebra using the CT-SDR. Next, two FPSs are passed through this tunnel and bone cement is then injected through the cannulated region of the FPS to form an augmented bridge between two pedicles and reinforce strength of the fixated spine. To experimentally analyze and study the feasibility of AB-SF technique, we first used our robotic system (i.e., a CT-SDR integrated with a robotic arm) to create two different fixation scenarios in which two J-shape tunnels, forming a bridge, were drilled at different depth of a vertebral phantom. Next, we implanted two FPSs within the drilled tunnels and then successfully simulated the bone cement augmentation process.

Keywords: Medical Robots and Systems, Physical Human-Robot Interaction, Computer Vision for Medical Robotics
Abstract: This paper presents an uncertainty-aware shared control and calibration method for micromanipulation using a digital microscope and a tool-mounted, multi-joint robotic arm, integrating real-time human intervention with a visual-motor policy. Our calibration algorithm leverages co-manipulation control to calibrate the hand-eye relationship without requiring knowledge of the kinematics of the microtool mounted on the robot while remaining robust to camera intrinsics errors.Experimental results show that the proposed calibration method achieves a 39.6% improvement in accuracy over established methods. Additionally, our control structure and calibration method reduces the time required to reach single-point targets from 5.74 s (best conventional method) to 1.91 s, and decreases trajectory tracking errors from 392 um to 40 um. These findings establish our method as a robust solution for improving reliability in high-precision biomedical micromanipulation.

Keywords: Medical Robots and Systems, Machine Learning for Robot Control, Vision-Based Navigation
Abstract: Vision-Language-Action (VLA) models have emerged as a prominent research area, showcasing significant potential across a variety of applications. However, their performance in endoscopy robotics, particularly endoscopy capsule robots that perform actions within the digestive system, remains unexplored. The integration of VLA models into endoscopy robots allows more intuitive and efficient interactions between human operators and medical devices, improving both diagnostic accuracy and treatment outcomes. In this work, we design CapsDT, a Diffusion Transformer model for capsule robot manipulation in the stomach. By processing interleaved visual inputs, and textual instructions, CapsDT can infer corresponding robotic control signals to facilitate endoscopy tasks. In addition, we developed a capsule endoscopy robot system, a capsule robot controlled by a robotic arm-held magnet, addressing different levels of four endoscopy tasks and creating corresponding capsule robot datasets within the stomach simulator. Comprehensive evaluations on various robotic tasks indicate that CapsDT can serve as a robust vision-language generalist, achieving state-of-the-art performance in various levels of endoscopy tasks while achieving a 26.25% success rate in real-world simulation manipulation.

Keywords: Medical Robots and Systems, Modeling, Control, and Learning for Soft Robots, Robot Safety
Abstract: Increased demand for minimally invasive surgery has accelerated the adoption of natural orifice transluminal endoscopic surgery. Meanwhile, the increasing complexity of endoscopic procedures has raised the demand for endoscopist competence. In this paper, we present

a novel endoscopic system to perform robotic intervention with a flexible endoscope to enhance the safety and efficiency of endoscopy. The system is composed of an endoscopic manipulation module (EMM) for control of endoscopic interventions, a 6 DOFs robotic arm for adjusting

the pose of EMM, and a haptic interface for tele-control. The compact system enables endoscopists to remotely and stably operate flexible endoscopes with one hand. Mas-ter-slave control strategy with active constraint is proposed to guide the motion of the flexible endoscope within the safe working space. The proposed system is validated by recruiting subjects to perform target area localization and endoscopic examinations through highly realistic upper digestive tract phantom. The system can reduce the time of target area localization by at least 10% compared to manual control. The mean angle errors can be reduced by 75% with the help of the constraint force. Results demonstrate the potential clinical value of the system in efficient operation and safety control.

Keywords: Computer Vision for Automation, Biologically-Inspired Robots
Abstract: Vision-based perception systems are typically exposed to large orientation changes in different robot applications. In such conditions, their performance might be compromised due to the inherit complexity of processing data captured under challenging motion. Integration of mechanical stabilizers to compensate for the camera rotation is not always possible due to the robot payload constraints. This paper presents a processing-based stabilization approach to compensate for the camera’s rotational motion both on events and on frames(i.e., images). Assuming that the camera’s attitude is available, we evaluate the benefits of stabilization in two perception applications: feature tracking and estimating the translation component of the camera’s ego-motion. The validation is performed using synthetic data and sequences from well-known event-based vision datasets. The experiments unveil that stabilization can improve feature tracking and camera ego-motion estimation accuracy in 27.37% and 34.82%, respectively. Concurrently, stabilization can reduce the processing time of computing the camera’s linear velocity by at least 25%.

Keywords: Computer Vision for Automation, Perception for Grasping and Manipulation, Dual Arm Manipulation
Abstract: Deformable Multi-Linear Objects (DMLOs), or Branched Deformable Linear Objects (BDLOs), are flexible objects that possess a linear structure similar to DLOs but also feature branching or bifurcation points where the object's path diverges into multiple sections. The representation of complex DMLOs, such as wiring harnesses, poses significant challenges in various applications, including robotic systems' perception and manipulation planning. This paper proposes an approach to address the robust and efficient estimation of a topological representation for DMLOs leveraging a graph-based description of the scene obtained via graph neural networks. Starting from a binary mask of the scene, graph nodes are sampled along the objects' estimated centerlines. Then, a data-driven pipeline is employed to learn the assignment of graph edges between nodes and to characterize the node's type based on their local topology and orientation. Finally, by utilizing the learned information, a solver combines the predictions and generates a coherent representation of the objects in the scene. The approach is experimentally evaluated using a test set of complex real-world DMLOs. Within an offline evaluation, the proposed approach achieves a Dice score exceeding 90% in predicting graph edges. Similarly, the identification accuracy of branch and intersection points in the graph topology is above 90%. Additionally, the method demonstrates efficient performance, achieving a runtime of over 20 FPS. In an online assessment employing a dual-arm robotic setup, the approach is successfully applied to disentangle three automotive wiring harnesses, demonstrating the effectiveness of the proposed approach in a real-world scenario.

Keywords: Computer Vision for Medical Robotics, Deep Learning Methods, Medical Robots and Systems
Abstract: Ultrasound (US)-guided needle insertion is widely employed in percutaneous interventions. However, providing feedback on the needle tip position via US image presents challenges due to noise, artifacts, and the thin imaging plane of US, which degrades needle features and leads to intermittent tip visibility. In this paper, a Mamba-based US needle tracker MambaXCTrack utilizing structured state space models cross-correlation (SSMX-Corr) and implicit motion prompt is proposed, which is the first application of Mamba in US needle tracking. The SSMX-Corr enhances cross-correlation by long-range modeling and global searching of distant semantic features between template and search maps, benefiting the tracking under noise and artifacts by implicitly learning potential distant semantic cues. By combining with cross-map interleaved scan (CIS), local pixel-wise interaction with positional inductive bias can also be introduced to SSMX-Corr. The implicit low-level motion descriptor is proposed as a non-visual prompt to enhance tracking robustness, addressing the intermittent tip visibility problem. Extensive experiments on a dataset with motorized needle insertion in both phantom and tissue samples demonstrate that the proposed tracker outperforms other state-of-the-art trackers while ablation studies further highlight the effectiveness of each proposed tracking module.

Keywords: Computer Vision for Transportation, Deep Learning for Visual Perception, Intelligent Transportation Systems
Abstract: Vision-based online vessel detection boosts the automation of waterways monitoring, transportation management and navigation safety. However, a significant gap exists in on-device deployment between general high-performance PCs/servers and embedded AI processors. Existing state-of-the-art (SOTA) online vessel detectors lack sufficient accuracy and are prone to high latency on the edge AI camera, especially in scenarios with dense vessels and diverse distributions. To solve the above issues, a novel lightweight framework with density-guided adaptive Transformer (LiVeDet) is proposed for the edge AI camera to achieve online on-device vessel detection. Specifically, a new instance-aware representation extractor is designed to suppress cluttered background noise and capture instance-aware content information. Additionally, an innovative vessel distribution estimator is developed to direct superior feature representation learning by focusing on local regions with varying vessel density. Besides, a novel dynamic region embedding is presented to integrate hierarchical features represented by multi-scale vessels. A new benchmark comprising 100 high-definition, high-frame rate video sequences from vessel-intensive scenarios is established to evaluate the efficacy of vessel detectors under challenging conditions prevalent in dynamic waterways. Extensive evaluations on this challenging benchmark demonstrate the robustness and efficiency of LiVeDet, achieving 32.9 FPS on the edge AI camera. Furthermore, real-world applications confirm the practicality of the proposed method.

Keywords: Computer Vision for Transportation, Collision Avoidance, Data Sets for Robotic Vision
Abstract: Time-to-Collision (TTC) estimation lies in the core of the forward collision warning (FCW) functionality, which is key to all Automatic Emergency Braking (AEB) systems. Although the success of solutions using frame-based cameras (e.g., Mobileye's solutions) has been witnessed in normal situations, some extreme cases, such as the sudden variation in the relative speed of leading vehicles and the sudden appearance of pedestrians, still pose significant risks that cannot be handled. This is due to the inherent imaging principles of frame-based cameras, where the time interval between adjacent exposures introduces considerable system latency to AEB. Event cameras, as a novel bio-inspired sensor, offer ultra-high temporal resolution and can asynchronously report brightness changes at the microsecond level. To explore the potential of event cameras in the above-mentioned challenging cases, we propose EvTTC, which is, to the best of our knowledge, the first multi-sensor dataset focusing on TTC tasks under high-relative-speed scenarios. EvTTC consists of data collected using standard cameras and event cameras, covering various potential collision scenarios in daily driving and involving multiple collision objects. Additionally, LiDAR and GNSS/INS measurements are provided for the calculation of ground-truth TTC. Considering the high cost of testing TTC algorithms on full-scale mobile platforms, we also provide a small-scale TTC testbed for experimental validation and data augmentation. All the data and the design of the testbed are open sourced, and they can serve as a benchmark that will facilitate the development of vision-based TTC techniques.

Keywords: Computer Vision for Transportation, Intelligent Transportation Systems, Deep Learning for Visual Perception
Abstract: In this work, we propose a novel approach, namely WeatherDG, that can generate realistic, weather-diverse, and driving-screen images based on the cooperation of two foundation models, i.e., Stable Diffusion (SD) and Large Language Model (LLM). Specifically, we first fine-tune the SD with source data, aligning the content and layout of generated samples with real-world driving scenarios. Then, we propose a procedural prompt generation method based on LLM, which can enrich scenario descriptions and help SD automatically generate more diverse, detailed images. In addition, we introduce a balanced generation strategy, which encourages the SD to generate high-quality objects of tailed classes under various weather conditions, such as riders and motorcycles. This segmentation-model-agnostic method can improve the generalization ability of existing models by additionally adapting them with the generated synthetic data. Experiments on three challenging datasets show that our method can significantly improve the segmentation performance of different state-of-the-art models on target domains. Notably, in the setting of ''Cityscapes to ACDC'', our method improves the baseline HRDA by 13.9% in mIoU.

Keywords: Computer Vision for Automation, Reactive and Sensor-Based Planning, Task Planning
Abstract: This paper addresses a gap between the capabilities and utilisation of robotics and automation in laboratory settings and builds upon the concept of Self-Driving Labs (SDL). We introduce an innovative approach to the temporal characterisation of materials. The article discusses the challenges posed by manual methods involving established laboratory equipment and presents an automated hyperspectral characterisation station. This station integrates robot-aided hyperspectral imaging (HSI), complex material characterisation modelling, and automated data analysis, offering a non-destructive and comprehensive approach. This work explains how the proposed assembly can automatically measure the half-life of biodegradable polymers with higher throughput and accuracy than manual methods. The investigation explores the effect of pH, number of average molecular weight (Mn), end groups, and blends on the degradation rate of polylactic acid (PLA). The novel contributions of the paper lie in introducing an adaptable classification station for characterisation and presenting an innovative methodology for polymer degradation rate measurements. The proposed system holds promise for expediting the development of high-throughput screening and characterisation methods within advanced material and chemistry laboratories. Note to Practitioners—The characterisation and classification of materials hold significant importance within the realms of materials science, chemistry, manufacturing, and the circular economy. We introduce an innovative approach by employing robotics to automate hyperspectral imaging and processing to accelerate and improve material characterisation within a laboratory environment. This methodology facilitates the time-dependent characterisation of materials. This automated system encompasses sample manipulation and handling for hyperspectral scanning, accompanied by automated image and data processing procedures to minimise manual interventions effectively. The developed system can be seamlessly integrated into lab settings, offering objective, accurate, and automated measurement of biodegradable polymer degradation rates.

Keywords: Computer Vision for Manufacturing, Computer Vision for Transportation
Abstract: Automated crack recognition has achieved remarkable progress in the past decades as a critical task in structure health monitoring, to ensure safety and durability in many industrial scenarios. However, imbalanced crack recognition remains challenging due to the scarcity of crack samples and the consequential limited diversity. To resolve this, Artificial Intelligence Generated Content (AIGC) has been gradually adopted to generate synthetic data and reduce reliance on large amounts of labeled crack samples. This paper assumes that a crack sample in the feature space can be regarded as a combination of crack and background semantics. Then, the decompose-compose feature augmentation framework (DeCo) is proposed to perform crack data synthesis in the feature space by randomly composing crack and background semantic-relevant features. Specifically, the contrastive learning-based decomposing loss is proposed to enforce two encoders to separately learn crack and background semantics from crack samples with a theoretical guarantee. After that, an effective cross-instance feature union strategy is proposed to synthesize diverse crack samples by composing the crack-relevant features from a crack sample and background-relevant features across other training samples. Experimental results show that DeCo performs favorably against state-of-the-art competitors in imbalanced crack recognition tasks.

Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems
Abstract: In this paper, we propose a novel unsupervised liver deformation correction method, Learning Coherent Point Drift Network (LCNet), for image-guided liver surgery (IGLS). In order to address the complete to partial registration problem, we first crop the preoperative liver point cloud to obtain the overlapping regions with the intraoperative point cloud. The cropped preoperative point set is then fed into the subsequent registration network for alignment. Secondly, we establish reliable correspondences between two point sets using the optimal transport (OT) module by leveraging both original points and learnt features, which are robust to the rigid transformation. Finally, we compute the displacements by solving the involved matrix equation in the Transformation module, where the point localisation noise is explicitly considered. In addition, we present three variants of the proposed approach, i.e., LCNet, LCNet-ED and LCNet-WD, where LCNet outperforms the other two, which demonstrates the superiority of the Chamfer loss. We have extensively evaluated LCNet on the MedShapeNet dataset consisting of 615 different liver shapes of real patients, and 3Dircadb consisting 20 another liver models of real patients. For example, when the overlap ratio is 25%, the deformation magnitude is 8 mm, the maximum noise magnitude is 2 mm and the rotation angle lies in the range of [-45^{circ},45^{circ}], LCNet achieves a root-mean-square error (RMSE) value being 3.21 mm on MedShapeNet, outperforming Lepard and RoITr, which are 5.41 mm (p<0.001) and 4.90 mm (p<0.001) respectively. Extensive experimental results, under different deformation and noise levels, demonstrate that LCNet exhibits significant improvements over existing state-of-the-art registration methods and holds significant application potential in IGLS.

Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems, Vision-Based Navigation
Abstract: Retinal vein cannulation (RVC) is a minimally invasive microsurgical procedure for treating retinal vein occlusion (RVO), a leading cause of vision impairment. However, the small size and fragility of retinal veins, coupled with the need for high-precision, tremor-free needle manipulation, pose significant challenges for surgeons. These limitations underscore the necessity for robotic assistance to enhance surgical accuracy, stability, and reproducibility. This study presents an automated robotic system with a top-down microscope and cross-sectional B-scan optical coherence tomography (OCT) imaging for precise depth sensing. Deep learning-based models enable real-time needle navigation, contact detection, and vein puncture recognition, using a chicken embryo model, a well-established biological surrogate for human retinal veins. The system autonomously detects needle position and puncture events with 85% accuracy. The experiments demonstrate notable reductions in navigation and puncture times compared to manual methods. Our results demonstrate the potential of integrating advanced imaging and deep learning to automate microsurgical tasks, providing a pathway for safer and more reliable RVC procedures with enhanced precision and reproducibility.

Keywords: Computer Vision for Medical Robotics, Surgical Robotics: Laparoscopy
Abstract: Autonomous suturing is a critical challenge in robot-assisted surgery, where accurate segmentation and pose estimation of suturing threads are essential prerequisites. However, suturing threads are easily occluded by moving instruments and embedded in deformable tissues which make the task much more challenging. To address this, we propose a coarse-to-fine network for detailed segmentation and pose estimation of suturing threads. The coarse stage aims to capture global thread structure, while the fine stage refines the detailed structure through error residual correction. A spatial context fusion module is incorporated to improve the perception of occluded regions, and weighted balanced cross entropy loss as well as hard sample mining strategy is implemented to enhance small target segmentation performance. To deal with severe occlusions, topological constraints are utilized to effectively identify and reconstruct invisible thread segments. Experiments have been conducted on three datasets collected from different surgical scenes including phantom, endoscopy, and microsurgery. Both quantitative and qualitative results have demonstrated that our proposed framework outperforms baseline methods on segmentation and pose estimation of suturing threads, particularly in detecting occluded threads. Our proposed framework generalizes well across different surgical scenarios, showing its potential for automatic suturing.

Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems
Abstract: 3D point cloud registration is an essential problem in computer vision, robotics, surgical navigation and augmented reality. Accurate registration of partially overlapped intraoperative point clouds (e.g., femoral reconstruction) remains critical yet challenging in orthopedic navigation due to incomplete overlap and dynamic noise. In this study, we propose a partial-to-partial point cloud registration framework based on directional spatial consistency. First, we extract overlapped areas from partially overlapping point clouds and leverage the point registration graph matching module to calculate the hard point matching matrix. Second, we sample nodes from the source point cloud and generate translation-invariant edge vectors (direction/scale-preserving) via their k-nearest neighbors, guided by predicted point correspondences. This bypasses translation ambiguities by encoding spatial consistency through edges, reducing pose estimation to 3DoF alignment (rotation). The loss explicitly couples point-level matches with edge-level geometric constraints for dual optimization. Building upon this framework, we extract reliable overlapping edge representations and prune their similarity matrix by thresholding low-confidence scores, effectively suppressing spurious matches. The proposed edge-aware matching mechanism further exploits the translation invariance of local structures to refine point correspondences with enhanced accuracy. Finally, we introduce a bidirectional registration mechanism to reinforce optimization stability, achieving state-of-the-art performance across benchmarks. Extensive experiments on ModelNet40, ShapeNet, and MedShapeNet validate our method under diverse scenarios: partial-to-partial, unseen categories, partial-to-full, and cross-dataset generalization, surpassing existing methods in registration accuracy. This code will be publicly released https://github.com/pidan0824/DSCGM.

Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems
Abstract: In this paper, we present a novel curve-to-surface registration method, termed Bi-directional Hybrid Mixture Model Registration based on Dual-constrained Tangent and Normal Vectors (BiHMM-DTN). While hybrid registration models incorporating tangent and normal vectors (HMM-TN) demonstrate success, their geometric constraints prove inadequate or inappropriate for sparse intraoperative point sets, frequently yielding suboptimal optimization outcomes. By critically revisiting the geometric constraints of HMM-TN, we propose a dual-constraints-based hybrid mixture model registration framework with enhanced intraoperative point cloud acquisition protocols. To deal with noise and outliers in preoperative and intraoperative point sets—caused by reconstruction inaccuracies and tracking errors respectively—our approach employs a bi-directional registration mechanism for curve-to-surface registration. We provide rigorous proofs validating the geometric completeness of the dual constraints within this mechanism. The BiHMM-DTN framework is formulated as a maximum likelihood estimation (MLE) problem and optimized using an expectation-maximization (EM) algorithm. Futhermore, to enhance convergence stability and accelerate optimization, the rotation matrix is updated iteratively through successive incremental steps. Extensive experiments on human femur and hip models demonstrate that our method outperforms state-of-the-art approaches, including both traditional optimization and deep learning methods, under various noise and outlier conditions. Furthermore, real-world phantom experiments highlight the potential clinical value of our method for surgical navigation applications. The codes and data are available at url{https://github.com/sam-zyzhang/BiHMM-DTN.git}.

Keywords: Computer Vision for Medical Robotics
Abstract: Endoscopic reconstruction plays a crucial role in surgical robotics. The dynamic lighting conditions and integrated camera-light source in endoscopic scenes create a distinct reconstruction challenge: shape ambiguity. To mitigate this, we propose a Gaussian Splatting (GS) based framework for endoscopic scene reconstruction, enhanced with reflectance regularization. We embed every 3D Gaussian point with physical reflective attributes and combine this representation with a physically based inverse rendering framework. By jointly training 3DGS for view synthesis with this reflectance regularization, we are able to attain high-quality geometry without changing the volume rendering pipeline. Our experiments demonstrate the superiority in both geometry representation and rendering performance compared to existing GS approaches, making it a practical solution for endoscopic applications. Project is available at: https://med-air.github.io/GSR2.

Keywords: Reinforcement Learning, Collision Avoidance
Abstract: Local optima issues challenge mobile robot mapless navigation with the dilemma of avoiding collisions and approaching the target. Planning-based methods rely on environmental models and manual strategies for updating local paths to guide the robot. In contrast, learning-based methods are capable of processing original sensor data to navigate the robot in real time but struggle with local optima problems. To address this, we designed reward rules that punishes the robot for revisiting passed areas that may traps the robot, and reward it for exploring local areas in diverse ways and escaping from local optima areas. Then, we improved the Soft Actor-Critic (SAC) algorithm by making its temperature parameter adaptive to the current training status, and memorize them in experiences for strategy updating, bringing additional exploratory behaviors and necessary stability into the training. Finally, with the assistance of auxiliary networks, the robot learns to handle various navigation tasks with local optima risks. Simulations demonstrate the advantages of our method in terms of both success rate and path efficiency compared to several existing methods. Experiments verified the proposed method in real-world scenarios.

Keywords: Reinforcement Learning, Sensor-based Control, AI-Based Methods
Abstract: This study develops a novel end-to-end formation strategy for leader-follower formation control of mobile robots that uses onboard LiDAR sensors in non-communication environments. The main contributions of this paper are twofold: Firstly, we propose a point cloud-based LiDAR servoing control method (PCLS) aimed at ensuring mobile robots achieve the predefined formation performance without direct communication. Secondly, an innovative two-stage Soft Actor-Critic (TSSAC) algorithm is presented, specifically designed for end-to-end training of PCLS. This algorithm skillfully combines the strengths of a distance-based agent (serving as a ``teacher") and a point cloud-based agent (serving as a ``student"), effectively addressing the issues of slow convergence and insufficient generalization in deep reinforcement learning methods that use high-dimensional features (such as point clouds, images) as inputs. Furthermore, as part of our method, we designed a novel reward function and normalized the point cloud inputs to provide consistent incentives for the agent across diverse formation tasks, thereby facilitating better learning and adaptation to formation tasks in different environments. Finally, through extensive experiments conducted in the Gazebo simulator and real-world environments, we confirmed the effectiveness of the proposed method. Compared to other formation control strategies, our approach relies solely on onboard LiDAR sensors, without the need for additional communication devices, while ensuring excellent transient and steady-state performance. The video of the test in the real-world environment can be found at https://youtu.be/jhggsLUczPA

Keywords: Reinforcement Learning
Abstract: Offline reinforcement learning (RL) presents distinct challenges as it relies solely on observational data. A central concern in this context is ensuring the safety of the learned policy by quantifying uncertainties associated with various actions and environmental stochasticity. Traditional approaches primarily emphasize mitigating epistemic uncertainty by learning risk-averse policies, often overlooking environmental stochasticity. In this study, we propose an uncertainty-aware distributional offline RL method to simultaneously address both epistemic uncertainty and environmental stochasticity. We propose a model-free offline RL algorithm capable of learning risk-averse policies and characterizing the entire distribution of discounted cumulative rewards, as opposed to merely maximizing the expected value of accumulated discounted returns. Our method is rigorously evaluated through comprehensive experiments in both risk-sensitive and risk-neutral benchmarks, demonstrating its superior performance.

Keywords: Reinforcement Learning, Cooperating Robots, Representation Learning
Abstract: In multi-robot systems, capturing the complex and dynamic interaction relationships is essential for enhancing autonomous collaboration. However, existing learning-based approaches usually overlook the understanding of these relationships, leading to reliability issues and hindering their application to real-world scenarios. This paper proposes a novel approach called Interpretable Heuristic Graph Structure Learning (IHGSL) to better comprehend the complex collaborative relationships in multi-robot systems. We first construct a predicate space to define diverse predicates that express fundamental relationships. Then we employ the variational information bottleneck technique to acquire a latent representation of the current observation by aligning it with the historical trajectory. On this basis, the predicates that the robot should currently focus on the most are learned, and some interaction relationships are established accordingly. Thereby an interpretable relationship graph is generated heuristically to guide the achievement of multi-robot autonomous collaborative decision-making. Through experimental evaluation, we demonstrate the process of relationship inference, thus validating the interpretability of IHGSL. Compared with existing methods, IHGSL also achieves superior collaboration performance, which highlights the effectiveness of the learned heuristic graph structure.

Keywords: Reinforcement Learning, Imitation Learning, Dexterous Manipulation
Abstract: We present Asymmetric Critic Guided Distillation (ACGD) a framework for learning multi-task dexterous manipulation policies that can manipulate articulated objects using images as input. ACGD is a scalable student-teacher distillation approach that utilizes behavior cloning to distill multiple expert policies into a single vision-based, multi-task student policy for dexterous manipulation. The expert policies are trained with traditional RL techniques with access to privileged state information of both the robot and the manipulated object, while the distilled student policy operates under realistic sensory constraints, specifically using only camera images and robot proprioception. During distillation, we use an expert-critic that provides action labels and value estimates to refine the student's action sampling through a dual IL/RL objective. In the multi-task setting, we achieve this through an aggregate critic for different single-task experts. Our approach exhibits strong performance compared to a number of state-of-the-art imitation learning (IL) and reinforcement learning (RL) baselines. We evaluate across a variety of multi-task dexterous manipulation benchmarks including bimanual manipulation, single-hand object articulation tasks, and a tendon-actuated hand and achieves state-of-the-art performance with 10-15% improvement over baseline algorithms. Visit our website (https://critic-guided-distillation.github.io) for more details.

Keywords: Reinforcement Learning, Machine Learning for Robot Control, Continual Learning
Abstract: Curriculum learning has emerged as a promising approach for training complex robotics tasks, yet current applications predominantly rely on manually designed curricula, which demand significant engineering effort and can suffer from subjective and suboptimal human design choices. While automated curriculum learning has shown success in simple domains like grid worlds and games where task distributions can be easily specified, robotics tasks present unique challenges: they require handling complex task spaces while maintaining relevance to target domain distributions that are only partially known through limited samples. To this end, we propose Grounded Adaptive Curriculum Learning, a framework specifically designed for robotics curriculum learning with three key innovations: (1) a task representation that consistently handles complex robot task design, (2) an active performance tracking mechanism that allows adaptive curriculum generation appropriate for the robot's current capabilities, and (3) a grounding approach that maintains target domain relevance through alternating sampling between reference and synthetic tasks. We validate GACL on wheeled navigation in constrained environments and quadruped locomotion in challenging 3D confined spaces, achieving 6.8% and 6.1% higher success rates, respectively, than state-of-the-art methods in each domain.

Keywords: Reinforcement Learning, Machine Learning for Robot Control, Model Learning for Control
Abstract: Learning a controller directly on the robot requires extreme sample efficiency. Model-based reinforcement learning (RL) methods are the most sample efficient, but they often suffer from a too long inference time to meet the robot control frequency requirements. In this paper, we address the sample efficiency and inference time challenges with two contributions. First, we define a general framework to deal with inference delays where the slow inference robot controller provides a sequence of actions to feed the control-hungry robotic platform without execution gaps. Then, we compare several RL algorithms in the light of this framework and propose RT-HCP, an algorithm that offers an excellent trade-off between performance, sample efficiency and inference time. We validate the superiority of RT-HCP with experiments where we learn a controller directly on a simple but high frequency FURUTA pendulum platform.

Keywords: Robot Safety, Reinforcement Learning, Continual Learning
Abstract: With the increasing presence of automated vehicles on open roads under driver supervision, disengagement cases are becoming more prevalent. While some data-driven planning systems attempt to directly utilize these disengagement cases for policy improvement, the inherent scarcity of disengagement data (often occurring as a single instance) restricts training effectiveness. Furthermore, some disengagement data should be excluded since the disengagement may not always come from the failure of driving policies, e.g. the driver may casually intervene for a while. To this end, this work proposes disengagement-reason-augmented reinforcement learning (DRARL), which enhances driving policy improvement process according to the reason of disengagement cases. Specifically, the reason of disengagement is identified by an out-of-distribution (OOD) state estimation model. When the reason doesn’t exist, the case will be identified as a casual disengagement case, which doesn’t require additional policy adjustment. Otherwise, the policy can be updated under a reason-augmented imagination environment, improving the policy performance of disengagement cases with similar reasons. The method is evaluated using real-world disengagement cases collected by autonomous driving robotaxi. Experimental results demonstrate that the method accurately identifies policy-related disengagement reasons, allowing the agent to handle both original and semantically similar cases through reason-augmented training. Furthermore, the approach prevents the agent from becoming overly conservative after policy adjustments. Overall, this work provides an efficient way to improve driving policy performance with disengagement cases.

Keywords: RGB-D Perception, Recognition
Abstract: 3D visual grounding is pivotal for enabling intelligent agents to find the target object in the 3D scene given the linguistic descriptions. However, contemporary methods are typically hindered by the scarcity of large-scale 3D datasets with fine-grained annotations and the complexity of modeling spatial relationships in the 3D space. Inspired by the exceptional performance of Vision Foundation Models (VFMs) and Vision Language Models (VLMs), we propose a novel label-free 3D visual grounding method, termed LF-3DVG , that minimizes the heavy reliance on fine-grained annotations and leverages the off-the-shelf vision foundation models for zero-shot 3D visual grounding. Our LF-3DVG is comprised of two main components, i.e., VFM-guided 3D Object Detection and VLM- based 3D Visual Grounding. Specifically, we first utilize SAM3D to generate high-quality instance masks for the objects in the 3D scene. Since SAM3D cannot provide categorical information, we further employ Semantic-SAM to assign class labels for the detected masks. As to the VLM-based 3D Visual Grounding, we first feed multi-view images and textual descriptions to the VLM for 2D visual grounding. To lift the 2D predictions to the 3D space, we design the 2D-3D object association module to effectively match the 2D detection results with the 3D boxes produced by the 3D detector, yielding the ultimate 3D visual grounding results. Through extensive experiments on the ScanRefer and Sr3D/Nr3D benchmarks, we demonstrate that our method consistently outperforms previous approaches. Our algorithm can also boost the performance of 3D visual grounding when given labeled training samples and can be seamlessly integrated into contemporary 3D visual grounding models.

Keywords: RGB-D Perception, Computer Vision for Automation, Deep Learning Methods
Abstract: Monocular depth estimation (MDE) is crucial for various computer vision applications, but existing methods often struggle to balance inference speed and accuracy when processing large-region visual information. This paper introduces LR^2Depth, a novel MDE method that addresses this challenge by utilizing large-kernel convolution on low-resolution feature maps for efficient large-region feature aggregation. Our approach leverages the fact that each pixel on low-resolution feature maps corresponds to a larger region of the original image, allowing for fast and accurate depth predictions at a lower inference cost. Extensive experiments on NYU-Depth-V2, KITTI, and SUN RGB-D datasets demonstrate that LR^2Depth not only achieves state-of-the-art performance but also operates approximately twice as fast as previous MDE methods. Notably, at the time of submission, LR^2Depth secured the top-1 position on the KITTI depth prediction online benchmark.

Keywords: RGB-D Perception, Computer Vision for Transportation
Abstract: Depth estimation has been widely studied and serves as the fundamental step of 3D perception for robotics and autonomous driving. Though significant progress has been made in monocular depth estimation in the past decades, these attempts are mainly conducted on the KITTI benchmark with only front-view cameras, which ignores the correlations across surround-view cameras. In this paper, we propose an Adjacent-View Transformer for Supervised Surround-view Depth estimation to jointly predict the depth maps across multiple surrounding cameras. Specifically, we employ a global-to-local feature extraction module that combines CNN with transformer layers for enriched representations. Further, the adjacent-view attention mechanism is proposed to enable the intra-view and inter-view feature propagation. The former is achieved by the self-attention module within each view, while the latter is realized by the adjacent attention module, which computes the attention across multi-cameras to exchange the multi-scale representations across surround-view feature maps. In addition, AVT-SSDepth has strong cross-dataset generaliza- tion. Extensive experiments show that our method achieves superior performance on both DDAD and nuScenes datasets.

Keywords: RGB-D Perception, Semantic Scene Understanding, Range Sensing
Abstract: 实时开放词汇场景理解对于视觉语言导航、具体智能和增强现实等应用中的高效 3D 感知至关重要。但是，现有方法存在实例分割不精确、静态语义更新和复杂查询处理受限的问题。为了解决这些问题，我们提出了 OpenFusion++，这是一种基于 TSDF 的实时 3D 语义几何重建系统。我们的方法通过融合来自基础模型的置信度图来优化 3D 点云，通过基于实例区域的自适应缓存动态更新全局语义标签，并采用双路径编码框架将对象属性与环境上下文集成以实现精确的查询响应。在 ICL 、 Replica 、 ScanNet 和 ScanNet++ 数据集上的实验表明，OpenFusion++ 在语义准确性和查询响应性方面都明显优于基线。

Keywords: RGB-D Perception, Grasping, Computer Vision for Automation
Abstract: Accurate 6D object pose estimation is a prerequisite for successfully completing robotic prehensile and nonprehensile manipulation tasks. At present, 6D pose estimation for robotic manipulation generally relies on depth sensors based on, e.g., structured light, time-of-flight, and stereo-vision, which can be expensive, produce noisy output (as compared with RGB cameras), and fail to handle transparent objects. On the other hand, state-of-the-art monocular depth estimation models (MDEMs) provide only affine-invariant depths up to an unknown scale and shift. Metric MDEMs achieve some successful zero-shot results on public datasets, but fail to generalize. We propose a novel framework, monocular one-shot metric-depth alignment, MOMA, to recover metric depth from a single RGB image, through a one-shot adaptation building on MDEM techniques. MOMA performs scale-rotation-shift alignments during camera calibration, guided by sparse ground-truth depth points, enabling accurate depth estimation without additional data collection or model retraining on the testing setup. MOMA supports fine-tuning the MDEM on transparent objects, demonstrating strong generalization capabilities. Real- world experiments on tabletop 2-finger grasping and suction- based bin-picking applications show MOMA achieves high success rates in diverse tasks, confirming its effectiveness.

Keywords: RGB-D Perception, Object Detection, Segmentation and Categorization
Abstract: Detection of deformable linear objects (DLOs) in three-dimensional space is essential for robotic manipulation of DLOs. However, their complex deformations and high degrees of freedom make perception highly susceptible to occlusions, noise, and data missing. To address these challenges, we propose a deep learning-based method that leverages the topological properties of DLOs to robustly detect keypoints from incomplete point clouds while preserving the topological order of keypoints. Our approach initializes a sequence of keypoints that adheres to the topological structure of DLOs. Then, these ordered keypoints are refined through bidirectional sequence learning. Simulation results demonstrate that our method generates accurate, uniform, and smooth keypoint sequences under varying levels of occlusion. Compared to existing baselines, our approach achieves superior performance. Real-world experiments further validate the generalization capability of our method in unseen and challenging scenarios involving occlusion and self-occlusion while maintaining real-time performance.

Keywords: RGB-D Perception, Deep Learning Methods, Deep Learning for Visual Perception
Abstract: Depth map enhancement using paired high-resolution RGB images offers a cost-effective solution for improving low-resolution depth data from lightweight ToF sensors. Nevertheless, naively adopting a depth estimation pipeline to fuse the two modalities requires groundtruth depth maps for supervision. To address this, we propose a self-supervised learning framework, SelfToF, which generates detailed and scale-aware depth maps. Starting from an image-based self-supervised depth estimation pipeline, we add low-resolution depth as inputs, design a new depth consistency loss, propose a scale-recovery module, and finally obtain a large performance boost. Furthermore, since the ToF signal sparsity varies in real-world applications, we upgrade SelfToF to SelfToF* with submanifold convolution and guided feature fusion. Consequently, SelfToF* maintain robust performance across varying sparsity levels in ToF data. Overall, our proposed method is both efficient and effective, as verified by extensive experiments on the NYU and ScanNet datasets. The code is available at https://github.com/denyingmxd/selftof.

Keywords: Deep Learning Methods, Deep Learning for Visual Perception, AI-Based Methods
Abstract: Accurate and robust medical image segmentation is fundamental and crucial for enhancing the autonomy of computer-aided diagnosis and intervention systems. Medical data collection normally involves different scanners, protocols, and populations, making domain adaptation (DA) a highly demanding research field to alleviate model degradation in the deployment site. To preserve the model performance across multiple testing domains, this work proposes the Curriculum-based Augmented Fourier Domain Adaptation (Curri-AFDA) for robust medical image segmentation. In particular, our curriculum learning strategy is based on the causal relationship of a model under different levels of data shift in the deployment phase, where the higher the shift is, the harder to recognize the variance. Considering this, we progressively introduce more amplitude information from the target domain to the source domain in the frequency space during the curriculum-style training to smoothly schedule the semantic knowledge transfer in an easier-to-harder manner. Besides, we incorporate the training-time chained augmentation mixing to help expand the data distributions while preserving the domain-invariant semantics, which is beneficial for the acquired model to be more robust and generalize better to unseen domains. Extensive experiments on two segmentation tasks of Retina and Nuclei collected from multiple sites and scanners suggest that our proposed method yields superior adaptation and generalization performance. Meanwhile, our approach proves to be more robust under various corruption types and increasing severity levels. In addition, we show that our method is also beneficial in the domain-adaptive classification task with skin lesion datasets.

Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Visual Learning
Abstract: Despite significant advances in self-supervised monocular depth estimation methods, achieving temporally consistent and accurate depth maps from frame sequences remains a formidable challenge. Existing approaches often estimate depth maps for individual frames in isolation, neglecting the rich geometric and temporal coherence present across frames. Consequently, this oversight leads to temporally inconsistent outputs, resulting in noticeable temporal flickering artifacts. In response, this paper presents TCNet, a Temporal Consistent Network for self-supervised monocular depth estimation. Specifically, we propose an Inter-frame Temporal Fusion (ITF) module to emphasize the influence of preceding images on the depth estimation of the current frame. The Temporal Consistency Loss (TCL) is proposed to leverage the temporal constraints between the depth maps of adjacent frames. Besides, TCNet can also be applied to both single-frame and multi-frame scenarios during inference. Experimental evaluations on the KITTI dataset demonstrate that our method surpasses state-of-the-art depth estimation methods in accuracy and temporal consistency. Our code will be made public.

Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Vision-Based Navigation
Abstract: Accurate semantic segmentation of both tactile paving and the obstacle is crucial for the safe mobility of visually impaired individuals. However, existing methods face two major challenges: (i) discontinuous segmentation fragments; (ii) Inaccurate obstacle recognition. To address challenge (i), we propose incorporating appearance priors of complete tactile pavings to prevent the model from directly learning irregular ground truth masks. To tackle challenge (ii), we propose introducing cross-modal semantic priors to complement the semantic information of obstacles. We implemented these strategies in proposed Dual Prior knowledge induced tactile paving and obstacle joint Segmentation Network (DPSN). Based on bilateral network architecture, DPSN merges obstacle category masks into tactile paving categories, constructing a complete tactile paving mask. Utilizing the complete mask, DPSN transfer appearance prior knowledge to detail features from boundary and structural perspectives. Concurrently, DPSN leverages the CLIP Text Encoder to guide visual feature decoding by attention mechanisms, transferring rich cross-modal semantic prior knowledge to the visual feature maps. Furthermore, we propose the TPO-Dataset, the first dataset for joint tactile paving and obstacle segmentation acquired from actual scenes. Experiments demonstrate that DPSN achieves state-of-the-art results on the TPO-Dataset, with relative gains of 27.16% in obstacle IoU and 30.53% in accuracy metrics compared to baseline methods. Notably, DPSN achieves real-time performance at 88.25 FPS on the maximum scale of 2048×512 resolution.

Keywords: Deep Learning for Visual Perception, RGB-D Perception, Deep Learning Methods
Abstract: Transparent objects are common in industrial automation and daily life. However, accurate visual perception of these objects remains challenging due to their reflective and refractive properties. Most previous studies fail to capture contextual information and typically rely on regression-based methods at the decoder stage, suffering from overfitting and unsatisfactory object details. To overcome these limitations, we present a novel depth completion framework for transparent objects with dif-fusion denoising approach (DCT-Diffusion). First, we adopt a transformer-based encoder to globally learn the depth rela-tionships from different parts of the input by modeling long-distance dependencies. Then, we propose to introduce the diffusion model to generate refined depth maps from random depth distribution. Through iterative refinement, our model can progressively enhance depth map details and achieves fi-ne-grained performance. Lastly, a conditioned fusion module is developed, which utilizes encoder features as visual conditions and fuses them with the denoising block at each step using augmented attention. Extensive comparative studies and cross-domain experiments prove that the DCT-Diffusion out-performs previous methods and significantly improves the robustness and generalization ability. Moreover, visualization results further illustrate that our method can generate depth maps with more complete geometry and clearer boundaries, achieving satisfactory results.

Keywords: Deep Learning for Visual Perception, Perception for Grasping and Manipulation
Abstract: General-purpose object placement is a fundamental capability of an intelligent generalist robot: being capable of rearranging objects following precise human instructions even in novel environments. This work is dedicated to achieving general-purpose object placement with "something something" instructions. Specifically, we break the entire process down into three parts, including object localization, goal imagination and robot control, and propose a method named SPORT. SPORT leverages a pre-trained large vision model for broad semantic reasoning about objects, and learns a diffusion-based pose estimator to ensure physically-realistic results in 3D space. Only object types (movable or reference) are communicated between these two parts, which brings two benefits. One is that we can fully leverage the powerful ability of open-set object recognition and localization since no specific fine-tuning is needed for the robotic scenario. Moreover, the diffusion-based estimator only need to "imagine" the object poses after the placement, while no necessity for their semantic information. Thus the training burden is greatly reduced and no massive training is required. The training data for the goal pose estimation is collected in simulation and annotated by using GPT-4. Experimental results demonstrate the effectiveness of our approach. SPORT can not only generate promising 3D goal poses for unseen simulated objects, but also be seamlessly applied to real-world settings.

Keywords: Deep Learning for Visual Perception, Computer Vision for Automation, Deep Learning Methods
Abstract: Estimating the 3D pose of a drone is important for anti-drone systems, but existing methods struggle with the unique challenges of drone keypoint detection. Drone propellers serve as keypoints but are difficult to detect due to their high visual similarity and diversity of poses. To address these challenges, we propose DroneKey, a framework that combines a 2D keypoint detector and a 3D pose estimator specifically designed for drones. In the keypoint detection stage, we extract two key-representations (intermediate and compact) from each transformer encoder layer and optimally combine them using a gated sum. We also introduce a pose-adaptive Mahalanobis distance in the loss function to ensure stable keypoint predictions across extreme poses. We built new datasets of drone 2D keypoints and 3D pose to train and evaluate our method, which have been publicly released. Experiments show that our method achieves an AP of 99.68% (OKS) in keypoint detection, outperforming existing methods. Ablation studies confirm that the pose-adaptive Mahalanobis loss function improves keypoint prediction stability and accuracy. Additionally, improvements in the encoder design enable real-time processing at 44 FPS. For 3D pose estimation, our method achieved an MAE-angle of 10.62°, an RMSE of 0.221m, and an MAE-absolute of 0.076m, demonstrating high accuracy and reliability. The code and dataset are available at https://github.com/kkanuseobin/DroneKey.

Keywords: Deep Learning for Visual Perception, Computer Vision for Transportation, Computer Vision for Automation
Abstract: Abstract— Semantic Scene Completion (SSC) constitutes a pivotal element in autonomous driving perception systems, tasked with inferring the 3D semantic occupancy of a scene from sensory data. To improve accuracy, prior research has implemented various computationally demanding and memory- intensive 3D operations, imposing significant computational requirements on the platform during training and testing. This paper proposes L2COcc, a lightweight camera-centric SSC framework that also accommodates LiDAR inputs. With our proposed efficient voxel transformer (EVT) and three types of cross-modal knowledge modules (FSD, TPVD, PAD), our method substantially reduce computational burden while main- taining high accuracy. The experimental evaluations demon- strate that our proposed method surpasses the current state-of- the-art vision-based SSC methods regarding accuracy on both the SemanticKITTI and SSCBench-KITTI-360 benchmarks, respectively. Additionally, our method is more lightweight, ex- hibiting a reduction in both memory consumption and inference time by over 25%. Code is available at the project page: https://studyingfufu.github.io/L2COcc/.

Keywords: Deep Learning for Visual Perception, Machine Learning for Robot Control
Abstract: While human vision inherently achieves robust cross-domain depth estimation through binocular coordination, robotic systems employing stereo matching still confront significant challenges in maintaining robustness across domains when performing real-time environmental depth perception. Furthermore, most stereo matching methods struggle with challenging regions such as object boundaries and non-overlapping areas on the left side of the left image, resulting in disparity maps that are relatively indistinct and lacking fine details. In this paper, we propose Learning More in Challenging Areas (LMC) to alleviate this problem, which enhances the domain generalization of the model through targeted training on challenging regions. LMC is a simple yet effective data-driven training framework primarily based on self-distillation. Specifically, 1) We pre-train models on a high-frequency dataset to improve perception ability on object boundaries; 2) We develop a self-distillation training strategy to benefit learning in non-overlapping areas on the left side of the left image; 3) We design an adaptive difficult area mask to balance the loss weight on other undefined challenging regions. Under our proposed training framework, GwcNet achieves 33% and 23% performance improvements in autonomous driving benchmarks KITTI 2012 and KITTI 2015 respectively, while preserving real-time inference efficiency without computational overhead.

Keywords: Deep Learning Methods, Visual Tracking, AI-Based Methods
Abstract: The 3D trajectory of a shuttlecock required for a badminton rally robot for human-robot competition demands real-time performance with high accuracy. However, the fast flight speed of the shuttlecock, along with various visual effects, and its tendency to blend with environmental elements, such as court lines and lighting, present challenges for rapid and accurate 2D detection. In this paper, we first propose the YO-CSA detection network, which optimizes and reconfigures the YOLOv8s model's backbone, neck, and head by incorporating contextual and spatial attention mechanisms to enhance model's ability in extracting and integrating both global and local features. Next, we integrate three major sub-tasks—detection, prediction, and compensation—into a real-time 3D shuttlecock trajectory detection system. Specifically, our system maps the 2D coordinate sequence extracted by YO-CSA into 3D space using stereo vision, then predicts the future 3D coordinates based on historical information, and re-projects them onto the left and right views to update the position constraints for 2D detection. Additionally, our system includes a compensation module to fill in missing intermediate frames, ensuring a more complete trajectory. We conduct extensive experiments on our own dataset to evaluate both YO-CSA's performance and system effectiveness. Experimental results show that YO-CSA achieves a high accuracy of 90.43% mAP@0.75, surpassing both YOLOv8s and YOLO11s. Our system performs excellently, maintaining a speed of over 130 fps across 12 test sequences.

Keywords: Deep Learning Methods, Computer Vision for Automation, Computer Vision for Manufacturing
Abstract: While most current RGB-D-based category-level object pose estimation methods achieve strong performance, they face significant challenges in scenes lacking depth information. In this paper, we propose a novel category-level object pose estimation approach that relies solely on RGB images. This method enables accurate pose estimation in real-world scenarios without the need for depth data. Specifically, we design a transformer-based neural network for category-level object pose estimation, where the transformer is employed to predict and fuse the geometric features of the target object. To ensure that these predicted geometric features faithfully capture the object's geometry, we introduce a geometric feature-guided algorithm, which enhances the network's ability to effectively represent the object's geometric information. Finally, we utilize the RANSAC-PnP algorithm to compute the object's pose, addressing the challenges associated with variable object scales in pose estimation. Experimental results on benchmark datasets demonstrate that our approach is not only highly efficient but also achieves superior accuracy compared to previous RGB-based methods. These promising results offer a new perspective for advancing category-level object pose estimation using RGB images.

Keywords: Deep Learning Methods, Multi-Robot Systems, Big Data in Robotics and Automation
Abstract: Multi-Agent Systems (MAS) excel at accomplishing complex objectives through the collaborative efforts of individual agents. Among the methodologies employed in MAS, Multi-Agent Reinforcement Learning (MARL) stands out as one of the most efficacious algorithms. However, when confronted with the complex objective of Formation Control with Collision Avoidance (FCCA): designing an effective reward function that facilitates swift convergence of the policy network to an optimal solution. In this paper, we introduce a novel framework that aims to overcome this challenge. By giving large language models (LLMs) on the prioritization of tasks and the observable information available to each agent,our framework generates reward functions that can be dynamically adjusted online based on evaluation outcomes by employing more advanced evaluation metrics rather than the rewards themselves. This mechanism enables the MAS to simultaneously achieve formation control and obstacle avoidance in dynamic environments with enhanced efficiency, requiring fewer iterations to reach superior performance levels. Our empirical studies, conducted in both simulation and real-world settings, validate the practicality and effectiveness of our proposed approach.

Keywords: Deep Learning Methods, Collision Avoidance, Learning from Experience
Abstract: The data-driven paradigm has shown great potential in solving many decision-making tasks. In the robot navigation realm, it also sparked a new trend. People believe powerful data-driven methods can learn efficient and general navigation policies from a vast offline dataset. However, robot navigation tasks differ from common planning tasks and present unique challenges. It often involves multi-objective optimization to meet arbitrary and ever-changing human preferences. It should also overcome the short-sighted problem to obtain globally optimal performance. Furthermore, high planning frequency is needed to address real-time demands. These factors obstruct the application of data-driven methods in robot navigation. To address these challenges, we integrate one of the most powerful data-driven methods, the diffusion model, into robot navigation. Our proposed approach, NaviDiffuser, utilizes a novel classification label to guide the diffusion model in capturing the complex connections between navigation and human preferences. Its Transformer network backbone outputs action sequences to alleviate short-sightedness. It also includes special distillation skills to boost the planning speed and quality. We conduct experiments in both simulated and real-world scenarios to evaluate our approach. In these experiments, NaviDiffuser not only demonstrates an extremely high arrival rate but also adjusts its navigation policy to align with different human preferences.

Keywords: Deep Learning Methods, Computer Vision for Transportation, Deep Learning for Visual Perception
Abstract: Road curbs are considered as one of the crucial and ubiquitous traffic features, which are essential for ensuring the safety of autonomous vehicles. Current methods for detecting curbs primarily rely on camera imagery or LiDAR point clouds. Image-based methods are vulnerable to fluctuations in lighting conditions and exhibit poor robustness, while methods based on point clouds circumvent the issues associated with lighting variations. However, it is the typical case that significant processing delays are encountered due to the voluminous amount of 3D points contained in each frame of the point cloud data. Furthermore, the inherently unstructured characteristics of point clouds poses challenges for integrating the latest deep learning advancements into point cloud data applications. To address these issues, this work proposes an annotation-free curb detection method leveraging Altitude Difference Image (ADI) (as shown in Fig. 1), which effectively mitigates the aforementioned challenges. Given that methods based on deep learning generally demand extensive, manually annotated datasets, which are both expensive and labor-intensive to create, we present an Automatic Curb Annotator (ACA) module. This module utilizes a deterministic curb detection algorithm to automatically generate a vast quantity of training data. Consequently, it facilitates the training of the curb detection model without necessitating any manual annotation of data. Finally, by incorporating a post-processing module, we manage to achieve state-of-the-art results on the KITTI 3D curb dataset with considerably reduced processing delays compared to existing methods, which underscores the effectiveness of our approach in curb detection tasks.

Keywords: Deep Learning Methods, Intelligent Transportation Systems, Autonomous Vehicle Navigation
Abstract: We introduce R2LDM, an innovative approach for generating dense and accurate 4D radar point clouds, guided by corresponding LiDAR point clouds. Instead of utilizing range images or bird’s eye view (BEV) images, we represent both LiDAR and 4D radar point clouds using voxel features, which more effectively capture 3D shape information. Subsequently, we propose the Latent Voxel Diffusion Model (LVDM), which performs the diffusion process in the latent space. Additionally, a novel Latent Point Cloud Reconstruction (LPCR) module is utilized to reconstruct point clouds from high-dimensional latent voxel features. As a result, R2LDM effectively generates LiDAR-like point clouds from paired raw radar data. We evaluate our approach on two different datasets, and the experimental results demonstrate that our model achieves 6- to 10-fold densification of radar point clouds, outperforming state-of-the-art baselines in 4D radar point cloud super-resolution. Furthermore, the enhanced radar point clouds generated by our method significantly improve downstream tasks, achieving up to 31.7% improvement in point cloud registration recall rate and 24.9% improvement in object detection accuracy.

Keywords: Deep Learning Methods, Task Planning, AI-Based Methods
Abstract: 随着具身智能的发展，许多研究 通过整合场景图和 GNN 取得了进展 进入任务规划。然而，大多数方法仍然面临 完全捕获顺序关系的挑战 在 Agent作和环境之间，使其 难以处理的动态变化和固有的复杂性 在具体任务中。本文提出了一种动态图 体现任务规划注意力网络 （DGETP） 到 动态处理场景图序列和机器人图 环境感知。在 DGETP 中，我们设计了一个 Hierarchical 动态图注意力网络 （H-DGAT） 通过两者 结构和时间注意力机制来模拟 Dynamic Evolution 功能。A 双分支 动作对象预测器 （DAP） 在 DGETP 中通过以下方式提出 将先前作和对象的序列引入 高效聚合历史信息。DAP 捕获 过去和未来作之间的时间依赖关系 通过显式序列建模，并减少预测 通过分离的双分支架构来降低复杂性 action 和 object prediction 的 通过目标特征融合进行关联。实验 显示 DGETP 将任务准确性提高了 30% 以上 场景，与其他场景相比，在看不见的场景中超过 15% 基线。在复杂场景中，DGETP 表现出强大的 泛化能力。最后，模拟环境 表示 DGETP 实现的目标比大多数 高ń

Keywords: Deep Learning Methods, AI-Based Methods, Probabilistic Inference
Abstract: Accurate trajectory prediction is crucial for autonomous driving, yet uncertainty in agent behavior and perception noise makes it inherently challenging. While multi modal trajectory prediction models generate multiple plausible future paths with associated probabilities, effectively quantifying uncertainty remains an open problem. In this work, we propose a novel multi-modal trajectory prediction approach based on evidential deep learning that estimates both positional and mode probability uncertainty in real time. Our approach leverages a Normal Inverse Gamma distribution for positional uncertainty and a Dirichlet distribution for mode uncertainty. Unlike sampling-based methods, it infers both types of uncertainty in a single forward pass, significantly improving efficiency. Additionally, we experimented with uncertainty-driven importance sampling to improve training efficiency by prioritizing underrepresented high-uncertainty samples over redundant ones. We perform extensive evaluations of our method on the Argoverse 1 and Argoverse 2 datasets, demonstrating that it provides reliable uncertainty estimates while maintaining high trajectory prediction accuracy.

Keywords: Multi-Robot Systems, Behavior-Based Systems, Swarm Robotics
Abstract: Efficient exploration of large-scale environments remains a critical challenge in robotics, with applications ranging from environmental monitoring to search and rescue operations. This article proposes Frontier Shepherding (FroShe), a bio-inspired multi-robot framework for large-scale exploration. The framework heuristically models frontier exploration based on the shepherding behavior of herding dogs, where frontiers are treated as a swarm of sheep reacting to robots modeled as shepherding dogs. FroShe is robust across varying environment sizes and obstacle densities, requiring minimal parameter tuning for deployment across multiple agents. Simulation results demonstrate that the proposed method performs consistently, regardless of environment complexity, and outperforms state-of-the-art exploration strategies by an average of 20% with three UAVs. The approach was further validated in real-world experiments using single- and dual-drone deployments in a forest-like environment.

Keywords: Micro/Nano Robots, Swarm Robotics, Field Robots
Abstract: The efficacy of targeted cancer drug therapy is significantly compromised by imprecise drug delivery mechanisms. Micro/nano robots (MNRs), characterized by their controllable motion, present a promising solution to this challenge. However, the non-Newtonian nature of blood, with its high viscosity and blood cells’ interference, poses substantial limitations on the upstream efficiency of MNRs. This paper innovatively discusses for the first time the effects of blood viscosity and blood cell interference on the motion of MNRs, investigating their upstream motion capabilities in blood through comprehensive theoretical modeling, simulation, and experimental validation. A dynamic model of MNR motion was developed, and the velocity formula for MNRs in non-Newtonian fluid was derived. Experiments were conducted using different magnetic fields in pure water, high-viscosity simulated blood, and diluted blood. Results indicated that under a gradient magnetic field, the upstream velocities of MNRs in pure water, simulated blood, and diluted blood were 45.0, 14.4, and 11.1 mm/s, respectively. Under a rotating magnetic field, the velocities of vortex swarms were 825, 240, and 145 µm/s, respectively. Increased fluid viscosity reduced MNR velocity by 70%, while blood cells caused an additional 10% reduction. This research establishes a theoretical and experimental framework for the upstream motion of MNRs against blood flow, enhancing their potential in targeted drug delivery and broader biomedical applications.

Keywords: Swarm Robotics, Multi-Robot Systems, Model Learning for Control
Abstract: Accurate aerodynamic interaction modeling in multi-drone tasks is crucial for enhancing system stability and efficiency, especially when facing major disturbances from downwash wake effects. Conventional data-driven and empirical models mainly address simplified cases where one drone hovers or all vehicles have low absolute and relative velocities (≤ 0.5 m/s), and rely merely on relative states. In this study, we use high-fidelity Computational Fluid Dynamics (CFD) simulations to explore quadrotor interactions at higher speeds (0.5–4.0 m/s). We find that as the absolute velocities of the UAVs rise, downwash effects change significantly. To account for these discrepancies, we present a data-driven model considering both the absolute and relative properties of the downwash problem. We propose a geometric deep neural network predictor and compare its performance with existing data-driven and empirical models. Validations on two quadrotor settings show that our model gives more reliable predictions in tough scenarios and performs better in training without rigorous fine-tuning. Finally, we combine our predictor with a nonlinear feedback controller to enhance flight control under downwash disturbances. However, we encounter limitations for our speed ranges during trajectory tracking such as delays and velocity loss. Despite these challenges, our encoding and prediction method shows to be a promising step to address the downwash effects at higher speeds. We release our dataset, method, and re-implementations at: https://github.com/pavelkharitenko/flare-dw

Keywords: Multi-Robot Systems, Reinforcement Learning, Optimization and Optimal Control
Abstract: Deception is a crucial strategy in adversarial scenarios, yet its application in multi-agent confrontations remains understudied. This paper investigates deception in a multi-robot Target-Attacker-Defender (MR-TAD) game, where Attackers aim to capture Targets while evading Defenders. To model deception effectively, we propose a hierarchical decision-making framework that integrates multi-agent reinforcement learning (MARL) for high-level deceptive strategies and optimal control for low-level motion control. Furthermore, we introduce a novel composite deception-oriented reward function, which combines hitting rewards, belief switch rewards, and position advantage rewards to facilitate the training of deceptive behaviors. Simulation results across varying numbers of robots demonstrate that incorporating deception significantly increases the success rate of Attackers, with an average improvement of over 70% compared to non-deceptive strategies. Additionally, real-world experiments with omnidirectional mobile robots further confirm the effectiveness of the proposed method. This study establishes a generalizable framework for modeling deception in multi-agent systems, with potential applications in various multi-agent scenarios.

Keywords: Multi-Robot Systems, Distributed Robot Systems, Flexible Robotics
Abstract: Navigating unknown three-dimensional (3D) rugged environments is challenging for multi-robot systems. Traditional discrete systems struggle with rough terrain due to limited individual mobility, while modular systems—where rigid, controllable constraints link robot units—improve traversal but suffer from high control complexity and reduced flexibility. To address these limitations, we propose the Multi-Robot System with Controllable Weak Constraints (MRS-CWC), where robot units are connected by constraints with dynamically adjustable stiffness. This adaptive mechanism softens or stiffens in real time during environmental interactions, ensuring a balance between flexibility and mobility. We formulate the system’s dynamics and control model and evaluate MRS-CWC against six baseline methods and an ablation variant in 100 benchmark simulations. Results show that MRS-CWC achieves the highest navigation completion rate and ranks second in success rate, efficiency, and energy cost in the highly rugged terrain group, outperforming all baseline methods without relying on environmental modeling, path planning, or complex control. Even where MRS-CWC ranks second, its performance is only slightly behind a more complex ablation variant with environmental modeling and path planning. Finally, we develop a physical prototype and validate its feasibility in a constructed rugged environment. For videos, simulation benchmarks, and code, please visit the project website https://wyd0817.github.io/project-mrs-cwc/.

Keywords: Motion and Path Planning, Swarm Robotics, Multi-Robot Systems
Abstract: The multi-drone waypoint traversal has significant potential for aerial robot swarms in various applications. However, it still faces challenges including low time efficiency, susceptibility to local minima, poor resilience to external disturbances, high computational complexity, and high communication burden. To address these issues, we propose a two-stage swarm planning framework by integrating an offline global trajectory generator and an online distributed local trajectory planner. This approach not only ensures time-optimality but also enhances resistance to external disturbances. Specifically, a complementary progress constraint (CPC)-based global trajectory planning method is first presented to generate globally optimal reference trajectories. Then, by taking these trajectories as global guidance, a local planner is designed to guarantee collision-free traversal. In the local planner, we present a distributed local re-planning algorithm by embedding the positional constraints constructed by Voronoi diagrams into the model predictive contouring control (MPCC). The drones only exchange their position information, significantly reducing the communication load. Additionally, the Voronoi-based spatial constraints allow the swarms to eliminate the collision risk caused by asynchronous communication. To reduce onboard computational resource requirements, the local planner adopts the real-time iteration (RTI) technique, executing the optimization only once per control cycle. Both simulation and real-world experiments demonstrate that our approach outperforms state-of-the-art methods in terms of waypoint tracking accuracy, safety, and global time optimality.

Keywords: Swarm Robotics, Agent-Based Systems
Abstract: In this paper, we propose a framework, collective behavioral cloning (CBC), to learn the underlying interaction mechanism and control policy of a swarm system. Given the trajectory data of a swarm system, we propose a graph variational autoencoder (GVAE) to learn the local interaction graph. Based on the interaction graph and swarm trajectory, we use behavioral cloning to learn the control policy of the swarm system. To demonstrate the practicality of CBC, we deploy it on a real-world decentralized vision-based robot swarm system. A visual attention network is trained based on the learned interaction graph for online neighbor selection. Experimental results show that our method outperforms previous approaches in predicting both the interaction graph and swarm actions with higher accuracy. This work offers a promising approach for understanding interaction mechanisms and swarm dynamics in future swarm robotics research. Code and data are available

Keywords: Multi-Modal Perception for HRI, Imitation Learning, Object Detection, Segmentation and Categorization
Abstract: This research aims to enhance the interaction between humans and robots, especially in environments where multiple objects of the same type or semantic ambiguities exist. Traditional command-based interactions typically require users to provide precise descriptions, which often poses a significant challenge for users. To address this issue, we propose a framework named TagGuideBot, which leverages Visual Language Models (VLMs) and utilizes object markers to help locate and identify objects in the environment. By integrating positional point prompts of the target objects with robot motion planning models, we aim to achieve a more accurate understanding and execution of complex commands, thus improving the efficiency and naturalness of interactions. Experimental results demonstrate that TagGuideBot effectively addresses the challenges posed by complex commands and environmental complexities, achieving an accuracy of 66.3% on user instructions extended beyond the training set, providing solid support for further optimization of human-robot interaction.

Keywords: Multi-Modal Perception for HRI, Emotional Robotics, Social HRI
Abstract: Subjective self-disclosure is an important feature of human social interaction. While much has been done in the social and behavioural literature to characterise the features and consequences of subjective self-disclosure, little work has been done thus far to develop computational systems that are able to accurately model it. Even less work has been done that attempts to model specifically how human interactants self-disclose with robotic partners. It is becoming more pressing as we require social robots to work in conjunction with and establish relationships with humans in various social settings. In this paper, our aim is to develop a custom multimodal attention network based on models from the emotion recognition literature, training this model on a large self-collected self-disclosure video corpus, and constructing a new loss function, the scale preserving cross entropy loss, that improves upon both classification and regression versions of this problem. Our results show that the best performing model, trained with our novel loss function, achieves an F1 score of 0.83, an improvement of 0.48 from the best baseline model. This result makes significant headway in the aim of allowing social robots to pick up on an interaction partner's self-disclosures, an ability that will be essential in social robots with social cognition.

Keywords: Multi-Modal Perception for HRI, Semantic Scene Understanding, Deep Learning for Visual Perception
Abstract: Visual grounding aims at identifying objects or regions in a scene based on natural language descriptions, which is essential for spatially aware perception in autonomous driving. However, existing visual grounding tasks typically depend on bounding boxes that often fail to capture fine-grained details. Not all voxels within a bounding box are occupied, resulting in inaccurate object representations. To address this, we introduce a benchmark for 3D occupancy grounding in challenging outdoor scenes. Built on the nuScenes dataset, it fuses natural language with voxel-level occupancy annotations, offering more precise object perception compared to the traditional grounding task. Moreover, we propose GroundingOcc, an end-to-end model designed for 3D occupancy grounding through multimodal learning. It combines visual, textual, and point cloud features to predict object location and occupancy information from coarse to fine. Specifically, GroundingOcc comprises a multimodal encoder for feature extraction, an occupancy head for voxel-wise predictions, and a grounding head for refining localization. Additionally, a 2D grounding module and a depth estimation module enhance geometric understanding, thereby boosting model performance. Extensive experiments on the benchmark demonstrate that our method outperforms existing baselines on 3D occupancy grounding. The dataset is available at https://github.com/RONINGOD/GroundingOcc.

Keywords: Multi-Modal Perception for HRI, AI-Enabled Robotics, Perception for Grasping and Manipulation
Abstract: In recent years, Vision-Language Models (VLMs) have exhibited powerful capacity of reasoning, decomposing long-horizon tasks and motion planning in robotic manipulation tasks. However, the current operating speed of VLMs has limited the interaction frequency of users and the model to several seconds, which disables the real-time perception of environmental changes when executing tasks released by VLM. We propose Real-time Object Detection- VLM (ROD-VLM), a novel framework which combines classical Object Detection Algorithm YOLO-v5x with VLM to achieve the real-time robotic environmental perception, reasoning and manipulation. Specifically, we introduce the concept of key frame to VLM model, capturing the crucial information through object detection algorithm to assist VLM in perceiving varying environment.Our comprehensive real-world experiments show that ROD-VLM possess an excellent capability in real-time environmental understanding, decision-making and action executing.

Keywords: Natural Dialog for HRI, Service Robotics, Multi-Modal Perception for HRI
Abstract: Turn-taking is a crucial aspect of human-robot interaction, directly influencing conversational fluidity and user engagement. While previous research has explored turn-taking models in controlled environments, their robustness in real-world settings remains underexplored. In this study, we propose a noise-robust voice activity projection (VAP) model, based on a Transformer architecture, to enhance real-time turn-taking in dialogue robots. To evaluate the effectiveness of the proposed system, we conducted a field experiment in a shopping mall, comparing the VAP system with a conventional cloud-based speech recognition system. Our analysis covered both subjective user evaluations and objective behavioral analysis. The results showed that the proposed system significantly reduced response latency, leading to a more natural conversation where both the robot and users responded faster. The subjective evaluations suggested that faster responses contribute to a better interaction experience.

Keywords: Natural Dialog for HRI, Multi-Modal Perception for HRI, Telerobotics and Teleoperation
Abstract: In recent years, numerous studies have been conducted on dialogue robots powered by large language models, enabling sophisticated interactions such as providing guidance and engaging in small talk. However, the interaction performance remains imperfect, and the robots sometimes cause problems during interactions. In this study, we aim to automatically detect such anomalies in human-robot interactions by creating a dataset and developing anomaly detection models. To this end, we created a dataset by manually annotating videos of in-the-wild interactions collected from our field experiment designed to test a framework of parallel conversations in which a human intervenes when a problem occurs in the interaction. Using this dataset, we trained classification models to construct anomaly detection models. We then conducted another field experiment in which the model's detection results were presented as alerts to operators within the parallel conversation framework. The results confirmed that providing alerts on the basis of the anomaly detection model was useful for facilitating operator intervention.

Keywords: Natural Dialog for HRI, Manipulation Planning, Motion and Path Planning
Abstract: Human-robot interaction requires robots to process language incrementally, adapting their actions in real-time based on evolving speech input. Existing approaches to language-guided robot motion planning typically assume fully specified instructions, resulting in inefficient stop-and-replan behavior when corrections or clarifications occur. In this paper, we introduce a novel reasoning-based incremental parser which integrates an online motion planning algorithm within the cognitive architecture. Our approach enables continuous adaptation to dynamic linguistic input, allowing robots to update motion plans without restarting execution. The incremental parser maintains multiple candidate parses, leveraging reasoning mechanisms to resolve ambiguities and revise interpretations when needed. By combining symbolic reasoning with online motion planning, our system achieves greater flexibility in handling speech corrections and dynamically changing constraints. We evaluate our framework in real-world human-robot interaction scenarios, demonstrating online adaptions of goal poses, constraints, or task objectives. Our results highlight the advantages of integrating incremental language understanding with real-time motion planning for natural and fluid human-robot collaboration. The experiments are demonstrated in the accompanying video at www.acin.tuwien.ac.at/42d5.

Keywords: Datasets for Human Motion, Data Sets for Robotic Vision, Data Sets for Robot Learning
Abstract: The event-based camera is a novel neuromorphic vision sensor that can perceive different dynamic behaviors due to its low latency, asynchronous data stream, and high dynamic range characteristics. There has been much work based on event cameras to solve problems such as object tracking, visual odometry, and gesture recognition. However, the adoption of event vision to analyze hand-object action in a dynamic environment, a problem that regular CMOS cameras cannot handle, is still lacking in relevant research. This work presents a richly annotated task-oriented hand-object action dataset consisting of asynchronous event streams, captured by the event-based camera system on different application scenarios. In addition, we design an attention-based residual spiking neural network (ARSNN) by learning temporal-wise and spatial-wise attention simultaneously and introducing a particular residual connection structure to achieve dynamic hand-object action recognition. Extensive experiments are validated by comparing with existing baseline methods to form a vision benchmark. We also show that the learned recognition model can be transferred to classify a real robot hand-object action.

Keywords: Autonomous Vehicle Navigation, Intelligent Transportation Systems
Abstract: Formulating a multi-robot obstacle avoidance policy is essential for enabling safe and efficient navigation in multi-robot environments, forming a critical component of the effective operation of multi-robot systems. Recently, reinforcement learning has been applied to improve the performance of decentralized, policy-driven robots in task execution. However, ensuring the safety of these agents during movement remains a significant challenge due to the inherent risks associated with the reinforcement learning process, such as frequent collisions. To address this issue and enhance the safety of policy-guided multi-robot navigation, we propose a novel policy based on imitation learning. This framework introduces a novel policy neural network that integrates a graph attention mechanism with the GRU network structure. The key innovation lies in utilizing the interactions between neighboring robots to enhance the safety of their movements. In a multi-robot simulation environment, robot behaviors are directed by the proposed policy. A comparative analysis was conducted between our approach and RL-RVO, one of the advanced methods in the field. The results demonstrate that our approach outperforms RL-RVO, achieving a higher success rate and significantly improving safety performance.

Keywords: Autonomous Vehicle Navigation, Simulation and Animation, Deep Learning for Visual Perception
Abstract: In the field of autonomous driving, sensor simulation is essential for generating rare and diverse scenarios that are difficult to capture in real-world environments. Current solutions fall into two categories: 1) CG-based methods, such as CARLA, which lack diversity and struggle to scale to the vast array of rare cases required for robust perception training; and 2) learning-based approaches, such as NeuSim, which are limited to specific object categories (vehicles) and require extensive multi-sensor data, hindering their applicability to generic objects.

To address these limitations, we propose SynthDrive, a scalable real2sim2real system that leverages 3D generation to automate asset mining, generation, and rare-case data synthesis.

Our framework introduces two key innovations: 1) Automated Rare-Case Mining and Synthesis. Given a text prompt describing specific objects, SynthDrive automatically mines image data from the Internet and then generates corresponding high-fidelity 3D assets, which eliminates the need for costly manual data collection. By integrating these assets into existing street-view data, our pipeline produces photorealistic rare-case data, supporting rapid scaling to diverse assets including irregular obstacles and temporary traffic facilities. 2) High-Fidelity 3D Generation. We propose a hybrid asset generation pipeline that combines a geometry-aware LRM, iterative mesh optimization, and an improved texture fusion algorithm. Our approach achieves 0.0164 Chamfer Distance on GSO dataset, outperforming InstantMesh by 14.1% in geometry accuracy, and achieves 19.05 PSNR (vs 16.84) for texture quality. This enables fine geometry details and high-resolution texture generation, which is essential for perception model training. Experiments demonstrate that SynthDrive-generated data improves performance of downstream perception tasks (2D and 3D detection on rare objects) by 2-4% mAP.

SynthDrive greatly lowers the data production cost and improves the diversity for corner-case data generation, showcasing extensive potential applications in the field of autonomous driving.

Keywords: Autonomous Vehicle Navigation, Collision Avoidance, Kinematics
Abstract: Autonomous driving is a high-performance,safety-critical task. Effectively controlling autonomous vehicles to improve their control performance and safety is critical, especially when dealing with complex and dynamic environments. However, in real-time obstacle avoidance (OA) scenarios, the planning layer often fails due to high computational complexity and response delays. This trade-off between computational efficiency and safety performance highlights a critical challenge: how to achieve an optimal balance between autonomous driving safety and real-time performance. Therefore, in recent years, addressing OA at the control layer has become a key research focus for enhancing the safety of autonomous vehicles. Considering MPC’s advantages in prediction and constraint handling, this paper is specifically integrated an OA safety distance constraint into MPC to effectively address OA in UGVs. First, a Taylor expansion is used to construct the error model of the UGV. Then, the safe distance constraints for obstacle avoidance is proposed to consider both tracking errors and closeness to obstacles. Additionally, physical constraints are also taken into account in the development of a safe obstacle avoidance MPC (SOAMPC). By incorporating safety distance constraints for obstacle avoidance and considering physical constraints, SOAMPC is proposed and implemented in the UGVs. Furthermore, essential control theoretic properties are established, including recursive feasibility, the guarantee of collision avoidance, and system stability. To evaluate the effectiveness of the SOAMPC controller, simulations and experiments are conducted in a multiple-obstacle environment. The results demonstrate that the SOAMPC method successfully avoids obstacles while maintaining stability. Comparing to other methods, the results demonstrate the superior efficiency of the proposed SOAMPC, while achieving accurate path tracking.

Keywords: Autonomous Vehicle Navigation, Deep Learning Methods, Simulation and Animation
Abstract: With the rapid advancement of autonomous driving technology, a lack of data has become a major obstacle to enhancing perception model accuracy. Researchers are now exploring controllable data generation using world models to diversify datasets. However, previous work has been limited to studying image generation quality on specific public datasets. There is still relatively little research on how to build data generation engines for real-world application scenes to achieve large-scale data generation for challenging scenes. In this paper, a simulator-conditioned scene generation engine based on world model is proposed. By constructing a simulation system consistent with real-world scenes, simulation data and labels, which serve as the conditions for data generation in the world model, for any scenes can be collected. It is a novel data generation pipeline by combining the powerful scene simulation capabilities of the simulation engine with the robust data generation capabilities of the world model. In addition, a benchmark with proportionally constructed virtual and real data, is provided for exploring the capabilities of world models in real-world scenes. Quantitative results show that these generated images significantly improve downstream perception models performance. Finally, we explored the generative performance of the world model in urban autonomous driving scenarios. All the data and code will be available at url{https://github.com/Li-Zn-H/SimWorld}.

Keywords: Autonomous Vehicle Navigation, Motion and Path Planning, AI-Enabled Robotics
Abstract: This work focuses on enhancing the generalization performance of deep reinforcement learning-based robot navigation in unseen environments. We present a novel data augmentation approach called scenario augmentation, which enables robots to navigate effectively across diverse settings without altering the training scenario. The method operates by mapping the robot's observation into an imagined space, generating an imagined action based on this transformed observation, and then remapping this action back to the real action executed in simulation. Through scenario augmentation, we conduct extensive comparative experiments to investigate the underlying causes of suboptimal navigation behaviors in unseen environments. Our analysis indicates that limited training scenarios represent the primary factor behind these undesired behaviors. Experimental results confirm that scenario augmentation substantially enhances the generalization capabilities of deep reinforcement learning-based navigation systems. The improved navigation framework demonstrates exceptional performance by producing near-optimal trajectories with significantly reduced navigation time in real-world applications.

Keywords: AI-Enabled Robotics, Embodied Cognitive Science, RGB-D Perception
Abstract: Vision-and-Language Navigation in Continuous Envi- ronments (VLN-CE) requires agents to navigate 3D environments based on visual observations and natural language instructions. Existing approaches, focused on topological and semantic maps, often face limitations in accurately understanding and adapting to complex or previously unseen environments, particularly due to static and offline map constructions. To address these chal- lenges, this paper proposes OVL-MAP, an innovative algorithm comprising three key modules: an online vision-and-language map construction module, a waypoint prediction module, and an action decision module. The online map construction module leverages robust open-vocabulary semantic segmentation to dynamically enhance the agent’s scene understanding. The waypoint prediction module processes natural language instructions to identify task- relevant regions, predict sub-goal locations, and guide trajectory planning. The action decision module utilizes the DD-PPO strategy for effective navigation. Evaluations on the Robo-VLN and R2R- CE datasets demonstrate that OVL-MAP significantly improves navigation performance and exhibits stronger generalization in unknown environments.

Keywords: Autonomous Vehicle Navigation, Motion and Path Planning, Intelligent Transportation Systems
Abstract: Navigating autonomous vehicles in open scenarios is a challenge due to the difficulties in handling unseen objects. Existing solutions either rely on small models that struggle with generalization or large models that are resource-intensive. While collaboration between the two offers a promising solution, the key challenge is deciding when and how to engage the large model. To address this issue, this paper proposes opportunistic collaborative planning (OCP), which seamlessly integrates efficient local models with powerful cloud models through two key innovations. First, we propose large vision model guided model predictive control (LVM-MPC), which leverages the cloud for LVM perception and decision making. The cloud output serves as a global guidance for a local MPC, thereby forming a closed-loop perception-to-control system. Second, to determine the best timing for large model query and service, we propose collaboration timing optimization (CTO), including object detection confidence thresholding (ODCT) and cloud forward simulation (CFS), which decides when to seek cloud assistance and when to offer cloud service. Extensive experiments show that the proposed OCP outperforms existing methods in terms of both navigation time and success rate.

Keywords: Autonomous Vehicle Navigation, Deep Learning Methods, Optimization and Optimal Control
Abstract: The highly nonlinear dynamics of vehicles present a major challenge for the practical implementation of optimal control and Model Predictive Control (MPC) approaches in path planning and tracking applications. Koopman operator theory offers a global linear representation of nonlinear dynamical systems, making it a promising framework for optimization-based vehicle control. This paper introduces a novel deep learning-based Koopman modeling approach that employs deep neural networks to capture the full vehicle dynamics, from pedal and steering inputs to chassis states, within a curvilinear Frenet frame. The superior accuracy of the Koopman model compared to identified linear models is shown for a double lane change maneuver. Furthermore, it is shown that an MPC controller deploying the Koopman model provides significantly improved performance while maintaining computational efficiency comparable to a linear MPC.

Keywords: Distributed Robot Systems, Multi-Robot Systems, Networked Robots
Abstract: Distributed task allocation in UAV swarms is sensitive to excessive bandwidth requirements and frequent inter-UAV communication. Combining reinforcement learning and traditional distributed task allocation demonstrates great potential in enhancing algorithm performance and optimizing communication. However, existing studies rely on ideal bandwidth assumptions and unrealistic time synchronization, making training and validation impractical under real-world conditions. This paper proposes a communication-aware distributed task allocation method that employs an asynchronous strategy learning framework and multi-objective optimization to reduce communication throughput and task conflicts. First, the allocation algorithm integrated with communication is formalized as an Asynchronous Constrained Decentralized Partially Observable Markov Decision Process (ACDEC-POMDP), which extends the original learning objective to accommodate asynchronous requirements. Channel access and other features are observations, actions are inter-agent adaptive gating mechanisms, and the shared reward reflects global task conflict changes. Second, to address the asynchronous data processing problem under the Centralized Training-Distributed Execution (CTDE), a method based on 'concatenation' is proposed to stitch together trajectories from different UAVs, enabling flexible and stable training deployment. Third, a Multi-Objective Coupled Proximal Policy Optimization (MOC-PPO) is proposed, aiming to simultaneously reduce bandwidth demands and minimize task conflicts, where a reinforcement learning method integrating Lagrangian-dual optimization is employed to solve for the optimal parameters. Finally, a Hardware-in-the-Loop (HIL) environment is built, using authentic network protocols and simulated channel transmissions, to bridge the gap between simulation-trained strategies and real-world communication deployment. The experimental results show that, compared with the original asynchronous method, the trained strategy can significantly reduce the communication overhead without increasing the optimization loss. In addition, the trained communication strategies demonstrate strong scalability-generalization.

Keywords: Multi-Robot Systems, Localization, Sensor Networks
Abstract: Relative localization is a crucial capability for multi-robot systems operating in GPS-denied environments. Existing approaches for multi-robot relative localization often depend on costly or short-range sensors like cameras and LiDARs. Consequently, these approaches face challenges such as high computational overhead (e.g., map merging) and difficulties in disjoint environments. To address this limitation, this paper introduces MGPRL, a novel distributed framework for multi-robot relative localization using convex-hull of multiple Wi-Fi access points (AP). To accomplish this, we employ co-regionalized multi-output Gaussian Processes for efficient Radio Signal Strength Indicator (RSSI) field prediction and perform uncertainty-aware multi-AP localization, which is further coupled with weighted convex hull-based alignment for robust relative pose estimation. Each robot predicts the RSSI field of the environment by an online scan of APs in its environment, which are utilized for position estimation of multiple APs. To perform relative localization, each robot aligns the convex hull of its predicted AP locations with that of the neighbor robots. This approach is well-suited for devices with limited computational resources and operates solely on widely available Wi-Fi RSSI measurements without necessitating any dedicated pre-calibration or offline fingerprinting. We rigorously evaluate the performance of the proposed MGPRL in ROS simulations and demonstrate it with real-world experiments, comparing it against multiple state-of-the-art approaches. The results showcase that MGPRL outperforms existing methods in terms of localization accuracy and computational efficiency.

Keywords: Multi-Robot Systems, Localization
Abstract: In this paper, we propose a theoretical framework for designing a multi-robot formation equipped with Ultra-wideband (UWB) sensors to localize a target robot. In the presence of noisy range measurements, the accuracy of the target robot’s pose estimation is highly dependent on the chosen formation geometry. Different from existing works, we account for the heterogeneous standard deviations of range measurements across different UWB transmitter-receiver pairs. We establish new optimality conditions for formation geometries and conduct a sensitivity analysis of optimal formations under robot positioning errors. In a 2D setting, we derive necessary and sufficient conditions for both optimality and robustness to robot positioning uncertainty. Experimental results confirm the heterogeneous standard deviations of UWB range measurements and validate the target robot’s confidence ellipse model. An experimental comparison of formation geometries, optimized with and without considering heterogeneous noise, emphasizes the importance of accounting for the heterogeneous standard deviations of range measurements. In addition, we experimentally demonstrate that robust formation geometries improve the target robot’s confidence ellipse in the presence of positioning errors.

Keywords: Multi-Robot Systems, Bioinspired Robot Learning, Machine Learning for Robot Control
Abstract: Quadruped robots have demonstrated remarkable versatility in various applications, from search and rescue to exploration. Recent advancements have shifted focus from individual robots to swarms, recognizing the potential of collaborative behaviors to achieve complex tasks beyond the capabilities of a single robot. Inspired by the cooperative hunting behaviors observed in nature, this paper presents a reinforcement learning framework for a swarm of quadruped robots to learn decentralized end-to-end hunting locomotion. In particular, we integrate stable and dynamic locomotion with hunting objectives and utilize a guidance vector as privileged information for efficient training. The framework concerns the control dynamics of quadruped robots, ensuring both low-level stability and high-level hunting coordination in muti-robot environments. The trained policy is deployed onto a real robot system, and the experimental results demonstrate coordinative behavior in various scenarios. The implementation code is released to benefit the community.

Keywords: Multi-Robot Systems, Collision Avoidance, Machine Learning for Robot Control
Abstract: With multi-agent systems increasingly deployed autonomously at scale in complex environments, ensuring safety of the data-driven policies is critical. Control Barrier Functions have emerged as an effective tool for enforcing safety constraints, yet existing learning-based methods often lack in scalability, generalization and sampling efficiency as they overlook inherent geometric structures of the system. To address this gap, we introduce symmetries-infused distributed CBFs, enforcing the satisfaction of intrinsic symmetries on learnable graph-based safety certificates. We theoretically motivate the need for equivariant parametrization of CBFs and policies, and propose a simple, yet efficient and adaptable methodology for constructing such equivariant group-modular networks via the compatible group actions. This approach encodes safety constraints in a distributed data-efficient manner, enabling zero-shot generalization to larger and denser swarms. Through extensive simulations on multi-robot navigation tasks, we demonstrate that our method outperforms state-of-the-art baselines in terms of safety, scalability, and task success rates, highlighting the importance of embedding symmetries in safe distributed neural policies.

Keywords: Multi-Robot Systems, Mapping, Localization
Abstract: Ground and aerial robots, with distinct sensing perspectives, acquire heterogeneous point clouds that exhibit limited overlap, presenting significant challenges for collaborative mapping. To address these challenges, this article proposes a robust LiDAR-based aerial-ground collaborative mapping framework for large-scale outdoor environments. Firstly, to perform reliable cross-source place recognition and detect loop closure between aerial-ground robots, a deep network that fuses multi-level bird’s-eye view (BEV) and geometric features is developed to ensure consistent feature extraction and emphasize overlaps between heterogeneous point clouds. Next, an overlapaware registration method is proposed to align point clouds within a detected loop closure. This method can strategically perform point cloud sparsification based on overlap ratio estimation, and mitigate the adverse effects of interfering points in non-overlapping regions. Furthermore, a graph optimization is implemented to consider all loop closure constraints simultaneously and ensure global map consistency. Comparative experiments on public and self-collected datasets demonstrate the superiority of the proposed approach. We will release our code at https://github.com/luanshuang/AGCMapping.

Keywords: Multi-Robot Systems, Motion and Path Planning, Intelligent Transportation Systems
Abstract: In warehouse logistics and post-disaster rescue, multi-UAV payload transport must navigate tight spaces, such as 1.2 m × 0.8 m aisles and collapsed pipelines as narrow as 0.6 m. Traditional four-DOF (translation and scaling) trajectory planning struggles under such constraints. To overcome this, we propose an optimization-based framework that introduces rotational degrees of freedom, expanding the solution space to five dimensions. Using the MINCO transformation, we reformulate constrained formation adjustment into an unconstrained optimization problem via smooth mappings and penalty functions, enabling simultaneous obstacle avoidance and formation control. The GeoSafe algorithm further enhances safe passage by integrating iterative region expansion and semi-definite programming to maximize obstacle-free space. Extensive simulations and real-world experiments show our method’s superiority over sampling-based and IF-based approaches in narrow passage traversal, computational efficiency, and formation scalability.

Keywords: Multi-Robot SLAM, SLAM, Cooperating Robots
Abstract: Abstract—This letter presents a sparse hierarchical LiDAR bundle adjustment method for online multi-robot collaborative simultaneous localization and mapping (C-SLAM). The motiva tion behind this work is that the pose graph cannot directly reflect map inconsistencies. As a result, the map divergence across multiple robots persists even after pose graph optimization. While existing methods have utilized LiDAR bundle adjustment (BA) to address the divergence issues, none of these methods are particularly effective at improving online localization accuracy in multi-robot scenarios. In this work, we propose a sparse hier archical mechanism where LiDAR bundle adjustment functions as a low-latency module within a centralized C-SLAM system. The hierarchical design accelerates the original BA process, while sparse selection further decreases the problem’s complexity, thereby improving computational efficiency. The combination of high-frequency multi-robot pose graph optimization (MR-PGO) and low-frequency multi-robot hierarchical bundle adjustment (MR-HBA) improves accuracy and provides real-time localization results. To validate the effectiveness of our proposed method, we conduct comparative experiments using multiple public datasets and a self-collected dataset, benchmarking against state-of-the art multi-robot SLAM systems. The results demonstrate that our method significantly enhances the accuracy of localization and mapping. Additionally, we have made the entire system available as an open-source implementation to benefit the broader research community.

Keywords: Grasping, Grippers and Other End-Effectors
Abstract: Robotic graspers are essential for enhancing the efficiency and versatility of robots in grasping tasks. In this paper, we propose a novel inflatable deployable origami grasper with a rigid-flexible coupling structure. The proposed grasper can achieve multiple deployment configurations under a single pneumatic actuation, enabling both deployment and grasping operations while also allowing for passive self-folding during deflation. The design and fabrication of the grasper are presented. Then, the stiffness model for the inflatable deployable origami unit is developed based on the equivalent truss method. Experimental results show that the grasper successfully grasps objects of various shapes and sizes in both enveloping and fingertip grasping modes, using either two or four fingers. With its simple mechanical system and high deploy/fold ratio, the proposed grasper holds significant potential for applications in industrial automation and space exploration.

Keywords: Grasping, AI-Based Methods, Evolutionary Robotics
Abstract: Task-aware robotic grasping is a challenging problem that requires the integration of semantic understanding and geometric reasoning. This paper proposes a novel framework that leverages Large Language Models (LLMs) and Quality Diversity (QD) algorithms to enable zero-shot task-conditioned grasp synthesis. The framework segments objects into meaningful subparts and labels each subpart semantically, creating structured representations that can be used to prompt an LLM. By coupling semantic and geometric representations of an object's structure, the LLM's knowledge about tasks and which parts to grasp can be applied in the physical world. The QD-generated grasp archive provides a diverse set of grasps, allowing us to select the most suitable grasp based on the task. We evaluated the proposed method on a subset of the YCB dataset with a Franka Emika robot. A consolidated ground truth for task-specific grasp regions is established through a survey. Our work achieves a weighted intersection over union (IoU) of 73.6% in predicting task-conditioned grasp regions in 65 task-object combinations. An end-to-end validation study on a smaller subset further confirms the effectiveness of our approach, with 88% of responses favoring the task-aware grasp over the control group. A binomial test shows that participants significantly prefer the task-aware grasp.

Keywords: Grasping, Deep Learning in Grasping and Manipulation
Abstract: High-level robotic manipulation tasks demand flexible 6-DoF grasp estimation to serve as a basic function. Previous approaches either directly generate grasps from point-cloud data, suffering from challenges with small objects and sensor noise, or infer 3D information from RGB images, which introduces expensive annotation requirements and discretization issues. Recent methods mitigate some challenges by retaining a 2D representation to estimate grasp keypoints and applying Perspective-n-Point (PnP) algorithms to compute 6-DoF poses. However, these methods are limited by their non-differentiable nature and reliance solely on 2D supervision, which hinders the full exploitation of rich 3D information. In this work, we present KGN-Pro, a novel grasping network that preserves the efficiency and fine-grained object grasping of previous KGNs while integrating direct 3D optimization through probabilistic PnP layers. KGN-Pro encodes paired RGB-D images to generate grasp center heatmaps and keypoint offsets, and further computes a 2D confidence map to weight keypoint contributions during re-projection error minimization. By modeling the weighted sum of squared re-projection errors probabilistically, the network effectively transmits 3D supervision to its 2D keypoint predictions, enabling end-to-end learning. Experiments on both simulated and real-world platforms demonstrate that KGN-Pro outperforms existing methods in terms of grasp cover rate and success rate. Project website: https://waitderek.github.io/kgnpro.

Keywords: Grasping, Manipulation Planning, Perception for Grasping and Manipulation
Abstract: Grasping large and flat objects (e.g., a book or a pan) is often regarded as an ungraspable task, which poses significant challenges due to the unreachable grasping poses. Prior research has exploited environmental interactions through Extrinsic Dexterity, utilizing external structures such as walls or table edges to facilitate object grasping. However, they are confined to task-specific policies while neglecting semantic perception and planning to identify optimal pre-grasp configurations. This limits their operational versatility, impeding effective adaptation to varied extrinsic dexterity constraints. In this work, we present ExDiff, a robot manipulation approach for extrinsic dexterity grasping in unrestricted environments. It utilizes Vision-Language Models (VLMs) to perceive the environmental state and generate instructions, followed by a Goal-Conditioned Action Diffusion (GCAD) model to predict the sequence of low-level actions. This diffusion model learns the low-level policy, conditioned on high-level instructions and cumulative rewards, which improves the generation of robot actions. Simulation experiments and real-world deployment results demonstrate that ExDiff effectively performs ungraspable tasks and generalizes to previously unseen target objects and scenes.

Keywords: Grasping, Manipulation Planning, Dual Arm Manipulation
Abstract: Visual manipulation relationship detection facilitates robots to achieve safe, orderly, and efficient grasping tasks. However, most existing algorithms only model object-level or relational-level dependency individually, lacking sufficient global information, which is difficult to handle different types of reasoning errors, especially in complex environments with multi-object stacking and occlusion. To solve the above problems, we propose Dual Graph Attention Networks (Dual-GAT) for visual manipulation relationship detection, with an object-level graph network for capturing object-level dependencies and a relational-level graph network for capturing relational triplets-level interactions. The attention mechanism assigns different weights to different dependencies, obtains more accurate global context information for reasoning, and gets a manipulation relationship graph. In addition, we use multi-view feature fusion to improve the occluded object features, then enhance the relationship detection performance in multi-object scenes. Finally, our method is deployed on the robot to construct a multi-object grasping system, which can be well applied to stacking environments. Experimental results on the datasets VMRD and REGRAD show that our method significantly outperforms others.

Keywords: Grasping, Deep Learning in Grasping and Manipulation, Dexterous Manipulation
Abstract: We introduce DexDiffuser, a novel dexterous grasping method that generates, evaluates, and refines grasps on partial object point clouds. DexDiffuser includes the conditional diffusion-based grasp sampler DexSampler and the dexterous grasp evaluator DexEvaluator. DexSampler generates high-quality grasps conditioned on object point clouds by iterative denoising of randomly sampled grasps. We also introduce two grasp refinement strategies: Evaluator-Guided Diffusion and Evaluator-based Sampling Refinement. The experiment results demonstrate that DexDiffuser consistently outperforms the state-of-the-art multi-finger grasp generation method FFHNet with an, on average, 9.12% and 19.44% higher grasp success rate in simulation and real robot experiments, respectively.

Keywords: Grasping, Perception for Grasping and Manipulation, Human-Robot Collaboration
Abstract: Achieving precise and generalizable grasping across diverse objects and environments is essential for intelligent and collaborative robotic systems. However, existing approaches often struggle with ambiguous affordance reasoning and limited adaptability to unseen objects, leading to suboptimal grasp execution. In this work, we propose GAT-Grasp, a gesture-driven grasping framework that directly utilizes human hand gestures to guide the generation of task-specific grasp poses with appropriate positioning and orientation. Specifically, we introduce a retrieval-based affordance transfer paradigm, leveraging the implicit correlation between hand gestures and object affordances to extract grasping knowledge from large-scale human-object interaction videos. By eliminating the reliance on pre-given object priors, GAT-Grasp enables zero-shot generalization to novel objects and cluttered environments. Real-world evaluations confirm its robustness across diverse and unseen scenarios, demonstrating reliable grasp execution in complex task settings.

Keywords: Grasping, Learning from Demonstration, Haptics and Haptic Interfaces
Abstract: Robotic manipulation is essential for the widespread adoption of robots in industrial and home settings and has long been a focus within the robotics community. Advances in artificial intelligence have introduced promising learning-based methods to address this challenge, with imitation learning emerging as particularly effective. However, efficiently acquiring high-quality demonstrations remains a challenge. In this work, we introduce an immersive VR-based teleoperation setup designed to collect demonstrations from a remote human user. We also propose an imitation learning framework called Haptic Action Chunking with Transformers (Haptic-ACT). To evaluate the platform, we conducted a pick-and-place task and collected 50 demonstration episodes. Results indicate that the immersive VR platform significantly reduces demonstrator fingertip forces compared to systems without haptic feedback, enabling more delicate manipulation. Additionally, evaluations of the Haptic-ACT framework in both the MuJoCo simulator and on a real robot demonstrate its effectiveness in teaching robots more compliant manipulation compared to the original ACT. Additional materials are available at https://sites.google.com/view/hapticact.

Keywords: Humanoid Robot Systems, Body Balancing, Multi-Contact Whole-Body Motion Planning and Control
Abstract: Standing and sitting are basic tasks that become in- creasingly difficult with age or frailty. Assisting these movements using humanoid robots is a complex challenge, particularly in determining where and how much force the robot should apply to effectively support the human’s dynamic motions. In this letter, we propose a method to compute assistive forces directly from the human’s dynamic balance, using criteria typically employed in humanoid robots. Specifically, we map humanoid dynamic balance metrics onto human motion to calculate the forces required to stabilize the human’s current posture. These forces are then applied at the appropriate locations on the human body by the humanoid. Our approach combines the variable height 3D divergent com- ponent of motion with gravito-inertial wrench cones to define a 3D balance region. Using centroidal feedback, we compute the required assistance force to maintain balance and distribute the resulting wrenches across the human’s body using a humanoid robot dynamically balanced according to the same criteria. We demonstrate the effectiveness of this framework through both sim- ulations and experiments, where a humanoid assists a person in sit-to-stand and stand-to-sit motions, with the person wearing an age-simulation suit to emulate frailty.

Keywords: Humanoid and Bipedal Locomotion, Reinforcement Learning, Imitation Learning
Abstract: Achieving robust locomotion on complex terrains remains a challenge due to high-dimensional control and environmental uncertainties. This paper introduces a teacher-prior framework based on the teacher-student paradigm, integrating imitation and auxiliary task learning to improve learning efficiency and generalization. Unlike traditional paradigms that strongly rely on encoder-based state embeddings, our framework decouples the network design, simplifying the policy network and deployment. A high-performance teacher policy is first trained using privileged information to acquire generalizable motion skills. The teacher’s motion distribution is transferred to the student policy, which relies only on noisy proprioceptive data, via a generative adversarial mechanism to mitigate performance degradation caused by distributional shifts. Additionally, auxiliary task learning enhances the student policy’s feature representation, speeding up convergence and improving adaptability to varying terrains. The framework is validated on a humanoid robot, showing a great improvement in locomotion stability on dynamic terrains and significant reductions in development costs. This work provides a practical solution for deploying robust locomotion strategies in humanoid robots.

Keywords: Humanoid Robot Systems, Optimization and Optimal Control, Dynamics
Abstract: Optimizing the initial angle of the shoulder's base frame is crucial for defining the workspace and enhancing the manipulation performance of humanoid robotic arms. Previous studies primarily emphasized geometric analyses, neglecting dynamic factors, which limits their practical applicability. This study presents a multi-metric optimization framework to enhance the dynamic performance of humanoid robotic arms by optimizing the shoulder’s initial angle. We formulate a cost function that incorporates torque efficiency, energy consumption, and overload ratio, utilizing the differential evolution (DE) algorithm for optimization. Furthermore, to address the limitations of conventional geometric workspace analysis, we introduce the concept of effective workspace, integrating dynamic constraints to quantitatively evaluate the effects of optimized shoulder angles. We validate the proposed framework in a hybrid simulation environment combining MuJoCo and RBDL, using the KIST humanoid and Unitree G1 robotic arms. Experimental results confirm that the optimized shoulder angles enhance torque distribution, expanding the effective workspace by 18.4% and 3.78% for the KIST and Unitree G1 robotic arms, respectively. These findings demonstrate that the proposed optimization framework enhances manipulation and dynamic performance as well as energy efficiency and system reliability, contributing to advancements in humanoid robotic arm design.

Keywords: Humanoid Robot Systems, Methods and Tools for Robot System Design, Optimization and Optimal Control
Abstract: The optimization of hardware and control of humanoid robots for multiple tasks is still an open challenge due to the competing objectives of different behaviors and the complexity of considering control architectures at the design level of a humanoid robot. In this work, we propose a unified multi-objective optimization framework that jointly optimizes both hardware and hierarchical control architectures of a humanoid robot to enhance performance in multiple tasks. Our method employs a Non-dominated Sorting Genetic Algorithm II (NSGA-II) to identify optimal robot morphology and control parameters while balancing trade-offs between diverse task requirements. By leveraging genetic algorithms, we enable the integration of discrete search spaces while overcoming the local minima limitations associated with classical nonlinear optimization techniques. Furthermore, the proposed approach directly incorporates the simulation results, ensuring that hardware optimization is performed considering the system dynamics. We validate our approach by optimizing a humanoid robot for two distinct tasks: walking and payload lifting, leveraging MuJoCo to evaluate the task performances. The proposed framework successfully identifies Pareto-optimal tradeoffs, providing a set of design solutions adaptable to different operational requirements.

Keywords: Humanoid Robot Systems, Engineering for Robotic Systems, Mechanism Design
Abstract: Almost all (not to say all) existing humanoids have not been designed with the ability to change their end-effectors (head, feet and hands) on-the-fly. Inspired by the tool-changing mechanisms in robotics automation and manufacturing, we propose an end-effectors changer mechanism suitable to humanoids. This letter explains why enabling humanoids with such a technology is important and why existing tool-changer mechanisms are not adapted. The proposed changer mechanism is not actuated. Yet, it does not require human intervention to assist the change of end-effectors. We assess our mechanism through a comparative study with existing tool-changers and demonstrations with the HRP-4 humanoid. We claim that our idea could be a new turn in the design of future humanoids and open perspective to modular sizing of robots. The proposed mechanism can possibly apply to animaloids to some extent.

Keywords: Humanoid Robot Systems, Reinforcement Learning, Motion Control
Abstract: Human motion retargeting for humanoid robots, transferring human motion data to robots for imitation, presents significant challenges but offers considerable potential for real-world applications. Traditionally, this process relies on human demonstrations captured through pose estimation or motion capture systems. In this paper, we explore a text-driven approach to obtain imitation motion data more flexibly and simply. To address the inherent discrepancies between the generated motion representations and the kinematic constraints of humanoid robots, we propose an angle signal network based on normposition and rotation loss (NPR Loss). It generates joint angles, which serve as inputs to a reinforcement learningbased whole-body motion control policy. The policy ensures tracking of the generated motions while maintaining the robot's stability during execution. Our experimental results demonstrate the efficacy of this approach, successfully transferring text-driven human motion to a real humanoid robot NAO.

Keywords: Humanoid Robot Systems, Software Tools for Robot Programming, Mechanism Design
Abstract: Sign language serves individuals with hearing impairments as a crucial communication mode operating through visual-manual means. While there has been established theory and agreement about embodiment in multiple fields, only limited research has deeply engaged to lower access to the physical body for spatial perception and engagement. Embodied robots are often cost-prohibitive, and existing open- source robot fabrication packages are limited in their ability to fully address communication nuances, typically running only on predefined programs. Reprogramming for broader bodily interactions, such as gestures in various domains (e.g., construction), is nearly impossible unless expertise precedes.

We introduce FABRIC, an end-to-end toolkit for fabricating and programming bodily language for unique human-robot interactions. The toolkit includes a fully 3D-printable robot, designed for consumer-grade FDM machinery, that learns from demonstration (LfD) to capture and translate users’ bodily expressions through its upper torso (arms and hands) movements. A visual programming interface enables appending or sequencing demonstrations from various sources, i.e., videos, cameras, and expandable word/phrase/sentence libraries.

Keywords: Optimization and Optimal Control, Contact Modeling, Whole-Body Motion Planning and Control
Abstract: We present a sampling-based model predictive control method that uses a generic physics simulator as the dynamical model. In particular, we propose a Model Predictive Path Integral controller (MPPI) that employs the GPU-parallelizable IsaacGym simulator to compute the forward dynamics of the robot and environment. Since the simulator implicitly defines the dynamic model, our method is readily extendable to different objects and robots, allowing one to solve complex navigation and contact-rich tasks. We demonstrate the effectiveness of this method in several simulated and real-world settings, including mobile navigation with collision avoidance, non-prehensile manipulation, and whole-body control for high-dimensional configuration spaces. This is a powerful and accessible open-source tool to solve many contact-rich motion planning tasks.

Keywords: Optimization and Optimal Control, Machine Learning for Robot Control
Abstract: Many robotics tasks, such as path planning or trajectory optimization, are formulated as optimal control problems (OCPs). The key to obtaining high performance lies in the design of the OCP's objective function. In practice, the objective function consists of a set of individual components that must be carefully modeled and traded off such that the OCP has the desired solution. It is often challenging to balance multiple components to achieve the desired solution and to understand, when the solution is undesired, the impact of individual cost components. In this paper, we present a framework addressing these challenges based on the concept of directional corrections. Specifically, given the solution to an OCP that is deemed undesirable, and access to an expert providing the direction of change that would increase the desirability of the solution, our method analyzes the individual cost components for their consistency with the provided directional correction. This information can be used to improve the OCP formulation, e.g., by increasing the weight of consistent cost components, or reducing the weight of -- or even redesigning -- inconsistent cost components. We also show that our framework can automatically tune parameters of the OCP to achieve consistency with a set of corrections.

Keywords: Optimization and Optimal Control, Motion and Path Planning, Machine Learning for Robot Control
Abstract: Model Predictive Path Integral (MPPI) is a popular sampling-based Model Predictive Control (MPC) algorithm for nonlinear systems. It optimizes trajectories by sampling control sequences and averaging them. However, a key issue with MPPI is the non-smoothness of the optimal control sequence, leading to oscillations in systems like fixed-wing aerial vehicles (FWVs). Existing solutions use post-hoc smoothing, which fails to bound control derivatives. This paper introduces a new approach: we add a projection filter pi to minimally correct control samples, ensuring bounds on control magnitude and higher-order derivatives. The filtered samples are then averaged using MPPI, leading to our pi-MPPI approach. We minimize computational overhead by using a neural accelerated custom optimizer for the projection filter. pi-MPPI offers a simple way to achieve arbitrary smoothness in control sequences. While we focus on FWVs, this projection filter can be integrated into any MPPI pipeline. Applied to FWVs, pi-MPPI is easier to tune than the baseline, resulting in smoother, more robust performance.

Keywords: Optimization and Optimal Control, Parallel Robots, Robotics and Automation in Agriculture and Forestry
Abstract: In robotics, structural design and behavior optimization have long been considered separate processes, resulting in the development of systems with limited capabilities. Recently, co-design methods have gained popularity, where bi-level formulations are used to simultaneously optimize the robot design and behavior for specific tasks. However, most implementations assume a serial or tree-type model of the robot, overlooking the fact that many robot platforms incorporate parallel mechanisms. In this paper, we present a first co-design formulation that explicitly incorporates parallel coupling constraints into the dynamic model of the robot. In this framework, an outer optimization loop focuses on the design parameters, in our case the transmission ratios of a parallel belt-driven manipulator, which map the desired torques from the joint space to the actuation space. An inner loop performs trajectory optimization in the actuation space, thus exploiting the entire dynamic range of the manipulator. We compare the proposed method with a conventional co-design approach based on a simplified tree-type model. By taking advantage of the actuation space representation, our approach leads to a significant increase in dynamic payload capacity compared to the conventional co-design implementation.

Keywords: Multi-Robot Systems, Optimization and Optimal Control, Logistics
Abstract: Robotic systems are routinely used in the logistics industry to enhance operational efficiency, but the design of robot workspaces remains a complex and manual task, which limits the system's flexibility to changing demands. This paper aims to automate robot workspace design by proposing a computational framework to generate a budget-minimizing layout by selectively placing stationary robots on a floor grid to sort packages from given input and output locations. Finding a good layout that minimizes the hardware budget while ensuring motion feasibility is a challenging combinatorial problem with nonconvex motion constraints. We propose a new optimization-based approach that models layout planning as a subgraph optimization problem subject to network flow constraints. Our core insight is to abstract away motion constraints from the layout optimization by precomputing a kinematic reachability graph and then extract the optimal layout on this ground graph. We validate the motion feasibility of our approach by proposing a simple task assignment and motion planning technique. We benchmark our algorithm on problems with various grid resolutions and number of outputs and show improvements in memory efficiency over a heuristic search algorithm. In addition, we demonstrate that our algorithm can be extended to handle various types of robot manipulators and conveyor belts, box payload constraints, and cost assignments.

Keywords: Optimization and Optimal Control, Aerial Systems: Mechanics and Control, Aerial Systems: Applications
Abstract: In this paper, we address the problem of tracking high-speed agile trajectories for Unmanned Aerial Vehicles (UAVs), where model inaccuracies can lead to large track- ing errors. Existing Nonlinear Model Predictive Controller (NMPC) methods typically neglect the dynamics of the low- level flight controllers such as underlying PID controller present in many flight stacks, and this results in suboptimal tracking performance at high speeds and accelerations. To this end, we propose a novel NMPC formulation, LoL-NMPC, which explicitly incorporates low-level controller dynamics and motor dynamics in order to minimize trajectory tracking errors while maintaining computational efficiency. By leveraging linear constraints inside low-level dynamics, our approach inherently accounts for actuator constraints without requiring additional reallocation strategies. The proposed method is validated in both simulation and real-world experiments, demonstrating improved tracking accuracy and robustness at speeds up to 98.57 km h−1 and accelerations of 3.5 g. Our results show an average 21.97 % reduction in trajectory tracking error over standard NMPC formulation, with LoL-NMPC maintaining real-time feasibility at 100 Hz on an embedded ARM-based flight computer.

Keywords: Optimization and Optimal Control, Motion and Path Planning, Autonomous Vehicle Navigation
Abstract: Optimizing trajectory costs for nonlinear control systems remains a significant challenge. Model Predictive Control (MPC), particularly sampling-based approaches such as the Model Predictive Path Integral (MPPI) method, has recently demonstrated considerable success by leveraging parallel computing to efficiently evaluate numerous trajectories. However, MPPI often struggles to balance safe navigation in constrained environments with effective exploration in open spaces, leading to infeasibility in cluttered conditions. To address these limitations, we propose DBaS-Log-MPPI, a novel algorithm that integrates Discrete Barrier States (DBaS) to ensure safety while enabling adaptive exploration with enhanced feasibility. Our method is efficiently validated through three simulation missions and one real-world experiment, involving a 2D quadrotor and a ground vehicle navigating through cluttered obstacles. We demonstrate that our algorithm surpasses both Vanilla MPPI and Log-MPPI, achieving higher success rates, lower tracking errors, and a conservative average speed.

Keywords: Robotics and Automation in Agriculture and Forestry, Agricultural Automation, Motion and Path Planning
Abstract: Autonomous agricultural vehicles (AAVs), including field robots and autonomous tractors, are becoming essential in modern farming by improving efficiency and reducing labor costs. A critical task in AAV operations is headland turning between crop rows. This task is challenging in orchards with limited headland space, irregular boundaries, operational constraints, and static obstacles. While traditional trajectory planning methods work well in arable farming, they often fail in cluttered orchard environments. This letter presents a novel trajectory planner that enhances the safety and efficiency of AAV headland maneuvers, leveraging advancements in autonomous driving. Our approach includes an efficient front-end algorithm and a high-performance back-end optimization. Applied to vehicles with various implements, it outperforms state-of-the-art methods in both standard and challenging orchard fields. This work bridges agricultural and autonomous driving technologies, facilitating a broader adoption of AAVs in complex orchards.

Keywords: Robotics and Automation in Agriculture and Forestry, Reactive and Sensor-Based Planning, Simulation and Animation
Abstract: Three-dimensional (3D) Plant Phenotyping enables comprehensive trait analysis for evaluating plant growth in precision agriculture. Current phenotyping frameworks are low-throughput due to frequent manual intervention and inefficiencies in handling large volumes of repetitive data. Existing view planners for phenotyping are limited to static sampling methods or narrow focus on specific plant organs, restricting their utility in capturing the full complexity of plant structures. To address these limitations, we propose SemP-NBV, a novel semantic-aware predictive next-best-view approach for sample-efficient 3D plant phenotyping. Evaluated in a photorealistic simulator with 12 plant categories, SemP-NBV achieves 15.3% more observed points than an even-space sampler at 8 images on average, while matching the reconstruction quality at 20 even-space images. We demonstrate that existing state-of-the-art predictive planners designed for artificial structures struggle with zero performance increase compared to even-space sampler for complex plant structures. Furthermore, our approach generates semantic information during reconstruction, reducing the need for post hoc semantic labeling, and streamlining the 3D phenotyping workflow. The project is available at https://github.com/ARLabXiang/SemP-NBV.

Keywords: Robotics and Automation in Agriculture and Forestry, Agricultural Automation, Manipulation Planning
Abstract: This study presents a methodology to safely manipulate branches to aid various agricultural tasks. Humans in a real agricultural environment often manipulate branches to perform agricultural tasks effectively, but current agricultural robots lack this capability. This proposed strategy to manipulate branches can aid in different precision agriculture tasks, such as fruit picking in dense foliage, pollinating flowers under occlusion, and moving overhanging vines and branches for navigation. The proposed method modifies RRT* to plan a path that satisfies the branch geometric constraints and obeys branch deformable characteristics. Re-planning is done to obtain a path that helps the robot exert force within a desired range so that branches are not damaged during manipulation. Experimentally, this method achieved a success rate of 78% across 50 trials, successfully moving a branch from the starting point to the target region.

Keywords: Robotics and Automation in Agriculture and Forestry, Data Sets for Robotic Vision
Abstract: Crop yield estimation is a relevant problem in agriculture, because an accurate yield estimate can support farmers’ decisions on harvesting or precision intervention. Robots can help to automate this process. To do so, they need to be able to perceive the surrounding environment to identify target objects such as trees and plants. In this paper, we introduce a novel approach to address the problem of hierarchical panoptic segmentation of apple orchards on 3D data from different sensors. Our approach is able to simultaneously provide semantic segmentation, instance segmentation of trunks and fruits, and instance segmentation of trees (a trunkwith its fruits). This allows us to identify relevant information such as individual plants, fruits, and trunks, and capture the relationship among them, such as precisely estimate the number of fruits associated to each tree in an orchard. To efficiently evaluate our approach for hierarchical panoptic segmentation, we provide a dataset designed specifically for this task. Our dataset is recorded in Bonn, Germany, in a real apple orchard with a variety of sensors, spanning from a terrestrial laser scanner to a RGB-D camera mounted on different robots platforms. The experiments show that our approach surpasses state-of-the-art approaches in 3D panoptic segmentation in the agricultural domain, while also providing full hierarchical panoptic segmentation. Our dataset is publicly available at https://www.ipb.uni-bonn.de/data/hops/. The open-source implementation of our approach is available at https://github.com/PRBonn/hapt3D.

Keywords: Agricultural Automation, Planning, Scheduling and Coordination, Field Robots
Abstract: Plant factory cultivation is widely recognized for its ability to optimize resource use and boost crop yields. To further increase the efficiency in these environments, we propose a general mixed-integer linear programming (MILP) framework that systematically schedules and coordinates dual-arm harvesting tasks, minimizing the overall harvesting makespan based on pre-mapped fruit locations. Specifically, we focus on a specialized dual-arm harvesting robot and employ pose coverage analysis of its end effector to maximize picking reachability. Additionally, we compare the performance of the dual-arm configuration with that of a single-arm vehicle, demonstrating that the dual-arm system can nearly double efficiency when fruit densities are roughly equal on both sides. Extensive simulations show a 10–20% increase in throughput and a significant reduction in the number of stops compared to nonoptimized methods. These results underscore the advantages of an optimal scheduling approach in improving the scalability and efficiency of robotic harvesting in plant factories.

Keywords: Agricultural Automation, Planning, Scheduling and Coordination, Reinforcement Learning
Abstract: The Entrance Dependent Vehicle Routing Problem (EDVRP) is a variant of the Vehicle Routing Problem (VRP) where the scale of cities influences routing outcomes, necessitating consideration of their entrances. This paper addresses EDVRP in agriculture, focusing on multi-parameter vehicle planning for irregularly shaped fields. To address the limitations of traditional methods, such as heuristic approaches, which often overlook field geometry and entrance constraints, we propose a Joint Probability Distribution Sampling Neural Network (JPDS-NN) to effectively solve the EDVRP. The network uses an encoder-decoder architecture with graph transformers and attention mechanisms to model routing as a Markov Decision Process, and is trained via reinforcement learning for efficient and rapid end-to-end planning. Experimental results indicate that JPDS-NN reduces travel distances by 48.4–65.4%, lowers fuel consumption by 14.0–17.6%, and computes two orders of magnitude faster than baseline methods, while demonstrating 15–25% superior performance in dynamic arrangement scenarios. Ablation studies validate the necessity of cross-attention and pre-training. The framework enables scalable, intelligent routing for large-scale farming under dynamic constraints.

Keywords: Computational Geometry, Visual Servoing, Agricultural Automation
Abstract: Robotization is considered a key solution to labor shortages in the agri-food industry. However, deploying robots in natural environments is challenging due to unpredictable factors such as plant variances and occlusions. This paper focuses on the localization of tomato trusses for autonomous harvesting by servoing a robot-mounted camera to different viewpoints. We build on previous work where the robot is provided with prior knowledge of the tomato plant. Specifically, the geometric relations between the trusses are modeled as ranges, which reflect uncertainty. Our main contribution is an approach that represents this uncertainty as polytope volumes. Polytopes enable scalable reasoning that facilitates likelihood estimation for viewpoint selection. Our method first constructs polytope hypotheses regarding the truss locations based on prior plant knowledge. It then refines the polytope shapes using Bayesian updates based on camera observations. Finally, the polytopes are used to select the next viewpoint that maximizes the chance of observing a new tomato truss. Experiments show that polytope-based viewpoint selection speeds up truss localization compared to earlier methods, advancing robotic harvesting.

Keywords: Sensor Fusion, AI-Based Methods
Abstract: The effective fusion of infrared and visible images could ehance environment perception during robot rescue mission by combining complementary information from both sensors. However, most existing fusion methods are developed for images captured under normal conditions, which limits their performance in real-world rescue scenarios where images often suffer from diverse degradation such as haze, low-light, noise, low-contrast, and so on. To address this challenge, we propose MT-Fusion, a novel deep learning workflow based on multi-task learning (MTL) that targets multiple specific degradation scenarios. MT-Fusion incorporates specific encoder modules for processing various degraded images, a degradation attention mechanism for fusion, and a shared decoder for image reconstruction. Extensive experiments demonstrate that our proposed specific scenario-guided image fusion strategy has obvious advantages in robot perception in the image fusion performance and degradation treatment. For the inference stage, we propose a selector that could automatically categorize the degradation of input and activate the specific encoder. The MT-Fusion also provides a more practical solution for enhancing robot rescue operations in challenging environments.

Keywords: Sensor Fusion, Autonomous Vehicle Navigation, Intelligent Transportation Systems
Abstract: Multi-sensor fusion is a key technology in the field of autonomous driving and robotics.Traditional offline multi-sensor fusion calibration methods rely on manual operations and fail to meet real-time requirements, while recent online calibration technologies have limited generalization capabilities.This paper proposes CalibMutiL, an end-to-end calibration network that departs from conventional deep feature fusion by leveraging multi-level RGB image features to guide point cloud alignment.CalibMutiL introduces a Multi-level Fusion module (MLF) that effectively utilizes the rich visual features of the image. In addition, we regard the alignment process as a sequence prediction problem and further improve the performance through an Iterative Refinement module (IRM).Evaluation of the KITTI odometry and raw dataset demonstrates the average calibration error reaches 0.81 cm and 0.09°. The generalization tests resulted in errors of 4.24cm and 0.13°, outperforming existing methods. Our implementation will be publicly available at https://github.com/VIP-G/CalibMutiL.

Keywords: Sensor Fusion, AI-Based Methods, Deep Learning Methods
Abstract: Wireless Electromagnetic Tracking (WEMT) enables non-line-of-sight (NLoS) pose estimation in robotics but faces accuracy limitations from restricted operational range and environmental interference. This paper proposes a WEMT-inertial fusion system enhanced by a learning-based Interacting Multiple Model (IMMNN) to address these challenges. The framework integrates a multi-transmitter array with a WEMT-IMU fusion tracker, leveraging IMMNN to mitigate performance degradation caused by nonlinear spatial noise and motion uncertainty in dynamic, array-based environments. IMMNN employs a graph attention network to dynamically model spatial correlations among array units, adaptively optimizing state transition probabilities across motion models. A gated recurrent framework further enhances robustness by analyzing residual sequences to suppress transient noise and outliers. Experimental results demonstrate that the proposed system achieves a root-mean-square error (RMSE) of 30.4 mm over an expanded 1.9×1.9 m² operational area. The graph attention mechanism enables adaptive spatial noise suppression and ensures stable tracking under rapid motion and electromagnetic disturbances. By synergizing model-driven filtering with data-driven learning, IMMNN effectively improves accuracy and robustness, advancing high-precision WEMT solutions for complex robotic applications.

Keywords: Localization, Sensor Fusion, SLAM
Abstract: Existing LiDAR-Inertial Odometry (LIO) systems typically use sensor-specific or environment-dependent measurement covariances during state estimation, leading to laborious parameter tuning and suboptimal performance in challenging conditions (e.g., sensor degeneracy, noisy observations). Therefore, we propose an Adaptive Kalman Filter (AKF) framework that dynamically estimates time-varying noise covariances of LiDAR and Inertial Measurement Unit (IMU) measurements, enabling context-aware confidence weighting between sensors. During LiDAR degeneracy, the system prioritizes IMU data while suppressing contributions from unreliable inputs like moving objects or noisy point clouds. Furthermore, a compact Gaussian-based map representation is introduced to model environmental planarity and spatial noise. A correlated registration strategy ensures accurate plane normal estimation via pseudo-merge, even in unstructured environments like forests. Extensive experiments validate the robustness of the proposed system across diverse environments, including dynamic scenes and geometrically degraded scenarios. Our method achieves reliable localization results across all MARS-LVIG sequences and ranks 8th on the KITTI Odometry Benchmark. The code will be released at https://github.com/xpxie/AKF-LIO.git.

Keywords: Marine Robotics, Sensor Fusion, Mapping
Abstract: Scene reconstruction is an essential capability for underwater robots navigating in close proximity to structures. Monocular vision-based reconstruction methods are unreliable in turbid waters and lack depth scale information. Sonars are robust to turbid water and non-uniform lighting conditions, however, they have low resolution and elevation ambiguity. This work proposes a real-time opti-acoustic scene reconstruction method that is specially optimized to work in turbid water. Our strategy avoids having to identify point features in visual data and instead identifies regions of interest in the data. We then match relevant regions in the image to corresponding sonar data. A reconstruction is obtained by leveraging range data from the sonar and elevation data from the camera image. Experimental comparisons against other vision-based and sonar-based approaches at varying turbidity levels, and field tests conducted in marina environments, validate the effectiveness of the proposed approach. We have made our code open-source to facilitate reproducibility and encourage community engagement.

Keywords: Sensor Fusion, Computer Vision for Automation, SLAM
Abstract: This work introduces BEV-LIO(LC), a novel LiDAR-Inertial Odometry (LIO) framework that combines Bird's Eye View (BEV) image representations of LiDAR data with geometry-based point cloud registration and incorporates loop closure (LC) through BEV image features. By normalizing point density, we project LiDAR point clouds into BEV images, thereby enabling efficient feature extraction and matching. A lightweight convolutional neural network (CNN) based feature extractor is employed to extract distinctive local and global descriptors from the BEV images. Local descriptors are used to match BEV images with FAST keypoints for reprojection error construction, while global descriptors facilitate loop closure detection. Reprojection error minimization is then integrated with point-to-plane registration within an iterated Extended Kalman Filter (iEKF). In the back-end, global descriptors are used to create a KD-tree-indexed keyframe database for accurate loop closure detection. When a loop closure is detected, Random Sample Consensus (RANSAC) computes a coarse transform from BEV image matching, which serves as the initial estimate for Iterative Closest Point (ICP). The refined transform is subsequently incorporated into a factor graph along with odometry factors, improving the global consistency of localization. Extensive experiments conducted in various scenarios with different LiDAR types demonstrate that BEV-LIO(LC) outperforms state-of-the-art methods, achieving competitive localization accuracy. Our code and video can be found at url{https://github.com/HxCa1/BEV-LIO-LC}.

Keywords: Sensor Fusion, Autonomous Vehicle Navigation, Deep Learning for Visual Perception
Abstract: High-definition (HD) map construction methods are crucial for providing precise and comprehensive static environmental information, which is essential for autonomous driving systems. While Camera-LiDAR fusion techniques have shown promising results by integrating data from both modalities, existing approaches primarily focus on improving model accuracy, often neglecting the robustness of perception models—a critical aspect for real-world applications. In this paper, we explore strategies to enhance the robustness of multi-modal fusion methods for HD map construction while maintaining high accuracy. We propose three key components: data augmentation, a novel multi-modal fusion module, and a modality dropout training strategy. These components are evaluated on a challenging dataset containing 13 types of multi-sensor corruption. Experimental results demonstrate that our proposed modules significantly enhance the robustness of baseline methods. Furthermore, our approach achieves state-of-the-art performance on the clean validation set of the NuScenes dataset. Our findings provide valuable insights for developing more robust and reliable HD map construction models, advancing their applicability in real-world autonomous driving scenarios.

Keywords: Mapping, Semantic Scene Understanding
Abstract: In recent years, vision-language models (VLMs) have advanced open-vocabulary mapping, enabling mobile robots to simultaneously achieve environmental reconstruction and high-level semantic understanding. While integrated object cognition helps mitigate semantic ambiguity in point-wise feature maps, efficiently obtaining rich semantic understanding and robust incremental reconstruction at the instance-level remains challenging. To address these challenges, we introduce OpenVox, a real-time incremental open-vocabulary probabilistic instance voxel representation. In the front-end, we design an efficient instance segmentation and comprehension pipeline that enhances language reasoning through encoding captions. In the back-end, we implement probabilistic instance voxels and formulate the cross-frame incremental fusion process into two subtasks: instance association and live map evolution, ensuring robustness to sensor and segmentation noise. Extensive evaluations across multiple datasets demonstrate that OpenVox achieves state-of-the-art performance in zero-shot instance segmentation, semantic segmentation, and open-vocabulary retrieval. The project page of OpenVox is available at https://open-vox.github.io/.