Keywords: Task Planning, AI-Based Methods, Planning, Scheduling and Coordination
Abstract: Recent advancements in Large Language Models (LLMs) have sparked a revolution across many research fields. In robotics, the integration of common-sense knowledge from LLMs into task and motion planning has drastically advanced the field by unlocking unprecedented levels of context awareness. Despite their vast collection of knowledge, large language models may generate infeasible plans due to hallucinations or missing domain information. To address these challenges and improve plan feasibility and computational efficiency, we introduce DELTA, a novel LLM-informed task planning approach. By using scene graphs as environment representations within LLMs, DELTA achieves rapid generation of precise planning problem descriptions. To enhance planning performance, DELTA decomposes long-term task goals with LLMs into an autoregressive sequence of sub-goals, enabling automated task planners to efficiently solve complex problems. In our extensive evaluation, we show that DELTA enables an efficient and fully automatic task planning pipeline, achieving higher planning success rates and significantly shorter planning times compared to the state of the art.

Keywords: Motion and Path Planning, Human-Centered Robotics, Human-Aware Motion Planning
Abstract: Robot navigation methods allow mobile robots to operate in applications such as warehouses or hospitals. While the environment in which the robot operates imposes requirements on its navigation behavior, most existing methods do not allow the end-user to configure the robot's behavior and priorities, possibly leading to undesirable behavior (e.g., fast driving in a hospital). We propose a novel approach to adapt robot motion behavior based on natural language instructions provided by the end-user. Our zero-shot method uses an existing Visual Language Model to interpret a user text query or an image of the environment. This information is used to generate the cost function and reconfigure the parameters of a Model Predictive Controller, translating the user's instruction to the robot's motion behavior. This allows our method to safely and effectively navigate in dynamic and challenging environments. We extensively evaluate our method's individual components and demonstrate the effectiveness of our method on a ground robot in simulation and real-world experiments, and across a variety of environments and user specifications.

Keywords: Task Planning, Computer Vision for Automation, Robotics and Automation in Life Sciences
Abstract: Recent advances in large language models (LLMs) have demonstrated exceptional reasoning capabilities in natural language processing, sparking interest in applying LLMs to task planning problems in robotics. Most studies focused on task planning for clear natural language commands that specify target objects and their locations. However, for more user-friendly task execution, it is crucial for robots to autonomously plan and carry out tasks based on abstract natural language commands that may not explicitly mention target objects or locations, such as ‘Put the food ingredients in the same place.’ In this study, we propose an LLM-based autonomous task planning framework that generates task plans for abstract natural language commands. This framework consists of two phases: an environment recognition phase and a task planning phase. In the environment recognition phase, a large vision-language model generates a hierarchical scene graph that captures the relationships between objects and spaces in the environment surrounding a robot agent. During the task planning phase, an LLM uses the scene graph and the abstract user command to formulate a plan for the given task. We validate the effectiveness of the proposed framework in the AI2THOR simulation environment, demonstrating its superior performance in task execution when handling abstract commands.

Keywords: Task Planning
Abstract: In robot task planning, large language models (LLMs) have shown significant promise in generating complex and long-horizon action sequences. However, it is observed that LLMs often produce responses that sound plausible but are not accurate. To address these problems, existing methods typically employ predefined error sets or external knowledge sources, requiring human efforts and computation resources. Recently, self-correction approaches have emerged, where LLM generates and refines plans, identifying errors by itself. Despite their effectiveness, they are more prone to failures in correction due to insufficient reasoning. In this paper, we propose a novel self-corrective planning of tasks with inverse prompting, named InversePrompt, which contains reasoning steps to provide interpretable groundings for feedback. It generates the inverse actions corresponding to generated actions and verifies if these inverse actions can restore the system to its original state, thereby explicitly validating the logical flow of the generated plans. The results on benchmark datasets show an average 16.3% higher success rate over existing LLM-based task planning methods. Our approach offers clearer justifications for feedback in real-world environments, resulting in more successful task completion than existing self-correction approaches across various scenarios.

Keywords: Motion and Path Planning, Integrated Planning and Control, Planning under Uncertainty
Abstract: This paper introduces and tests a framework that integrates traffic regulation compliance into automated driving systems (ADS). The framework enables ADS to follow traffic laws and make informed decisions based on the driving environment. Using RGB camera inputs and a vision-language model (VLM), the system generates descriptive text to support a regulation-aware decision-making process, ensuring legal and safe driving practices. This information is combined with a machine-readable ADS regulation database to guide future driving plans within legal constraints. Key features include: 1) a regulation database supporting ADS decision-making, 2) an automated process using sensor input for regulation-aware path planning, and 3) validation in both simulated and real-world environments. Particularly, the real-world vehicle tests not only assess the framework's performance but also evaluate the potential and challenges of VLMs to solve complex driving problems by integrating detection, reasoning, and planning. This work enhances the legality, safety, and public trust in ADS, representing a significant step forward in the field.

Keywords: Integrated Planning and Learning, Motion and Path Planning, Intelligent Transportation Systems
Abstract: Motion planners are essential for the safe operation of automated vehicles across various scenarios. However, no motion planning algorithm has achieved perfection in the literature, and improving its performance is often time-consuming and labor-intensive. To tackle the aforementioned issues, we present DrPlanner, the first framework designed to automatically diagnose and repair motion planners using large language models. Initially, we generate a structured description of the planner and its planned trajectories from both natural and programming languages. Leveraging the profound capabilities of large language models in addressing reasoning challenges, our framework returns repaired planners with detailed diagnostic descriptions. Furthermore, the framework advances iteratively with continuous feedback from the evaluation of the repaired outcomes. Our approach is validated using both search- and sampling-based motion planners for automated vehicles; experimental results highlight the need for demonstrations in the prompt and show the ability of our framework to effectively identify and rectify elusive issues.

Keywords: SLAM, Mapping
Abstract: This letter introduces a novel framework for dense Visual Simultaneous Localization and Mapping (VSLAM) based on Gaussian Splatting. Recently, SLAM based on Gaussian Splatting has shown promising results. However, in monocular scenarios, the Gaussian maps reconstructed lack geometric accuracy and exhibit weaker tracking capability. To address these limitations, we jointly optimize sparse visual odometry tracking and 3D Gaussian Splatting scene representation for the first time. Estimating depth maps on visual odometry keyframes window using a fast Multi-View Stereo (MVS) network for the geometric supervision of Gaussian maps. Furthermore, we propose a depth smooth loss and Sparse-Dense Adjustment Ring (SDAR) to reduce the negative effect of estimated depth maps and preserve the consistency in scale between the visual odometry and Gaussian maps. We have evaluated our system across various synthetic and real-world datasets. The accuracy of our poses estimation surpasses existing methods and achieves state-of-the-art. Additionally, it outperforms previous monocular methods in terms of novel view synthesis and geometric reconstruction fidelities.

Keywords: SLAM, Mapping, Localization
Abstract: The 3D Gaussian Splatting (3DGS)-based SLAM system has garnered widespread attention due to its excellent performance in real-time high-fidelity rendering. However, in real-world environments filled with dynamic objects, existing 3DGS-based SLAM systems often face mapping errors and tracking drift issues. To address this, we propose GARAD-SLAM, a real-time 3DGS-based SLAM system tailored for dynamic scenes. In terms of tracking, unlike traditional methods, we directly perform dynamic segmentation on Gaussians and map them back to the front end to obtain dynamic point labels through a Gaussian pyramid network, achieving precise dynamic removal and robust tracking. For mapping, we impose rendering penalties on dynamically labeled Gaussians updated through the network to avoid irreversible erroneous removal caused by simple pruning. Our results on real-world datasets demonstrate that our method is competitive in tracking compared to baseline methods, generating fewer artifacts and higher-quality reconstructions in rendering.

Keywords: SLAM, Localization, Mapping
Abstract: The joint optimization of Neural Radiance Fields (NeRF) and camera trajectories has been widely applied in SLAM tasks due to its superior dense mapping quality and consistency. NeRF-based SLAM learns camera poses using constraints by implicit map representation. A widely observed phenomenon that results from the constraints of this form is jerky and physically unrealistic estimated camera motion, which in turn affects the map quality. To address this deficiency of current NeRF-based SLAM, we propose in this paper TS-SLAM (TS for Trajectory Smoothness). It introduces smoothness constraints on camera trajectories by representing them with uniform cubic B-splines with continuous acceleration that guarantees smooth camera motion. Benefiting from the differentiability and local control properties of B-splines, TS-SLAM can incrementally learn the control points end-to-end using a sliding window paradigm. Additionally, we regularize camera trajectories by exploiting the dynamics prior to further smooth trajectories. Experimental results demonstrate that TS-SLAM achieves superior trajectory accuracy and improves mapping quality versus NeRF-based SLAM that does not employ the above smoothness constraints.

Keywords: SLAM, Mapping, Vision-Based Navigation
Abstract: Real-time SLAM with dense 3D mapping is computationally challenging, especially on resource-limited devices. The recent development of 3D Gaussian Splatting (3DGS) offers a promising approach for real-time dense 3D reconstruction. However, existing 3DGS-based SLAM systems struggle to balance hardware simplicity, speed, and map quality. Most systems excel in one or two of the aforementioned aspects but rarely achieve all. A key issue is the difficulty of initializing 3D Gaussians while concurrently conducting SLAM. To address these challenges, we present Monocular GSO (MGSO), a novel real-time SLAM system that integrates photometric SLAM with 3DGS. Photometric SLAM provides dense structured point clouds for 3DGS initialization, accelerating optimization and producing more efficient maps with fewer Gaussians. As a result, experiments show that our system generates reconstructions with a balance of quality, memory efficiency, and speed that outperforms the state-of-the-art. Furthermore, our system achieves all results using RGB inputs. We evaluate the Replica, TUM-RGBD, and EuRoC datasets against current live dense reconstruction systems. Not only do we surpass contemporary systems, but experiments also show that we maintain our performance on laptop hardware, making it a practical solution for robotics, A/R, and other real-time applications.

Keywords: Deep Learning for Visual Perception, Visual Learning, SLAM
Abstract: 3D Gaussian Splatting (3DGS) has become a popular solution in SLAM, as it can produce high-fidelity novel views. However, previous GS-based methods primarily target indoor scenes and rely on RGB-D sensors or pre-trained depth estimation models, hence underperforming in outdoor scenarios. To address this issue, we propose a RGB-only gaussian splatting SLAM method for unbounded outdoor scenes—OpenGS-SLAM. Technically, we first employ a pointmap regression network to generate consistent pointmaps between frames for pose estimation. Compared to commonly used depth maps, pointmaps include spatial relationships and scene geometry across multiple views, enabling robust camera pose estimation. Then, we propose integrating the estimated camera poses with 3DGS rendering as an end-to-end differentiable pipeline. Our method achieves simultaneous optimization of camera poses and 3DGS scene parameters, significantly enhancing system tracking accuracy. Specifically, we also design an adaptive scale mapper for the pointmap regression network, which provides more accurate pointmap mapping to the 3DGS map representation. Our experiments on the Waymo dataset demonstrate that OpenGS-SLAM reduces tracking error to 9.8% of previous 3DGS methods, and achieves state-of-the-art results in novel view synthesis. Project page: https://opengsslam.github.io/.

Keywords: Mapping, SLAM, Embodied Cognitive Science
Abstract: Visual SLAM has regained attention due to its ability to provide perception capabilities and simulation test data for Embodied AI. However, traditional SLAM systems struggle to meet the demands of high-quality scene reconstruction, and Gaussian SLAM systems, despite their rapid rendering and high-quality mapping capabilities, lack effective pose optimization methods and face challenges in geometric reconstruction. To address these issues, we introduce FGO-SLAM, a Gaussian SLAM system that employs an opacity radiance field as the scene representation to enhance geometric mapping performance. After initial pose estimation, we apply global adjustment to optimize camera poses and sparse point cloud, ensuring robust tracking of our system. Additionally, we maintain a globally consistent opacity radiance field based on 3D Gaussians and introduce depth distortion and normal consistency terms to refine the scene representation. Furthermore, after constructing tetrahedral grids, we identify level sets to directly extract surfaces from 3D Gaussians. Results across various real-world and large-scale synthetic datasets demonstrate that our method achieves state-of-the-art tracking accuracy and mapping performance.

Keywords: RGB-D Perception, Computer Vision for Automation, Data Sets for Robotic Vision
Abstract: Point cloud registration is a fundamental task in intelligent robots, aiming to achieve globally consistent geometric structures and providing data support for robotic manipulation. Due to the limited view of measurement devices, it is necessary to collect point clouds from multiple views to construct a complete model. Previous multi-view registration methods rely on sufficient overlap and registering all pairs of point clouds, resulting in slow convergence and high cumulative errors. To solve these challenges, we present a multi-view registration method based on the point-to-plane model and pose graph. We introduce a robust kernel into the objective function to diminish registration errors caused by mismatched points. Additionally, an enhanced Euclidean clustering method is proposed for extracting object point clouds. Subsequently, by establishing pose constraints on non-adjacent frames of point clouds, the cumulative error is reduced, achieving global optimization based on the pose graph. Experimental results demonstrate the robustness of our method with respect to overlap ratios, successfully registering point clouds with overlap ratio exceeding 30%. In comparison to other techniques, our method can reduce the E(R) of multi-view registration by 13.54% and E(t) by 18.72%, effectively reducing the cumulative error.

Keywords: Localization, Mapping
Abstract: LiDAR odometry is essential for many robotics applications, including 3D mapping, navigation, and simultaneous localization and mapping. LiDAR odometry systems are usually based on some form of point cloud registration to compute the ego-motion of a mobile robot. Yet, few of today's LiDAR odometry systems consider domain-specific knowledge or the kinematic model of the mobile platform during the point cloud alignment. In this paper, we present Kinematic-ICP, a LiDAR odometry system that focuses on wheeled mobile robots equipped with a 3D LiDAR and moving on a planar surface, which is a common assumption for warehouses, offices, hospitals, etc. Our approach introduces kinematic constraints within the optimization of a traditional point-to-point iterative closest point scheme. In this way, the resulting motion follows the kinematic constraints of the platform, effectively exploiting the robot's wheel odometry and the 3D LiDAR observations. We dynamically adjust the influence of LiDAR measurements and wheel odometry in our optimization scheme, allowing the system to handle degenerate scenarios such as feature-poor corridors. We evaluate our approach on robots operating in large-scale warehouse environments, but also outdoors. The experiments show that our approach achieves top performances and is more accurate than wheel odometry and common LiDAR odometry systems. Kinematic-ICP has been recently deployed in the Dexory fleet of robots operating in warehouses worldwide at their customers' sites, showing that our method can run in the real world alongside a complete navigation stack.

Keywords: Computer Vision for Medical Robotics, Representation Learning, Medical Robots and Systems
Abstract: Point cloud registration aims to provide estimated transformations to align 3D point clouds, which plays a crucial role in pose estimation of various navigation systems, such as surgical guidance systems and autonomous vehicles. Despite the impressive performance of recent models on benchmark datasets, many rely on complex modules like KPConv and Transformers, which impose significant computational and memory demands. These requirements hinder their practical application, particularly in resource-constrained environments such as mobile robotics. In this paper, we propose a novel point cloud registration network that leverages a pure MLP architecture, constructing geometric information offline. This approach eliminates the computational and memory burdens associated with traditional complex feature extractors and significantly reduces training time and resource consumption. Our method is the first to replace 3D coordinate inputs with offline-constructed geometric encoding, improving generalization and stability, as demonstrated by Maximum Mean Discrepancy (MMD) comparisons. This efficient and accurate geometric representation marks a significant advancement in point cloud analysis, particularly for applications requiring fast and reliable processing.

Keywords: Mapping, Localization, SLAM
Abstract: While global point cloud registration systems have advanced significantly in all aspects, many studies have focused on specific components, such as feature extraction, graph-theoretic pruning, or pose solvers. In this paper, we take a holistic view on the registration problem and develop an open-source and versatile C++ library for point cloud registration, called textit{KISS-Matcher}. textit{KISS-Matcher} combines a novel feature detector, textit{Faster-PFH}, that improves over the classical fast point feature histogram (FPFH). Moreover, it adopts a k-core-based graph-theoretic pruning to reduce the time complexity of rejecting outlier correspondences. Finally, it combines these modules in a complete, user-friendly, and ready-to-use pipeline. As verified by extensive experiments, KISS-Matcher has superior scalability and broad applicability, achieving a substantial speed-up compared to state-of-the-art outlier-robust registration pipelines while preserving accuracy. Our code will be available at href{https://github.com/MIT-SPARK/KISS-Matcher}{texttt{ht tps://github.com/MIT-SPARK/KISS-Matcher}}.

Keywords: RGB-D Perception, Mapping
Abstract: Point cloud registration is a critical problem in computer vision and robotics, especially in the field of navigation. Current methods often fail when faced with high outlier rates or take a long time to converge to a suitable solution. In this work, we introduce a novel algorithm for point cloud registration called SANDRO (Splitting strategy for point cloud Alignment using Non-convex anD Robust Optimization), which combines an Iteratively Reweighted Least Squares (IRLS) framework with a robust loss function with graduated non-convexity. This approach is further enhanced by a splitting strategy designed to handle high outlier rates and skewed distributions of outliers. SANDRO is capable of addressing important limitations of existing methods, as in challenging scenarios where the presence of high outlier rates and point cloud symmetries significantly hinder convergence. SANDRO achieves superior performance in terms of success rate when compared to the state-of-the-art methods, demonstrating a 20% improvement from the current state of the art when tested on the Redwood real dataset and 60% improvement when tested on synthetic data.

Keywords: RGB-D Perception, Vision-Based Navigation, Visual Tracking
Abstract: Gas concentration estimation is crucial for understanding and mitigating climate change. While most research and monitoring efforts focus on major greenhouse gases such as CO2, significantly less attention has been given to trace gases like NO2, which play a critical role in atmospheric chemistry and air quality. This paper aims to enhance trace gas concentration estimation by integrating physics-based models into data-driven neural network frameworks. Furthermore, to improve large-scale estimation accuracy, we incorporate in-situ measurements to refine neural network models trained on satellite observations. The resulting model can provide reliable large-scale gas concentration estimates, particularly for locations lacking precise in-situ measurements. This approach offers a novel pathway to enhance the accuracy and applicability of gas monitoring for climate and environmental research. While NO2 serves as the target trace gas in this study, the proposed framework is potentially applicable to the prediction of other atmospheric gas concentrations.

Keywords: Deep Learning for Visual Perception, Computer Vision for Automation, Computer Vision for Transportation
Abstract: Semantic segmentation is a key technique that enables mobile robots to understand and navigate surrounding environments autonomously. However, most existing works focus on segmenting known objects, overlooking the identification of unknown classes, which is common in real-world applications. In this paper, we propose a feature-oriented framework for open-set semantic segmentation on LiDAR data, capable of identifying unknown objects while retaining the ability to classify known ones. We design a decomposed dual-decoder network to simultaneously perform closed-set semantic segmentation and generate distinctive features for unknown objects. The network is trained with multi-objective loss functions to capture the characteristics of known and unknown objects. Using the extracted features, we introduce an anomaly detection mechanism to identify unknown objects. By integrating the results of close-set semantic segmentation and anomaly detection, we achieve effective feature-driven LiDAR open-set semantic segmentation. Evaluations on both SemanticKITTI and nuScenes datasets demonstrate that our proposed framework significantly outperforms state-of-the-art methods. The source code will be made publicly available at https://github.com/nubot-nudt/DOSS.

Keywords: Deep Learning for Visual Perception, Sensor Fusion
Abstract: Multi-modal 3D semantic segmentation is vital for applications such as autonomous driving and virtual reality (VR). To effectively deploy these models in real-world scenarios, it is essential to employ cross-domain adaptation techniques that bridge the gap between training data and real-world data. Recently, self-training with pseudo-labels has emerged as a predominant method for cross-domain adaptation in multi-modal 3D semantic segmentation. However, generating reliable pseudo-labels necessitates stringent constraints, which often result in sparse pseudo-labels after pruning. This sparsity can potentially hinder performance improvement during the adaptation process. We propose an image-guided pseudo-label enhancement approach that leverages the complementary 2D prior knowledge from the Segment Anything Model (SAM) to introduce more reliable pseudo-labels, thereby boosting domain adaptation performance. Specifically, given a 3D point cloud and the SAM masks from its paired image data, we collect all 3D points covered by each SAM mask that potentially belong to the same object. Then our method refines the pseudo-labels within each SAM mask in two steps. First, we determine the class label for each mask using majority voting and employ various constraints to filter out unreliable mask labels. Next, we introduce Geometry-Aware Progressive Propagation (GAPP) which propagates the mask label to all 3D points within the SAM mask while avoiding outliers caused by 2D-3D misalignment. Experiments conducted across multiple datasets and domain adaptation scenarios demonstrate that our proposed method significantly increases the quantity of high-quality pseudo-labels and enhances the adaptation performance over baseline methods.

Keywords: Deep Learning for Visual Perception, Learning Categories and Concepts, Visual Servoing
Abstract: In this paper, we perform robot manipulation activities in real-world environments with language contexts by integrating a compact referring image segmentation model into the robot's perception module. First, we propose CLIPU^2Net, a lightweight referring image segmentation model designed for fine-grain boundary and structure segmentation from language expressions. Then, we deploy the model in an eye-in-hand visual servoing system to enact robot control in the real world. The key to our system is the representation of salient visual information as geometric constraints, linking the robot’s visual perception to actionable commands. Experimental results on 46 real-world robot manipulation tasks demonstrate that our method outperforms traditional visual servoing methods relying on labor-intensive feature annotations, excels in fine-grain referring image segmentation with a compact decoder size of 6.6 MB, and supports robot control across diverse contexts.

Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Recognition
Abstract: In autonomous driving, thermal image semantic segmentation has emerged as a critical research area, owing to its ability to provide robust scene understanding under adverse visual conditions. In particular, unsupervised domain adaptation (UDA) for thermal image segmentation can be an efficient solution to address the lack of labeled thermal datasets. Nevertheless, since these methods do not effectively utilize the complementary information between RGB and thermal images, they significantly decrease performance during domain adaptation. In this paper, we present a comprehensive study on cross-spectral UDA for thermal image semantic segmentation. We first propose a novel masked mutual learning strategy that promotes complementary information exchange by selectively transferring results between each spectral model while masking out uncertain regions. Additionally, we introduce a novel prototypical self-supervised loss designed to enhance the performance of the thermal segmentation model in nighttime scenarios. This approach addresses the limitations of RGB pre-trained networks, which cannot effectively transfer knowledge under low illumination due to the inherent constraints of RGB sensors. In experiments, our method achieves higher performance over previous UDA methods and comparable performance to state-of-the-art supervised methods.

Keywords: Recognition, Object Detection, Segmentation and Categorization, Computer Vision for Automation
Abstract: Video segmentation is essential for advancing robotics and autonomous driving, particularly in open-world settings where continuous perception and object association across video frames are critical. While the Segment Anything Model (SAM) has excelled in static image segmentation, extending its capabilities to video segmentation poses significant challenges. We tackle two major hurdles: a) SAM’s embedding limitations in associating objects across frames, and b) granularity inconsistencies in object segmentation. To this end, we introduce VideoSAM, an end-to-end framework designed to address these challenges by improving object tracking and segmentation consistency in dynamic environments. VideoSAM integrates an agglomerated backbone, RADIO, enabling object association through similarity metrics and introduces Cycle-ack-Pairs Propagation with a memory mechanism for stable object tracking. Additionally, we incorporate an autoregressive object-token mechanism within the SAM decoder to maintain consistent granularity across frames. Our experiments on the UVO and BURST benchmark, and also robotic videos, demonstrate VideoSAM’s effectiveness and robustness in real-world scenarios. All codes will be available.

Keywords: Deep Learning for Visual Perception, Perception for Grasping and Manipulation, Object Detection, Segmentation and Categorization
Abstract: Transparent object perception is indispensable for numerous robotic tasks. However, accurately segmenting and estimating the depth of transparent objects remain challenging due to complex optical properties. Existing methods primarily delve into only one task using extra inputs or specialized sensors, neglecting the valuable interactions among tasks and the subsequent refinement process, leading to suboptimal and blurry predictions. To address these issues, we propose a monocular framework, which is the first to excel in both segmentation and depth estimation of transparent objects, with only a single-image input. Specifically, we devise a novel semantic and geometric fusion module, effectively integrating the multi-scale information between tasks. In addition, drawing inspiration from human perception of objects, we further incorporate an iterative strategy, which progressively refines initial features for clearer results. Experiments on two challenging synthetic and real-world datasets demonstrate that our model surpasses state-of-the-art monocular, stereo, and multi-view methods by a large margin of about 38.8%-46.2% with only a single RGB input. Codes and models are publicly available at https://github.com/L-J-Yuan/MODEST.

Keywords: Legged Robots, Biologically-Inspired Robots, Dynamics, Rough Terrain Locomotion
Abstract: Modeling and controlling legged robot locomotion on terrains with densely distributed large rocks and boulders are fundamentally challenging. Unlike traditional methods which often consider these rocks and boulders as obstacles and attempt to find a clear path to circumvent them, in this study we aim to develop methods for robots to actively utilize interaction forces with these "obstacles" for locomotion and navigation. To do so, we studied the locomotion of a quadrupedal robot as it traversed a simplified obstacle field, and discovered that with different gaits, the robot could passively converge to distinct orientations. A compositional return map explained this observed passive convergence, and enabled theoretical prediction of the steady-state orientation angles for any given quadrupedal gait. We experimentally demonstrated that with these predictions, a legged robot could effectively generate desired shape of trajectories amongst large, slippery obstacles, simply by switching between different gaits. Our study offered a novel method for robots to exploit traditionally-considered "obstacles" to achieve agile movements on challenging terrains.

Keywords: Reinforcement Learning, Legged Robots
Abstract: Quadruped robots face limitations in long-range navigation efficiency due to their reliance on legs. To ameliorate the limitations, we introduce a Reinforcement Learning-based Active Transporter Riding method (RL-ATR), inspired by humans' utilization of personal transporters, including Segways. The RL-ATR features a transporter riding policy and two state estimators. The policy devises adequate maneuvering strategies according to transporter-specific control dynamics, while the estimators resolve sensor ambiguities in non-inertial frames by inferring unobservable robot and transporter states. Comprehensive evaluations in simulation validate proficient command tracking abilities across various transporter-robot models and reduced energy consumption compared to legged locomotion. Moreover, we conduct ablation studies to quantify individual component contributions within the RL-ATR. This riding ability could broaden the locomotion modalities of quadruped robots, potentially expanding the operational range and efficiency.

Keywords: Legged Robots, Reinforcement Learning
Abstract: Learning diverse skills for quadruped robots presents significant challenges, such as mastering complex transitions between different skills and handling tasks of varying difficulty. Existing imitation learning methods, while successful, rely on expensive datasets to reproduce expert behaviors. Inspired by introspective learning, we propose Progressive Adversarial Self-Imitation Skill Transition (PASIST), a novel method that eliminates the need for complete expert datasets. PASIST autonomously explores and selects high-quality trajectories based on predefined target poses instead of demonstrations, leveraging the Generative Adversarial Self-Imitation Learning (GASIL) framework. To further enhance learning, We develop a skill selection module to mitigate mode collapse by balancing the weights of skills with varying levels of difficulty. Through these methods, PASIST is able to reproduce skills corresponding to the target pose while achieving smooth and natural transitions between them. Evaluations on both simulation platforms and the Solo 8 robot confirm the effectiveness of PASIST, offering an efficient alternative to expert-driven learning.

Keywords: Legged Robots, Reinforcement Learning, Machine Learning for Robot Control
Abstract: A core strength of Model Predictive Control (MPC) for quadrupedal locomotion has been its ability to enforce constraints and provide interpretability of the sequence of commands over the horizon. However, despite being able to plan, MPC struggles to scale with task complexity, often failing to achieve robust behavior on rapidly changing surfaces. On the other hand, model-free Reinforcement Learning (RL) methods have outperformed MPC on multiple terrains, showing emergent motions but inherently lack any ability to handle constraints or perform planning. To address these limitations, we propose a framework that integrates proprioceptive planning with RL, allowing for agile and safe locomotion behaviors through the horizon. Inspired by MPC, we incorporate an internal model that includes a velocity estimator and a Dreamer module. During training, the framework learns an expert policy and an internal model that are co-dependent, facilitating exploration for improved locomotion behaviors. During deployment, the Dreamer module solves an infinite-horizon MPC problem, adapting actions and velocity commands to respect the constraints. We validate the robustness of our training framework through ablation studies on internal model components and demonstrate improved robustness to training noise. Finally, we evaluate our approach across multi-terrain scenarios in both simulation and hardware.

Keywords: Whole-Body Motion Planning and Control, Reinforcement Learning, Legged Robots
Abstract: Combining manipulation with the mobility of legged robots is essential for a wide range of robotic applications. However, integrating an arm with a mobile base significantly increases the system’s complexity, making precise end-effector control challenging. Existing model-based approaches are often constrained by their modeling assumptions, leading to limited robustness. Meanwhile, recent Reinforcement Learning (RL) implementations restrict the arm’s workspace to be in front of the robot or track only the position to obtain decent tracking accuracy. In this work, we address these limitations by introducing a whole-body RL formulation for end-effector pose tracking in a large workspace on rough, unstructured terrains. Our proposed method involves a terrain-aware sampling strategy for the robot’s initial configuration and end-effector pose commands, as well as a game-based curriculum to extend the robot’s operating range. We validate our approach on the ANYmal quadrupedal robot with a six DoF robotic arm. Through our experiments, we show that the learned controller achieves precise command tracking over a large workspace and adapts across varying terrains such as stairs and slopes. On deployment, it achieves a pose-tracking error of 2.64 cm and 3.64◦, outperforming existing competitive baselines.

Keywords: Perception-Action Coupling, Legged Robots, Reinforcement Learning
Abstract: Developing versatile quadruped robots that can smoothly perform various actions and tasks in real-world environments remains a significant challenge. This paper introduces a novel vision-language-action (VLA) model, mixture of robotic experts (MoRE), for quadruped robots that aim to introduce reinforcement learning (RL) for fine-tuning large-scale VLA models with a large amount of mixed-quality data. method~integrates multiple low-rank adaptation modules as distinct experts within a dense multi-modal large language model (MLLM), forming a sparse-activated mixture of experts model. This design enables the model to effectively adapt to a wide array of downstream tasks. Moreover, we employ a reinforcement learning-based training objective to train our model as a Q-function after deeply exploring the structural properties of our tasks. Effective learning from automatically collected mixed-quality data enhances data efficiency and model performance. Extensive experiments demonstrate that method~outperforms all baselines across six different skills and exhibits superior generalization capabilities in out-of-distribution scenarios. We further validate our method in real-world scenarios, confirming the practicality of our approach and laying a solid foundation for future research on multi-task learning in quadruped robots.

Keywords: Human-Centered Robotics, Human-Aware Motion Planning, Human and Humanoid Motion Analysis and Synthesis
Abstract: The article presents a dedicated method for recognizing human postures using classification and clustering options. The ultimate goal of the research is to recognise human actions based on posture sequences. Such a task imposes expectations on the developed method. For this purpose, a Sparse Autoencoder combined with a Self-Organized Map (SOM) is proposed. SOM is equipped with an additional layer of post-labeling or clustering. This entire structure is called the extended SOM. Two task-oriented modifications are applied to improve SOM performance -- a dedicated angular distance measure and a neighbourhood function for updating the SOM weights. The research contribution is the concept of extended SOM, which is trained using unlabeled data and classifies or clusters the human postures. The Sparse Autoencoder preserves the characteristics of the data while reducing its dimensionality. Better classification efficiency of the developed method is demonstrated compared to other representative methods. Ablation studies illustrate how the introduced modifications improve classification results. The developed method is characterised by good resolution in distinguishing postures. A discussion of the concept's usefulness is provided at the end of the article.

Keywords: Human Detection and Tracking, Human and Humanoid Motion Analysis and Synthesis
Abstract: Millimeter-wave (mmWave) radar offers robust sensing capabilities in diverse environments, making it a highly promising solution for human body reconstruction due to its privacy-friendly and non-intrusive nature. However, the significant sparsity of mmWave point clouds limits the estimation accuracy. To overcome this challenge, we propose a two-stage deep learning framework that enhances mmWave point clouds and improves human body reconstruction accuracy. Our method includes a mmWave point cloud enhancement module that densifies the raw data by leveraging temporal features and a multi-stage completion network, followed by a 2D-3D fusion module that extracts both 2D and 3D motion features to refine SMPL parameters. The mmWave point cloud enhancement module learns the detailed shape and posture information from 2D human masks in single-view images. However, image-based supervision is involved only during the training phase, and the inference relies solely on sparse point clouds to maintain privacy. Experiments on multiple datasets demonstrate that our approach outperforms state-of-the-art methods, with the enhanced point clouds further improving performance when integrated into existing models.

Keywords: Human-Centered Robotics, Gesture, Posture and Facial Expressions, Multi-Modal Perception for HRI
Abstract: Human activity recognition (HAR) based on millimeter wave (mmWave) radar has recently attracted significant interest due to its diverse applications in intelligent robots and human-computer interaction (HCI), including the healthcare monitoring robot. 2-dimensional (2D) histogram features of radar point clouds have demonstrated high accuracy in HAR. But further expansion and refinement of this technique is needed. This paper presents a new precise non-invasive HAR framework based on radar point cloud 2D histograms. Our method enhances conventional 2D histograms by integrating fixed radar sensing boundaries into the histograms, which shows the relative spatial position changes of the target points detected by radar. Additionally, we have concatenated Doppler features (i.e., range-Doppler and angle-Doppler histograms) with the point cloud histograms, resulting in a more comprehensive feature representation than conventional point cloud histograms. We investigated the overfitting issue in stacked hybrid networks and established a multi-layer hybrid network with an optimal number of stacked layers for HAR. In the evaluation, our approach achieves state-of-the-art accuracy, with 99.72% on mmWaveRadarWalking dataset and 98.67% on CI4R-Human-Activity-Recognition dataset, respectively. The proposed method can be applied in the fields of robotics and HCI.

Keywords: Multi-Modal Perception for HRI, Robot Companions, Social HRI
Abstract: Engagement is a key concept in Human-Robot Interaction (HRI), as high engagement often leads to improved user experience and task performance. However, accurately estimating engagement during interactions is challenging. In this study, we propose a Dynamic Bayesian Network (DBN) to infer user engagement from various modalities, including head rotation, eye movements, facial expressions captured through visual sensors, as well as facial temperature variations measured by a thermal camera. Data was gathered from a human-robot interaction (HRI) experiment, where a robot guided participants and encouraged them to share their thoughts and insights on environmental issues. Our approach successfully combines these diverse features to offer a thorough assessment of user engagement. The network was tested on its capacity to classify participants as either engaged or not engaged, achieving an accuracy of 0.83 and an Area Under the Curve (AUC) of 0.82. These findings underscore the strength of our DBN in detecting user engagement during interactions.

Keywords: Cognitive Modeling, Embodied Cognitive Science, Social HRI
Abstract: Scaling social robot studies is constrained due to the need for human interaction, making large participant recruitment impractical. Robotics simulators help mitigate this limitation but generally lack the realism to accurately simulate social cues. We introduce a cognitive robotic simulation scheme to evaluate social attention models in physical environments. By projecting ground-truth priority maps to a simulated environment, we can directly compare predicted maps using common saliency metrics. Using the iCub robot, we assess a dynamic scanpath model that predicts attention targets, simulating human scanpaths. Evaluations with the FindWho and MVVA datasets show strong correlations between robot-captured metrics and direct-streamed video metrics. Our results indicate robustness of the social attention model to noise and real-world conditions, suggesting its practical usability for predicting personalized scanpaths in real settings. This approach reduces the need for extensive human-robot interaction studies in the early stages of study design, enabling the scalability and reproducibility of social robot evaluations.

Keywords: Brain-Machine Interfaces, Human Factors and Human-in-the-Loop, Intention Recognition
Abstract: Identifying a brain signal that enables the detection of incorrect execution in human-robot interaction (HRI) is considered a holy grail for real-time systems. A major challenge in achieving this is the inherent imbalance caused by the sparsity of error-related potential (ErrP) events in streaming electroencephalogram (EEG) data, which often leads models to learn irrelevant features and perform poorly. Thus, while Deep learning-based ErrP detection has seen considerable advancements, the variability in individual user reaction times introduces labelling errors, complicating model adaptation to new subjects. Moreover, most deep learning methods are developed and validated on discrete, offline experiments using pre-defined windows, which fail to translate effectively to continuous, real-time HRI. Addressing these challenges is crucial to improving the robustness and adaptability of real-time ErrP detection in practical HRI applications. Here, we develop a causal EEG conformer framework, combining a Convolutional neural network (CNN) encoder and a transformer with causal attention for real-time prediction of ErrP signals during HRI. We evaluated our ErrP model in a pseudo-online environment in both inter-session and inter-subject cross-validation settings for exoskeleton assistive robotics. Our model demonstrated superior performance in decoding accuracy and efficiency, showcasing better generalization for real-world dynamic HRI applications.

Keywords: Marine Robotics, Field Robots, Model Learning for Control
Abstract: In this paper, we propose a novel cross-platform fault-tolerant surfacing controller for underwater robots, based on reinforcement learning (RL). Unlike conventional approaches, which require explicit identification of malfunctioning actuators, our method allows the robot to surface using only the remaining operational actuators without needing to pinpoint the failures. The proposed controller learns a robust policy capable of handling diverse failure scenarios across different actuator configurations. Moreover, we introduce a transfer learning mechanism that shares a part of the control policy across various underwater robots with different actuators, thus improving learning efficiency and generalization across platforms. To validate our approach, we conduct simulations on three different types of underwater robots: a hovering-type AUV, a torpedo shaped AUV, and a turtle-shaped robot (U-CAT). Additionally, real-world experiments are performed, successfully transferring the learned policy from simulation to a physical U-CAT in a controlled environment. Our RL-based controller demonstrates superior performance in terms of stability and success rate compared to a baseline controller, achieving an 85.7 percent success rate in real-world tests compared to 57.1 percent with a baseline controller. This research provides a scalable and efficient solution for fault-tolerant control for diverse underwater platforms, with potential applications in real-world aquatic missions.

Keywords: Marine Robotics, Autonomous Agents, Reinforcement Learning
Abstract: Autonomous underwater navigation faces significant challenges due to the complexity of the environment, limited localization methods, and poor visibility. This paper investigates the performance of various reinforcement learning (RL) algorithms—Proximal Policy Optimization (PPO), Trust Region Policy Optimization (TRPO), Soft Actor-Critic (SAC), Twin Delayed DDPG (TD3), and Advantage Actor-Critic (A2C)—to improve navigation capabilities of low-cost underwater robots equipped with multi-modal sensors. Advanced depth estimation models such as MiDaS and Depth Anything, combined with domain randomization techniques, are employed to enhance the system's robustness and generalization across varying underwater conditions.

The proposed approach integrates real-time sensor data and historical actions to enable 3D maneuvering in simulated environments, leading to significant improvements in sensor fusion, depth perception, and obstacle avoidance. Simulation results demonstrate that the combination of RL techniques with sensor fusion considerably improves mapless autonomous underwater exploration, providing a robust solution for navigating unstructured aquatic environments.

Keywords: Marine Robotics, Perception for Grasping and Manipulation
Abstract: When navigating and interacting in challenging environments where sensory information is imperfect and incomplete, robots must make decisions that account for these shortcomings. We propose a novel method for quantifying and representing such perceptual uncertainty in 3D reconstruction through occupancy uncertainty estimation. We develop a framework to incorporate it into grasp selection for autonomous manipulation in underwater environments. Instead of treating each measurement equally when deciding which location to grasp from, we present a framework that propagates uncertainty inherent in the multi-view reconstruction process into the grasp selection. We evaluate our method with both simulated and the real world data, showing that by accounting for uncertainty, the grasp selection becomes robust against partial and noisy measurements. Code will be made available at https://onurbagoren.github.io/PUGS/

Keywords: Field Robots, Marine Robotics, Reinforcement Learning
Abstract: Controlling AUVs can be challenging because of the effect of complex non-linear hydrodynamic forces acting on the robot, which are significant in water and cannot be ignored. The problem is exacerbated for small AUVs for which the dynamics can change significantly with payload changes and deployments under different hydrodynamic conditions. The common approach to AUV control is a combination of passive stabilization with added buoyancy on top and weights on the bottom, and a PID controller tuned for simple and smooth motion primitives. However, the approach comes at the cost of sluggish controls and often the need to re-tune controllers with configuration changes. In this paper, we propose a fast (trainable in minutes), reinforcement learning-based approach for full 6 degree of freedom (DOF) control of a thruster-driven AUVs, taking 6-DOF command-conditioned inputs direct to thruster outputs. We present a new, highly parallelized simulator for underwater vehicle dynamics. We demonstrate this approach through zero-shot sim-to-real (with no tuning) transfer onto a real AUV that produces comparable results to hand-tuned PID controllers. Furthermore, we show that domain randomization on the simulator produces policies that are robust to small variations in vehicle's physical parameters.

Keywords: Marine Robotics, Aerial Systems: Mechanics and Control, Motion Control
Abstract: Coupling-Tiltable Unmanned Aerial-Aquatic Vehicles (UAAVs) have gained increasing importance, yet lack comprehensive analysis and suitable controllers. This paper analyzes the underwater motion characteristics of a self-designed UAAV, Mirs-Alioth, and designs a controller for it. The effectiveness of the controller is validated through experiments. The singularities of Mirs-Alioth are derived as Singular Thrust Tilt Angle (STTA), which serve as an essential tool for an analysis of its underwater motion characteristics. The analysis reveals several key factors for designing the controller. These include the need for logic switching, using a Nussbaum function to compensate control direction uncertainty in the auxiliary channel, and employing an auxiliary controller to mitigate coupling effects. Based on these key points, a control scheme is designed. It consists of a controller that regulates the thrust tilt angle to the singular value, an auxiliary controller incorporating a Saturated Nussbaum function, and a logic switch. Eventually, two sets of experiments are conducted to validate the effectiveness of the controller and demonstrate the necessity of the Nussbaum function.

Keywords: Marine Robotics, Motion Control, Robust/Adaptive Control
Abstract: 水下机器人在探索水生环境中发挥着至关重要的作用。灵活调整姿态的能力，尤其是俯仰，对于水下机器人在狭窄空间内有效完成任务至关重要。然而，由姿态变化导致的高度耦合的六自由度动力学和有限空间区域内的复杂湍流带来了重大挑战。为了解决水下机器人的姿态控制问题，本文研究了站位保持期间的大范围俯仰角跟踪以及同步滚转和偏航角控制，以实现多功能姿态调整。基于动态建模，本文提出了一种自适应积分滑模控制器 （AISMC），该控制器将积分模块集成到传统的滑模控制 （SMC） 中，并自适应地调整开关增益，以提高跟踪精度、减少颤振并增强鲁棒ö

Keywords: Aerial Systems: Perception and Autonomy, Deep Learning Methods, Automation Technologies for Smart Cities
Abstract: Obtaining accurate and timely predictions of the wind through an urban environment is a challenging task, but has wide-ranging implications for the safety and efficiency of autonomous aerial vehicles in future urban airspaces. Prior work relies strongly on global information about the environment, such as a precise map of the city and in-situ wind measurements at various locations, to run expensive computational fluid dynamics solvers to predict the entire wind flow field. In contrast, this paper introduces a new method to estimate the wind flow field in a region around the robot in real time, utilizing on-board range measurements to sense nearby buildings and sparse wind measurements to infer windspeed and direction. We propose that this information sufficiently characterizes the structure of the wind flow field in the local region of interest. To that end, we introduce a deep learning-based approach to predict local flow fields from range measurements. Our results indicate that a neural network trained on numerous simulated winds through small randomized maps is capable of reconstructing local wind flows while generalizing to larger environments with over 200 buildings. This contribution empowers computationally-constrained aerial robots to reason about the structure of local wind flow fields, thereby enabling new planning, control, and estimation strategies in windy urban environments without textit{a priori} knowledge of the map.

Keywords: Aerial Systems: Applications, Sensorimotor Learning, Reinforcement Learning
Abstract: Flying through body-size narrow gaps in the environment is one of the most challenging moments for an underactuated multirotor. We explore a purely data-driven method to master this flight skill in simulation, where a neural network directly maps pixels and proprioception to continuous low-level control commands. This learned policy enables whole-body control through gaps with different geometries demanding sharp attitude changes (e.g., near-vertical roll angle). The policy is achieved by successive model-free reinforcement learning (RL) and online observation space distillation. The RL policy receives (virtual) point clouds of the gaps' edges for scalable simulation and is then distilled into the high-dimensional pixel space. However, this flight skill is fundamentally expensive to learn by exploring in RL due to restricted feasible solution space. We propose to reset the agent as states on the trajectories by a model-based trajectory optimizer to alleviate this problem. The presented training pipeline is compared with baseline methods, and ablation studies are conducted to identify the key ingredients of the method. The immediate next step is to scale up the variation of gap sizes and geometries in anticipation of emergent policies and demonstrate the sim-to-real transformation.

Keywords: Aerial Systems: Perception and Autonomy, Simulation and Animation, Visual Learning
Abstract: We present VisFly, a quadrotor simulator designed to efficiently train vision-based flight policies using reinforcement learning algorithms. VisFly offers a user-friendly framework and interfaces, leveraging Habitat-Sim's rendering engines to achieve frame rates exceeding 10,000 frames per second for rendering motion and sensor data. The simulator incorporates differentiable physics and is seamlessly wrapped with the Gym environment, facilitating the straightforward implementation of various learning algorithms. It supports the directly importing open-source scene datasets compatible with Habitat-Sim, enabling training on diverse real-world environments simultaneously. To validate our simulator, we also make three reinforcement learning examples for typical flight tasks relying on visual observations. The simulator is now available at [https://github.com/SJTU-ViSYS-team/VisFly].

Keywords: Aerial Systems: Perception and Autonomy, Reinforcement Learning, AI-Enabled Robotics
Abstract: Reinforcement learning (RL) has achieved outstanding success in complex robot control tasks, such as drone racing, where the RL agents have outperformed human champions in a known racing track. However, these agents fail in unseen track configurations, always requiring complete retraining when presented with new track layouts. This work aims to develop RL agents that generalize effectively to novel track configurations without retraining. The naive solution of training directly on a diverse set of track layouts can overburden the agent, resulting in suboptimal policy learning as the increased complexity of the environment impairs the agent’s ability to learn to fly. To enhance the generalizability of the RL agent, we propose an adaptive environment-shaping framework that dynamically adjusts the training environment based on the agent’s performance. We achieve this by leveraging a secondary RL policy to design environments that strike a balance between being challenging and achievable, allowing the agent to adapt and improve progressively. Using our adaptive environment shaping, one single racing policy efficiently learns to race in diverse and challenging tracks. Experimental results validated in both simulation and the real world show that our method enables drones to successfully fly complex and unseen race tracks, outperforming existing environment-shaping techniques.

Keywords: Field Robots, Learning from Experience, Multi-Robot Systems
Abstract: Terrain awareness is an essential milestone to enable truly autonomous off-road navigation. Accurately predicting terrain characteristics allows optimizing a vehicle's path against potential hazards. Recent methods use deep neural networks to predict traversability-related terrain properties in a self-supervised manner, relying on proprioception as a training signal. However, onboard cameras are inherently limited by their point-of-view relative to the ground, suffering from occlusions and vanishing pixel density with distance. This paper introduces a novel approach for self-supervised terrain characterization using an aerial perspective from a hovering drone. We capture terrain-aligned images while sampling the environment with a ground vehicle, effectively training a simple predictor for vibrations, bumpiness, and energy consumption. Our dataset includes 2.8 km of off-road data collected in forest environment, comprising 13 484 ground-based images and 12 935 aerial images. Our findings show that drone imagery improves terrain property prediction by 21.37 % on the whole dataset and 37.35 % in high vegetation, compared to ground images. We conduct ablation studies to identify the main causes of these performance improvements. We also demonstrate the real-world applicability of our approach by scouting an unseen area with a drone, planning and executing an optimized path on the ground.

Keywords: Aerial Systems: Perception and Autonomy, Deep Learning for Visual Perception, Vision-Based Navigation
Abstract: Optical flow estimation is a critical task for tiny mobile robotics to enable safe and accurate navigation, obstacle avoidance, and other functionalities. However, optical flow estimation on tiny robots is challenging due to limited onboard sensing and computation capabilities. In this paper, we propose EdgeFlowNet, a high-speed, low-latency dense optical flow approach for tiny autonomous mobile robots by harnessing the power of edge computing. We demonstrate the efficacy of our approach by deploying EdgeFlowNet on a tiny quadrotor to perform static obstacle avoidance, flight through unknown gaps and dynamic obstacle dodging. EdgeFlowNet is about 20X faster than the previous state-of-the-art approaches while improving accuracy by over 20% and using only 1.08W of power enabling advanced autonomy on palm-sized tiny mobile robots.

Keywords: Multi-Robot Systems, Swarm Robotics, Cooperating Robots
Abstract: Distributed multi-robot coordination is critical to achieving reliable robotic missions that exploit the collective capability of swarm robots. In particular, the consensus and formation control problems have been extensively studied, resulting in distributed controllers that enable robots to rely only on information from themselves and their immediate neighbors. However, these algorithms are usually designed for specific objectives (e.g., cooperative object transportation, environmental coverage, etc.), requiring the controllers to be re-designed for domain variations. Therefore, we propose a new parametric framework inspired by gravitational fields that allow simultaneous coordination of robots at multiple levels, enabling generalization and domain adaptation. Our approach is built on top of a connectivity-preserving formation controller, with need-based and task-based ad hoc coordination at private, local, and global layers of a swarm robot team. We demonstrate the remarkable potential of our framework through extensive simulations and real-world swarm robot experiments in three representative multi-robot tasks involving tight coordination: 1) robot-initiated rendezvous at different coordination layers, 2) coordinated boundary tracking and coverage of environmental processes, and 3) accommodating task executions and motion control while satisfying the coordination laws.

Keywords: Multi-Robot Systems, Motion Control, Nonholonomic Motion Planning
Abstract: In this letter, we propose a novel formation controller for nonholonomic agents to form general parametric curves. First, we derive a unified parametric representation for both open and closed curves. Then, a leader-follower formation controller is designed to drive agents to form the desired parametric curves using the curve coefficients as feedbacks. We consider directed communications and constant input disturbances rejection in the controller design. Rigorous Lyapunov-based stability analysis proves the asymptotic stability of the proposed controller. The convergence of the orientations of agents to some constant values is also guaranteed. The method has the potential to be extended to deal with various real-world applications, such as object enclosing. Detailed numerical simulations and experimental studies are conducted to verify the performance of the proposed method.

Keywords: Multi-Robot Systems, Motion Control, Distributed Robot Systems
Abstract: This paper presents a unified approach to realize versatile distributed maneuvering with generalized formations. Specifically, we decompose the robots' maneuvers into two independent components, i.e., interception and enclosing, which are parameterized by two independent virtual coordinates. Treating these two virtual coordinates as dimensions of an abstract manifold, we derive the corresponding singularity-free guiding vector field (GVF), which, along with a distributed coordination mechanism based on the consensus theory, guides robots to achieve various motions (i.e., versatile maneuvering), including (a) formation tracking, (b) target enclosing, and (c) circumnavigation. Additional motion parameters can generate more complex cooperative robot motions. Based on GVFs, we design a controller for a nonholonomic robot model. Besides the theoretical results, extensive simulations and experiments are performed to validate the effectiveness of the approach.

Keywords: Multi-Robot Systems, Optimization and Optimal Control, Cooperating Robots
Abstract: This paper presents experiments for embedded cooperative distributed model predictive control applied to a team of hovercraft floating on an air hockey table. The hovercraft collectively solve a centralized optimal control problem in each sampling step via a stabilizing decentralized real-time iteration scheme using the alternating direction method of multipliers. The efficient implementation does not require a central coordinator, executes onboard the hovercraft, and facilitates sampling intervals in the millisecond range. The formation control experiments showcase the flexibility of the approach on scenarios with point-to-point transitions, trajectory tracking, collision avoidance, and moving obstacles.

Keywords: Multi-Robot Systems, Machine Learning for Robot Control
Abstract: Coordinated multi-robot navigation is an essential ability for a team of robots operating in diverse environments. Robot teams often need to maintain specific formations, such as wedge formations, to enhance visibility, positioning, and efficiency during fast movement. However, complex environments such as narrow corridors challenge rigid team formations, which makes effective formation control difficult in real-world environments. To address this challenge, we introduce a novel Adaptive Formation with Oscillation Reduction (AFOR) approach to improve coordinated multi-robot navigation. We develop AFOR under the theoretical framework of hierarchical learning and integrate a spring-damper model with hierarchical learning to enable both team coordination and individual robot control. At the upper level, a graph neural network facilitates formation adaptation and information sharing among the robots. At the lower level, reinforcement learning enables each robot to navigate and avoid obstacles while maintaining the formations. We conducted extensive experiments using Gazebo in the Robot Operating System (ROS), a high-fidelity Unity3D simulator with ROS, and real robot teams. Results demonstrate that AFOR enables smooth navigation with formation adaptation in complex scenarios and outperforms previous methods.

More details of this work are provided on the project website: https://hcrlab.gitlab.io/project/afor.

Keywords: Multi-Robot Systems, Reinforcement Learning, Motion and Path Planning
Abstract: In multi-robot exploration, a team of mobile robot is tasked with efficiently mapping an unknown environments. While most exploration planners assume omnidirectional sensors like LiDAR, this is impractical for small robots such as drones, where lightweight, directional sensors like cameras may be the only option due to payload constraints. These sensors have a constrained field-of-view (FoV), which adds complexity to the exploration problem, requiring not only optimal robot positioning but also sensor orientation during movement. In this work, we propose MARVEL, a neural framework that leverages graph attention networks, together with novel frontiers and orientation features fusion technique, to develop a collaborative, decentralized policy using multi-agent reinforcement learning (MARL) for robots with constrained FoV. To handle the large action space of viewpoints planning, we further introduce a novel information-driven action pruning strategy. MARVEL improves multi-robot coordination and decision-making in challenging large-scale indoor environments, while adapting to various team sizes and sensor configurations (i.e., FoV and sensor range) without additional training. Our extensive evaluation shows that MARVEL’s learned policies exhibit effective coordinated behaviors, outperforming state-of-the-art exploration planners across multiple metrics. We experimentally demonstrate MARVEL’s generalizability in large-scale environments, of up to 90m by 90m, and validate its practical applicability through successful deployment on a team of real drone hardware.

Keywords: Multi-Robot Systems, Search and Rescue Robots, Swarm Robotics
Abstract: Multi-Robot Systems (MRS) are increasingly deployed for hazardous tasks in urban environments. Among many tasks, search and rescue remains challenging as it deals with exploration in an unknown indoor constrained environment. For example, without global knowledge of the map of a building floor, it is not advantageous to choose one path over another at a corridor junction. Also, if the assigned frontiers are far from the robot, backtracking along a corridor will cost more than moving forward. Since exploration along corridors is similar to solving a maze, this paper examines classical maze-solving algorithms that are known to be computationally fast and lightweight, such as the Right Hand Rule (RHR), Random Mouse (RM), and more. The authors have identified two gaps that need to be addressed before these algorithms can be applied to physical MRS. Firstly, these algorithms are not designed for the cooperation of multiple agents in exploration. Secondly, they are often applied to only a low-fidelity simulation environment, which requires some work to make these algorithms transferable to work in the commonly used occupancy grid map environment. In this paper, the authors introduced RACE, a fast and lightweight collective urban exploration and search algorithm based on a modified and condensed version of the Ant Colony Optimization (ACO) algorithm. The proposed solution is successfully verified in a low-fidelity simulation, evaluated against other exploration and search algorithms like RHR and RM. An innovative approach of RACE Simulation to Physical implementation is presented and a physical system evaluation is performed to evaluate RACE against a Rapidly-Exploring Random Tree algorithm. Finally, the proposed solution is further verified with a physical experiment, which a quadrupedal robot is assigned to explore part of a floor of SUTD, spanning approximately (55m x 40m). RACE also showed potential in handling challenging close-loop and dead-end environments.

Keywords: Reinforcement Learning, Multi-Robot Systems, Cooperating Robots
Abstract: Collaborative multi-agent exploration of unknown environments is crucial for search and rescue operations. Effective real-world deployment must address challenges such as limited inter-agent communication and static and dynamic obstacles. This paper introduces a novel decentralized collaborative framework based on Reinforcement Learning to enhance multi-agent exploration in unknown environments. Our approach enables agents to decide their next action using an agent-centered field-of-view occupancy grid, and features extracted from A* algorithm-based trajectories to frontiers in the reconstructed global map. Furthermore, we propose a constrained communication scheme that enables agents to share their environmental knowledge efficiently, minimizing exploration redundancy. The decentralized nature of our framework ensures that each agent operates autonomously, while contributing to a collective exploration mission. Extensive simulations in Gymnasium and real-world experiments demonstrate the robustness and effectiveness of our system, while all the results highlight the benefits of combining autonomous exploration with inter-agent map sharing, advancing the development of scalable and resilient robotic exploration systems.

Keywords: Path Planning for Multiple Mobile Robots or Agents
Abstract: Learning to predict spatiotemporal (ST) environmental processes from a sparse set of samples collected autonomously is a difficult task from both a sampling perspective (collecting the best sparse samples) and from a learning perspective (predicting the next timestep). In this work, we focus on investigating the sample collection process via multi-robot informative path planning. We present an approach for incorporating multi-robot informative path planning into a spatiotemporal adaptive sampling framework while considering path length constraints for sampling location selection. We also incorporate informative path planning to determine the best path to collect samples along while en route to collecting the desired sample. We achieve this in a decentralized manner by decoupling the process into two stages: the first stage uses our spatiotemporal mixture of Gaussian Processes (STMGP) model to determine the most informative sampling location via a mutual information lower bound heuristic and the second stage plans an informative path to collect the desired sample and other additional informative samples via submodular function optimization. Moreover, we effectively leverage peer-to-peer communication to enable coordination. Simulation results on real-world spatiotemporal data are provided to validate the effectiveness of our proposed approach.

Keywords: Multi-Robot Systems, Path Planning for Multiple Mobile Robots or Agents, Nonholonomic Motion Planning
Abstract: This letter focuses on the multiple nonholonomic robots motion planning (MRMP) problem in congested and complex environments, where the complexity escalates dramatically with the increase in the number of robots, frequently leading to deadlocks. We present the Dueling Priority-Based Search (D-PBS), an efficient and scalable priority-based motion planner for multiple nonholonomic car-like robots, capable of enabling robots to move safely to destinations in spatially-constrained settings. We achieve this by adopting the alternate dueling collision resolution approach, coupled with the exploration of comprehensive priority relationships, effectively addressing the deadlock situations. We also introduce a novel priority-binding algorithm to enhance the scalability of our planner in restricted spaces densely populated with robots. Experimental evaluations in various scenarios demonstrate that D-PBS outperforms standard approaches to MRMP, offering superior path quality and scalability for larger robot swarms.

Keywords: Haptics and Haptic Interfaces, Telerobotics and Teleoperation, RGB-D Perception
Abstract: Vision-based haptic feedback provides cost-effective preemptive protection and real-time guidance, enhancing teleoperation with reduced system complexity. However, challenges arise as the instrument approaches target object, leading to occlusion of the point cloud behind the remote instrument, known as the self-occlusion issue. Prior solutions relying on historical point clouds or multiple viewpoints to refill the occluded region encounter adaptability issues for prolonged occlusion and limited space, thus hindering practical implementation. This paper introduces a novel non-refilling-based method for haptic force rendering, leveraging the correspondence between the tool-tip position and the tip position of the shadow-like occluded region. Experimental results demonstrate the proposed method's resilience across self-occlusion and dynamic environments, highlighting its practical applicability in robotic teleoperation.

Keywords: Haptics and Haptic Interfaces, Telerobotics and Teleoperation, Force Control, Passivity
Abstract: In this article, we characterize the passivity of a class of haptic systems modeled as a simple sampled-data system. Passivity is guaranteed by ensuring that there is enough damping in the haptic interface. A necessary and sufficient bound was determined in earlier work, but the corresponding mathematical expressions were complicated, and the derivations were not completely rigorous. In this article, a more tractable expression is derived. Based on the improved expression, passivity conditions are obtained for several classes of transfer functions representing virtual environments.

Keywords: Haptics and Haptic Interfaces, Wearable Robotics, Virtual Reality and Interfaces
Abstract: Haptic feedback enhances user interaction with systems by adding the sense of touch, thereby improving immersion and realism in applications like virtual reality (VR), augmented reality (AR), video games, education, and robotic surgery. To address the challenges in mechanically actuated haptic feedback devices such as limited mobility, mechanical wear, and complex mechanical structures, several research sought to develop electromagnetic haptic feedback systems. However, they also suffer from the rapid decay of magnetic force with distance, thus restricting their workspace size and application potential. In this paper, we propose a novel electromagnetic haptic feedback device that is actuated by magnetic torque instead of magnetic force. By controlling the magnetic torque, which decays with distance only at a third-order rate, our device achieves a large workspace—a 200-mm-diameter hemisphere—while still delivering perceptible real-time haptic feedback within the hemisphere. While using the device, the user wears a lightweight haptic thimble housing a permanent magnet on their finger, which enables 2 degree-of-freedom (DoF) haptic feedback. A 13-coil electromagnet array serves as the source of the magnetic field. A mathematical model is proposed to determine the currents in the electromagnet array to generate the desired amount of haptic feedback torque. We conducted two experiments to prove the viability of the device. A haptic feedback accuracy experiment was conducted and validated the device's ability to generate sufficient torque within a large workspace. A user evaluation experiment showed that the device achieved an overall accuracy of 77.86% in a virtual enclosure exploration task, indicating its effectiveness and usability in haptic feedback applications.

Keywords: Haptics and Haptic Interfaces, Soft Robot Materials and Design, Wearable Robotics
Abstract: This paper explores the feasibility of using magnetoresponsive silicone as the primary mechanism for generating vibrotactile feedback in haptic interfaces. The distinctive feature of this research lies in the integration of magnetoresponsive silicone, a flexible material that responds to electromagnetic fields to produce localized vibrations. Preliminary experiments evaluate the performance of these actuators, focusing on their ability to produce controlled vibrations across a range of frequencies and amplitudes relevant to human tactile perception. Building on this foundation, we introduce the VibroFlex Pad, a haptic interface featuring a magnetoresponsive silicone sheet and an array of electromagnets. The VibroFlex Pad demonstrates its versatility in generating varied tactile effects and simulating dynamic wave-like movements across its surface. To assess the VibroFlex Pad's effectiveness, a user study was conducted, separately evaluating tactile accuracy, overall performance, and user comfort. The findings suggest that the VibroFlex Pad offers reliable and precise vibrotactile feedback, highlighting its potential to enhance wearable haptic technologies and improve the user experience in a variety of applications.

Keywords: Physical Human-Robot Interaction, Safety in HRI, Biologically-Inspired Robots
Abstract: Human-robot physical interaction (pHRI) is a rapidly evolving research field with significant implications for physical therapy, search and rescue, and telemedicine. However, a major challenge lies in accurately understanding human constraints and safety in human-robot physical experiments without an IRB and physical human experiments. Concerns regarding human studies include safety concerns, repeatability, and scalability of the number and diversity of participants. This paper examines whether a physical approximation can serve as a stand-in for human subjects to enhance robot autonomy for physical assistance. This paper introduces the SHULDRD (Shoulder Haptic Universal Limb Dynamic Repositioning Device), an economical and anatomically similar device designed for real-time testing and deployment of pHRI planning tasks onto robots in the real world. SHULDRD replicates human shoulder motion, providing crucial force feedback and safety data. The device's open-source CAD and software facilitate easy construction and use, ensuring broad accessibility for researchers. By providing a flexible platform able to emulate infinite human subjects, ensure repeatable trials, and provide quantitative metrics to assess the effectiveness of the robotic intervention, SHULDRD aims to improve the safety and efficacy of human-robot physical interactions.

Keywords: Haptics and Haptic Interfaces, Telerobotics and Teleoperation, Micro/Nano Robots
Abstract: The precise manipulation of microrobots presents challenges arising from their small size and susceptibility to external disturbances. To address these challenges, we present the experimental evaluation of a haptic shared control teleoperation framework for the locomotion of multiple microrobots, relying on a kinesthetic haptic interface and a custom electromagnetic system. Six combinations of haptic and shared control strategies are evaluated during a safe 3D navigation scenario in a cluttered environment. 18 participants are asked to steer two spherical magnetic microrobots among obstacles to reach a predefined goal, under different conditions. For each condition, participants are provided with different obstacle avoidance and navigation guidance cues. Results show that providing assistance in avoiding obstacles guarantees safer performance, regardless if the assistance is autonomous or delivered through a haptic repulsive force. Moreover, autonomous obstacle avoidance also reduces the completion time by 30% compared to haptic obstacle avoidance and no obstacle avoidance cases, although haptic feedback is preferred by the users. Finally, providing haptic guidance towards the target improves by the 65% the positioning accuracy of the microrobots with respect to not providing this guidance.

Keywords: Assembly, Performance Evaluation and Benchmarking, Robotics and Automation in Construction
Abstract: Structural stability is a necessary condition for successful construction of an assembly. However, designing a stable assembly requires a non-trivial effort since a slight variation in the design could significantly affect the structural stability. To address the challenge, this paper studies the stability of assembly structures, in particular, block stacking assembly. The paper proposes a new optimization formulation, which optimizes over force balancing equations, for inferring the structural stability of 3D block stacking structures. The proposed stability analysis is verified on hand-crafted Lego examples. The experiment results demonstrate that the proposed method can correctly predict whether the structure is stable. In addition, it outperforms the existing methods since it can accurately locate the weakest parts in the design, and more importantly, solve any given assembly structures. To further validate the proposed method, we provide StableLego: a comprehensive dataset including 50k+ 3D objects with their Lego layouts. We test the proposed stability analysis and include the stability inference for each corresponding object in StableLego. Our code and the dataset are available at https://github.com/intelligent-control-lab/StableLego.

Keywords: Assembly, Visual Learning, Computer Vision for Automation
Abstract: Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labeled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.

Keywords: Assembly, Path Planning for Multiple Mobile Robots or Agents, Parallel Robots
Abstract: Coordinated multi-agent robotic construction provides a means to build infrastructure in extreme environments and improve efficiency in high performance applications. Planning methods are key to understanding and achieving the scope of such applications, and are typically tailored to specific models of construction material and a consideration of passivity or activity thereof. Here, we focus on the NASA Automated Reconfigurable Mission Adaptive Digital Assembly Systems (ARMADAS) model, which includes passive lightweight structural modules and small robots that traverse the structure. We present an algorithm for calculating a build plan for robots under the constraints of this type of system. We then evaluate the quality of this plan experimentally. Many of the techniques we use can be applied to any robotic assembly system whose robots perform locomotion over the structure that they are building.

Keywords: Grasping, Assembly
Abstract: Tangram assembly, the art of human intelligence and manipulation dexterity, is a new challenge for robotics and reveals the limitations of state-of-the-arts. Here, we describe our initial exploration and highlight key problems in reasoning, planning, and manipulation for robotic tangram assembly. We present MRChaos (Master Rules from Chaos), a robust and general solution for learning assembly policies that can generalize to novel objects. In contrast to conventional methods based on prior geometric and kinematic models, MRChaos learns to assemble randomly generated objects through self-exploration in simulation without prior experience in assembling target objects. The reward signal is obtained from the visual observation change without manually designed models or annotations. MRChaos retains its robustness in assembling various novel tangram objects that have never been encountered during training, with only silhouette prompts. We show the potential of MRChaos in wider applications such as cutlery combinations. The presented work indicates that radical generalization in robotic assembly can be achieved by learning in much simpler domains.

Keywords: Human-Centered Robotics, Embodied Cognitive Science, Planning under Uncertainty
Abstract: In this paper we explore if human mental models of objects, even when flawed, can be integrated with a collaborative robot's decision making framework to allow it to make smarter choices under partial observability for different object-related tasks such as assembly and troubleshooting. We demonstrate how (1) these informative causal models can be extracted from humans through crowdsourcing, (2) object assembly and troubleshooting can be formulated as Partially Observable Markov Decision Processes (POMDPs) and (3) our extracted causal models can be incorporated into those models in the form of approximate priors. Finally, (4) we use systematic experimentation in simulation to demonstrate the success of this approach, with 2X average improvement in reward observed for object assembly tasks, and 1.4X average improvement in reward observed for troubleshooting tasks.

Keywords: Robotics and Automation in Construction, Assembly
Abstract: This paper explores automated robotic construction of clocháin, a type of corbelled rock shelter, traditionally crafted by skilled workers. While robots have been employed for simple dry-stacking tasks in the past, such as construction of stone walls or vertical stone towers, the question of whether robots possess the capacity to construct more functional structures remains unanswered. This study presents a significant step forward in robotic dry-stacking of functional structures: the assembly of natural stones into freestanding clocháin structures. We also present a set of stackability measures to aid stone selection, which significantly improves the stability of the planned structures. Our sequential filtering approach, originally designed for planning stone walls, plays a foundational role in achieving stable clochán construction. Experimental results validate the effectiveness of the stackability measures and demonstrate the physical execution of dry-stacking clocháin. The progress demonstrated in this paper opens the door to robotic construction of a wide range of utility structures in unstructured environments.

Keywords: Reinforcement Learning, Task and Motion Planning, Representation Learning
Abstract: Research into robotics applications of deep reinforcement learning (DRL) has increasingly been focussed on learning precise object manipulation and trajectory planning. Extending these tasks to continuous robot-object interactions with the surface of complex geometries remains an open problem. In this paper we investigate end-to-end DRL solutions for depowdering tasks that work by directing a pressurized air stream onto the object's surfaces using a blast nozzle head mounted on a robotic arm. We develop a GPU accelerated vectorized cleaning effect for integration into RL training and consider ways to expose vision-less trajectory synthesis for surface treatment applications to the RL agent based on UV mapping. Our experimental evaluation demonstrates that DRL has the potential to be used for generating object-specific agents for depowdering tasks on a variety of 3D objects without requiring intermediate path planners even in a full 3D motion setup. Finally, we show that DRL-generated trajectories can be transferred to a real-world setup. Our task formulation lends itself to approximate a wide range of surface treatment applications (e.g., cleaning and spray painting) with various effects.

Keywords: Reinforcement Learning, Legged Robots
Abstract: Legged locomotion over various terrains is challenging and requires precise perception of the robot and its surroundings from both proprioception and vision. However, learning directly from high-dimensional visual input is often data-inefficient and intricate. To address this issue, traditional methods attempt to learn a teacher policy with access to privileged information first and then learn a student policy to imitate the teacher's behavior with visual input. Despite some progress, this imitation framework prevents the student policy from achieving optimal performance due to the information gap between inputs. Furthermore, the learning process is unnatural since animals intuitively learn to traverse different terrains based on their understanding of the world without privileged knowledge. Inspired by this natural ability, we propose a simple yet effective method, World Model-based Perception (WMP), which builds a world model of the environment and learns a policy based on the world model. We illustrate that though completely trained in simulation, the world model can make accurate predictions of real-world trajectories, thus providing informative signals for the policy controller. Extensive simulated and real-world experiments demonstrate that WMP outperforms state-of-the-art baselines in traversability and robustness. Videos and Code are available at: https://wmp-loco.github.io/.

Keywords: Reinforcement Learning, Learning from Demonstration, Aerial Systems: Applications
Abstract: This paper addresses the challenge of Velocity Vector Control (VVC) for fixed-wing UAVs using Reinforcement Learning (RL) in the presence of imperfect demonstrations. The multi-objective and long-horizon nature of VVC introduces significant spatial and temporal complexities, complicating RL's exploration. While demonstration-based RL methods can help mitigate exploration challenges, their effectiveness is often limited by the quality of the provided demonstrations. To tackle this, we propose V-Pilot, a novel approach that integrates: (1) a controller equipped with a control law model to reduce action oscillation, thus alleviating temporal exploration issues, and (2) a VVC-specific training workflow for iterative policy refinement and demonstration quality improvement. This framework is designed to enhance the performance of demonstration-based RL under imperfect demonstrations. We evaluate V-Pilot on the fixed-wing UAV RL environment, FlyCraft. Experimental results demonstrate that V-Pilot outperforms PID and Behavioral Cloning across multiple performance metrics.

Keywords: Reinforcement Learning, Social HRI, Emotional Robotics
Abstract: Expressive robotic behavior is essential for the widespread acceptance of robots in social environments. Recent advancements in learned legged locomotion controllers have enabled more dynamic and versatile robot behaviors. However, determining the optimal behavior for interactions with different users across varied scenarios remains a challenge. Current methods either rely on natural language input, which is efficient but low-resolution, or learn from human preferences, which, although high-resolution, is sample inefficient. This paper introduces a novel approach that leverages priors generated by pre-trained LLMs alongside the precision of preference learning. Our method, termed Language-Guided Preference Learning (LGPL), uses LLMs to generate initial behavior samples, which are then refined through preference-based feedback to learn behaviors that closely align with human expectations. Our core insight is that LLMs can guide the sampling process for preference learning, leading to a substantial improvement in sample efficiency. We demonstrate that LGPL can quickly learn accurate and expressive behaviors with as few as four queries, outperforming both purely language-parameterized models and traditional preference learning approaches. Website with videos: lgpl-gaits.github.io/

Keywords: Planning, Scheduling and Coordination, Multi-Robot Systems, Reinforcement Learning
Abstract: Optimizing large-scale logistics is computationally challenging due to its scale and requirement to be robust to stochastic and time-varying weather disturbances. However, prior research in multi-agent reinforcement learning (MARL) does not address scenarios that capture complexity of logistics operations influenced by dynamic weather patterns. To address this gap, we suggest a new MARL environment, textsc{Marine} that has two types of agents equipped with limited resources and integrates real wave data to model the influences of weather on the replenishment at sea (RAS) operation. To this end, we propose SchedHGNN, a novel MARL algorithm that incorporates a heterogeneous graph neural network and an intrinsic reward scheme to enhance agent coordination and mitigate challenges induced by environment non-stationarity. Our results show that the combination of effective RAS scheduling and improved communication enables our model to outperform competitive baselines by up to 37.8%. This achievement marks a significant advancement in applying MARL to complex, real-world logistics scenarios.

Keywords: Prosthetics and Exoskeletons, Optimization and Optimal Control, Rehabilitation Robotics, Ultrasound Imaging
Abstract: A hybrid exoskeleton is a class of wearable robotic technology that simultaneously uses a powered exoskeleton and functional electrical stimulation (FES) to generate assistive joint torques for people with impaired mobility due to neurological disorders such as spinal cord injury (SCI). The hybrid assistive technology benefits from FES that actively elicits force from paralyzed muscles via their neural excitation, leading to muscle strengthening. The main technical barrier to realizing the hybrid technology is to attain stable coordination between FES and the exoskeleton despite the quick onset of FES-induced muscle fatigue, which causes a rapid decline in the muscle force. Current methods to measure the induced fatigue lack direct muscle state measurements and may be ineffective at capturing the muscle force decay due to FES. Instead, ultrasound (US) imaging accurately quantifies FES-related muscle contractility and fatigue due to the direct visualization of muscle fibers. In this paper, we use real-time US imaging-derived muscle strain changes as biomarkers of FES-induced fatigue in an optimal controller that modulates exoskeleton assistance and FES dosage. To demonstrate that real-time US imaging is a promising muscle-machine interface technology that can optimize shared control in a hybrid exoskeleton, we perform experiments involving continuous seated knee extension and over-ground walking tasks on two participants with SCI and four participants without disabilities. Furthermore, this work helps design a novel and unprecedented robotic gait technology with the capability to impart FES-associated therapeutic benefits while assisting the gait of neurologically impaired individuals, including those with SCI, stroke, multiple sclerosis, etc.

Keywords: Mechanism Design, Prosthetics and Exoskeletons, Wearable Robotics
Abstract: This paper presents a novel semi-passive knee exoskeleton designed to provide running assistance. It incorporates an energy-efficient clutch mechanism activated by a mini servomotor which engages and disengages the spring that supports the leg during running. The exoskeleton extracts energy during the running phase when the muscles are acting as brakes (negative power), stores it in the spring, and then returns this energy during the positive power phase (when the muscles are acting as motors). The exoskeleton controller implements an inertial measurement unit (IMU) sensor to estimate the shank orientation that determines when to engage and disengage the spring. Two experiments designed to probe the functionality of the exoskeleton were conducted to evaluate its control performance and actuation, and the exoskeleton's biomechanical impact on three subjects. The findings showed that the control mechanism could be engaged and disengaged in real time. The maximum moment created on the knee muscles was 17 Nm, although the device could supply 28 Nm. The ratio of the consumed servo energy consumption to the subjects’ saved energy was 1:160 (0.1W input to 16W saved). This study thus paves the way for the development of lightweight, inexpensive exoskeletons that can contribute to their greater availability for a broader range of individuals.

Keywords: Rehabilitation Robotics, Wearable Robotics, Biologically-Inspired Robots
Abstract: This paper presents a comparative analysis of model-based control strategies for a Bionic Ankle Tensegrity Exoskeleton (BATE). The BATE is designed to mimic the self-stress equilibrium and self-supporting characteristics of the human ankle biotensegrity structure. Model-based control strategies are conventional methods that can help discover the principles of complex tensegrity systems. The high dimensions and non-linearity of the BATE pose challenges for physical modelling and require unique model-based control strategies. In this study, we propose a modelling method that considers interaction force and explore the trajectory tracking performance and robustness of the ankle exoskeleton under three power-assisted control methods: position control, force control, and hybrid force-position contorl. The experimental results suggest that the PC method offers superior tracking performance and robustness compared to the other two methods. This method can be used for early rehabilitation training to improve flexibility. The control concept emphasizes its advantages over current wearable exoskeletons and introduces new ideas for high-performance exoskeletons.

Keywords: Prosthetics and Exoskeletons, Rehabilitation Robotics, Wearable Robotics
Abstract: Self-balancing lower limb rehabilitation exoskeletons (SLLREs) allow individuals with lower limb dysfunction to walk without the use of crutches. Stable and human-like walking motions are crucial for SLLREs because achieving a close imitation of healthy human walking is a key goal in rehabilitation therapy. Existing SLLREs can realize stable walking but lack human-like features. This paper designs a walking motion generator based on hierarchical optimization to generate a human-like walking motion with variable hip height, heel-strike, toe-off, and knee-stretched features. This hierarchically optimized human-like walking motion generator consists of a knee-stretched optimizer and an optimization-based stabilizing filter. Specifically, the knee-stretched optimizer realizes the stretched knee feature by optimizing the hip trajectory with varying heights. And the stabilizing filter realizes stable walking by optimizing the hip trajectory in the sagittal plane direction.To validate the effectiveness of the proposed human-like walking motion generator, walking experiments were conducted on SLLRE AutoLEE-G3 both in a simulation environment and the real world. The experimental results show that the human-like walking motions look more natural and reduce the required torque for the knee joint compared with knee-bent walking.

Keywords: Prosthetics and Exoskeletons, Wearable Robotics, Tendon/Wire Mechanism
Abstract: This letter presents a novel cable-driven exosuit intended for head-neck support and movement assistance. Mobility limitations in the head-neck, such as dropped head syndrome, can result from various neurological disorders. Current solutions, ranging from static neck collars to rigid-link robotic neck exoskeletons, are unsatisfactory. Neck collars are the most used clinically but fail to restore head-neck motion. Rigid-link neck exoskeletons can enable head movement but are bulky and restrictive. In this letter, we present the design of this exosuit, an analysis of its ability to balance the gravitational moment of the head in simulation, and the results of a user study comparing its kinematic performance to a state-of-the-art rigid-link neck exoskeleton. The exosuit is able to support the head across its full range of motion according to simulation results. It fits users of different sizes and participants exhibited more natural head-neck movement wearing the exosuit as compared to wearing the rigid-link exoskeleton. The exosuit allowed more head rotations than the rigid-link neck exoskeleton and required less compensatory torso movement for three daily tasks (looking for traffic, drinking from a bottle, and picking up an object from the floor). Its absolute range of motion was also much larger than the one allowed by the rigid-link neck exoskeleton. These results demonstrate the kinematic benefits of a cable-driven neck exosuit and provide justification for studying the use of such an exosuit for head-neck movement assistance in patient groups.

Keywords: Modeling, Control, and Learning for Soft Robots, Kinematics, Motion and Path Planning, Soft Robot Applications
Abstract: Soft and hyper-elastic materials possess properties of resilience and flexibility, characterizing a class of Soft-Continuum Manipulators (SCM). The latter describes a robot structure with an infinite number of degrees of freedom (DoFs), useful for mobility and manipulation. However, these geometric characteristics are source of modeling and control problems. In this paper, a Pythagorean Hodograph (PH) curve based Reduced-Order-Model (ROM) relying on the Gauss-Lobatto quadrature is investigated for the modeling and the control of SCM. This allows, first, reducing the dimension of the SCM kinematics based on the PH parametric curves with a predefined length and second, developing the shape kinematics control from its control polygon. The use of the Gauss-Lobatto quadrature allows to move independently the PH curve control points, while preserving PH features of length and minimum curve energy. These features are important to control in real-time the shape of the SCM. The proposed approach has been validated numerically and experimentally, carried out on a bio-inspired Soft continuum Elephant Trunk Robot.

Keywords: Modeling, Control, and Learning for Soft Robots, Motion and Path Planning, Soft Robot Applications
Abstract: Tendon-driven continuum robots (TDCRs), with their flexible backbones, offer the advantage of being used for navigating complex, cluttered environments. However, to do so, they typically require multiple segments, often leading to complex actuation and control challenges. To this end, we propose a novel approach to navigate cluttered spaces effectively for a single-segment long TDCR which is the simplest topology from a mechanical point of view. Our key insight is that by leveraging contact with the environment we can achieve multiple curvatures without mechanical alterations to the robot. Specifically, we propose a search-based motion planner for a single-segment TDCR. This planner, guided by a specially designed heuristic, discretizes the configuration space and employs a best-first search. The heuristic, crucial for efficient navigation, provides an effective cost-to-go estimation while respecting the kinematic constraints of the TDCR and environmental interactions. We empirically demonstrate the efficiency of our planner-testing over 525 queries in environments with both convex and non-convex obstacles, our planner is demonstrated to have a success rate of about 80% while baselines were not able to obtain a success rate higher than 30%. The difference is attributed to our novel heuristic which is shown to significantly reduce the required search space.

Keywords: Modeling, Control, and Learning for Soft Robots, Surgical Robotics: Steerable Catheters/Needles, Tendon/Wire Mechanism
Abstract: The flexibility and dexterity of cable-driven continuum robots (CDCRs) make them well-suited for intricate tasks such as minimally invasive surgery. However, the complexity of accurately modeling their dynamics has limited their broader adoption and effective control. Current models either oversimplify the dynamics by assuming quasi-static conditions or overcomplicate them, making real-time application challenging. Additionally, many existing models neglect the critical coupling between the robot's body and actuator dynamics, a factor essential for accurate control. In this paper, we propose a new, minimal dynamics model for CDCRs that strikes a balance between simplicity and accuracy. Our model captures the essential dynamics of both the robot and its actuators, providing a practical tool for control design. We also establish connections between our model and those used for other robotic systems, enabling the transfer of well-established control strategies to CDCRs. The model is validated through hardware experiments, demonstrating its capability to effectively address complex control challenges in CDCR applications.

Keywords: Mechanism Design, Tendon/Wire Mechanism, Actuation and Joint Mechanisms
Abstract: Articulated continuum robots (ACRs) are characterized by flexibility, controllability, and adaptability and perform excellently in complex and constrained environments. However, the large number of motor drives limit the ACRs' portability and make them cumbersome to control. This paper presents a novel tendon-driven ACR composed of stabilized self-locking joints (SLJs) connected in series. After triggering the mechanical constraints with shape memory alloy coils, each joint can be maintained in either a self-locking or release state with zero power consumption. Consequently, even with a single set of drive units, the ACR can operate in multiple modes, enabling variable motion performance and workspace adaptability, effectively reducing the number of motors. The ACR's stiffness also varies with the locking state of its SLJs, and no motor drive is required to maintain its shape when all SLJs are self-locking. The performance and reliability of the SLJ prototype were validated. The workspace of the ACR prototype model was analyzed, and its partial motion performance, motion error, and variable stiffness were verified.

Keywords: Actuation and Joint Mechanisms, Biomimetics, Soft Robot Materials and Design
Abstract: Tensegrity structures have been widely explored for their lightweight, high-stiffness, and foldable properties. These unique characteristics have enabled their application in various fields including robotics. Tensegrity robots have demonstrated diverse locomotion modes offering versatile solutions for navigation in complex environments. Recent efforts in bio-inspired robotics have led to designs mimicking the movement of natural organisms, such as earthworms. However, existing designs, particularly those utilizing motor-pulley mechanisms for robot body contraction, face significant challenges due to their bulky actuation systems that reduce locomotion efficiency. This paper introduces a novel tensegrity robot, "Tensiworm," inspired by the peristaltic locomotion of an earthworm. Composed of three icosahedron tensegrity unit cells connected in series, the Tensiworm robot employs a sequential contraction and relaxation mechanism driven by active cable members made of shape memory actuators. This innovative design achieves a 59.13% folding ratio and weighs only 46.9 grams. The robot can travel a distance equal to its body length in approximately ten cycles with an average speed of 10.01 mm per minute. Furthermore, the use of thinner, flexible structural members broadens possibilities for development of millimeter-scale tensegrity robots, which hold significant potential for biomedical applications, including in-vivo testing and targeted drug delivery.

Keywords: Simulation and Animation, Contact Modeling, Medical Robots and Systems
Abstract: Continuum 机器人提供高度的灵活性和多个自由度，使其成为导航窄流腔的理想选择。然而，准确模拟它们在大变形和频繁环境接触下的行为仍然具有挑战性。当前求解这些机器人变形的方法，例如模型降阶法和高斯-塞德尔 （GS） 方法，都存在明显的缺点。随着接触点数量的增加，他们的计算速度会降低，并且难以在速度和模型精度之间取得平衡。为了克服这些限制，我们引入了一种名为 Acc-FEM 的新型有限元方法 （FEM）。Acc-FEM 采用大变形准静态有限元模型，并集成了加速求解器方案，以高效处理多触点仿真。此外，它还利用图形处理单元 （GPU） 的并行计算来实时更新有限元模型和

Keywords: Logistics, Grasping
Abstract: This paper introduces a novel Experience Forest algorithm designed for multi-object grasping (MOG). Different from single-object grasping, for MOG, the hand poses a few steps before the end of grasping play important roles in the success of MOG. But similar to single-object grasping, the hand poses that are far away from the end grasping pose are not as relevant. Therefore, the proposed approach invented the Experience Forest structure to organize the finger movement sequences collected in naive MOG approaches with a set of trees instead of a single tree. The algorithm propagates success or failure results in the trials from end-pose nodes only to the nodes representing several preceding hand poses. When using the trees to generate a grasping sequence, the algorithm generates a finger-movement policy that follows a MOG synergy at the beginning and then transits to a tree in the Experience Forest and then employs a breadth-first search to achieve a more reliable solution. Tested on various objects using a UR5e robotic arm and Barrett hand in both simulated and real environments, the strategy significantly boosts efficiency in object transfer tasks by up to 60%, marking a 10% improvement over our previous methods.

Keywords: Deep Learning in Grasping and Manipulation, Perception for Grasping and Manipulation, Grasping
Abstract: Grasp learning in noisy environments, such as occlusions, sensor noise, and out-of-distribution (OOD) objects, poses significant challenges. Recent learning-based approaches focus primarily on capturing aleatoric uncertainty from inherent data noise. The epistemic uncertainty, which represents the OOD recognition, is often addressed by ensembles with multiple forward paths, limiting real-time application. In this paper, we propose an uncertainty-aware approach for 6-DoF grasp detection using evidential learning to comprehensively capture both uncertainties in real-world robotic grasping. As a key contribution, we introduce vMF-Contact, a novel architecture for learning hierarchical contact grasp representations with probabilistic modeling of directional uncertainty as von Mises–Fisher (vMF) distribution. To achieve this, we analyze the theoretical formulation of the second-order objective on the posterior parametrization, providing formal guarantees for the model's ability to quantify uncertainty and improve grasp prediction performance. Moreover, we enhance feature expressiveness by applying partial point reconstructions as an auxiliary task, improving the comprehension of uncertainty quantification as well as the generalization to unseen objects. In the real-world experiments, our method demonstrates a significant improvement by 39% in the overall clearance rate compared to the baselines. The code is available under: https://github.com/YitianShi/vMF-Contact/tree/main

Keywords: Mobile Manipulation, Legged Robots, Whole-Body Motion Planning and Control
Abstract: Legged robots with advanced manipulation capabilities have the potential to significantly improve household duties and urban maintenance. Despite considerable progress in developing robust locomotion and precise manipulation methods, seamlessly integrating these into cohesive whole-body control for real-world applications remains challenging. In this paper, we present a modular framework for robust and generalizable whole-body loco-manipulation controller based on a single arm-mounted camera. By using reinforcement learning (RL), we enable a robust low-level policy for command execution over 5 dimensions (5D) and a grasp-aware high-level policy guided by a novel metric, Generalized Oriented Reachability Map (GORM). The proposed system achieves state-of-the-art one-time grasping accuracy of 89% in real world, including challenging tasks such as grasping transparent objects. Through extensive simulations and real-world experiments, we demonstrate that our system can effectively manage a large workspace, from floor level to above body height, and perform diverse whole-body loco-manipulation tasks.

Keywords: Deep Learning in Grasping and Manipulation, In-Hand Manipulation, Dexterous Manipulation
Abstract: In-hand manipulation and grasping are fundamental yet often separately addressed tasks in robotics. For deriving in-hand manipulation policies, reinforcement learning has recently shown great success. However, the derived controllers are not yet useful in real-world scenarios because they often require a human operator to place the objects in suitable initial (grasping) states. Finding stable grasps that also promote the desired in-hand manipulation goal is an open problem. In this work, we propose a method for bridging this gap by leveraging the critic network of a reinforcement learning agent trained for in-hand manipulation to score and select initial grasps. Our experiments show that this method significantly increases the success rate of in-hand manipulation without requiring additional training. We also present an implementation of a full grasp manipulation pipeline on a real-world system, enabling autonomous grasping and reorientation even of unwieldy objects.

Keywords: Deep Learning in Grasping and Manipulation, Sensor Fusion, Prosthetics and Exoskeletons
Abstract: One of the most important research challenges in upper-limb prosthetics is enhancing the user-prosthesis communication to closely resemble the experience of a natural limb. As prosthetic devices become more complex, users often struggle to control the additional degrees of freedom. In this context, leveraging shared-autonomy principles can significantly improve the usability of these systems. In this paper, we present a novel eye-in-hand prosthetic grasping system that follows these principles. Our system initiates the approach-to-grasp action based on user's command and automatically configures the DoFs of a prosthetic hand. First, it reconstructs the 3D geometry of the target object without the need of a depth camera. Then, it tracks the hand motion during the approach-to-grasp action and finally selects a candidate grasp configuration according to user's intentions. We deploy our system on the Hannes prosthetic hand and test it on able-bodied subjects and amputees to validate its effectiveness. We compare it with a multi-DoF prosthetic control baseline and find that our method enables faster grasps, while simplifying the user experience. Code and demo videos are available online at this https URL.

Keywords: Localization, SLAM, Visual Learning
Abstract: State-of-the-art hierarchical localisation pipelines (HLoc) employ image retrieval (IR) to establish 2D-3D correspondences by selecting the top-k most similar images from a reference database. While increasing k improves localisation robustness, it also linearly increases computational cost and runtime, creating a significant bottleneck. This paper investigates the relationship between global and local descriptors, showing that greater similarity between the global descriptors of query and database images increases the proportion of feature matches. Low similarity queries significantly benefit from increasing k, while high similarity queries rapidly experience diminishing returns. Building on these observations, we propose an adaptive strategy that adjusts k based on the similarity between the query's global descriptor and those in the database, effectively mitigating the feature-matching bottleneck. Our approach reduces computational costs and processing time without sacrificing accuracy. Experiments on three indoor and outdoor datasets show that AIR-HLoc reduces feature matching time by up to 30% while preserving state-of-the-art accuracy. The results demonstrate that AIR-HLoc facilitates a latency-sensitive localisation system.

Keywords: Localization, Mapping, RGB-D Perception
Abstract: Recently, neural radiance fields (NeRF) have gained significant attention in the field of visual localization. However, existing NeRF-based approaches either lack geometric constraints or require extensive storage for feature matching, limiting their practical applications. To address these challenges, we propose an efficient and novel visual localization approach based on the neural implicit map with complementary features. Specifically, to enforce geometric constraints and reduce storage requirements, we implicitly learn a 3D keypoint descriptor field, avoiding the need to explicitly store point-wise features. To further address the semantic ambiguity of descriptors, we introduce additional semantic contextual feature fields, which enhance the quality and reliability of 2D-3D correspondences. Besides, we propose descriptor similarity distribution alignment to minimize the domain gap between 2D and 3D feature spaces during matching. Finally, we construct the matching graph using both complementary descriptors and contextual features to establish accurate 2D-3D correspondences for 6-DoF pose estimation. Compared with the recent NeRF-based approach, our method achieves a 3x faster training speed and a 45x reduction in model storage. Extensive experiments on two widely used datasets demonstrate that our approach outperforms or is highly competitive with other state-of-the-art NeRF-based visual localization methods.

Keywords: Deep Learning for Visual Perception, Visual Learning, Localization
Abstract: Robust and efficient local feature matching plays a crucial role in applications such as SLAM and visual localization for robotics. Despite great progress, it is still very challenging to extract robust and discriminative visual features in scenarios with drastic lighting changes, low texture areas, or repetitive patterns. In this paper, we propose a new lightweight network called LiftFeat, which lifts the robustness of raw descriptor by aggregating 3D geometric feature. Specifically, we first adopt a pre-trained monocular depth estimation model to generate pseudo surface normal label, supervising the extraction of 3D geometric feature in terms of predicted surface normal. We then design a 3D geometry-aware feature lifting module to fuse surface normal feature with raw 2D descriptor feature. Integrating such 3D geometric feature enhances the discriminative ability of 2D feature description in extreme conditions. Extensive experimental results on relative pose estimation, homography estimation, and visual localization tasks, demonstrate that our LiftFeat outperforms some lightweight state-of-the-art methods. Code will be released at : https://github.com/lyp-deeplearning/LiftFeat.

Keywords: Neurorobotics, Deep Learning for Visual Perception, Sensor-based Control
Abstract: The Dynamic Vision Sensor (DVS) is a distinctive visual sensor that exclusively responds to alterations in pixel brightness, enabling the real-time capture of swift and subtle movements with reduced power consumption and data bandwidth requirements. This paper proposes a DVS-aware visual perception method and presents its application for pose estimation of mobile robots. Specifically, a new marker is designed to provide pose reference data that leverages the inherent advantages of DVS more effectively. Moreover, we formulate a pose recognition system incorporating DVS, an algorithm based on Spiking Convolutional Neural Networks (SCNN) and a neuromorphic computing accelerator (Lynxi HS110). Such a formulation can well explore the DVS's advantages, as its event-triggered feature matches the nature of SCNN while the neuromorphic hardware enables efficient, low-power execution, making the system highly suitable for real-time embedded applications. Comparative analysis with traditional ARcode-based pose recognition methods reveals that our innovative approach demonstrates significant advantages in recognition speed and energy efficiency. The whole system is deployed on mobile robots and evaluated in real-world scenarios.

Keywords: Object Detection, Segmentation and Categorization, Aerial Systems: Applications, Aerial Systems: Perception and Autonomy
Abstract: Research in visual perception has shown that the human visual system utilizes high-level feedback information to guide lower-level processing, enabling adaptation to signals of varying characteristics. Inspired by this, we propose the Feedback multi-Level feature EXtractor (Flex) to dynamically adjust feature selection in object detection based on image-wise and instance-level feedback information. This is particularly beneficial for applications such as aerial object detection, UAV-based target recognition and autonomous vehicle navigation, where global image quality issues like sensor degradation, foggy, or rainy conditions can impact detection performance. Flex adapts to variations in image quality, refining the feature extraction process to improve robustness against these challenges. Experimental results demonstrate that Flex consistently enhances a range of state-of-the-art methods on challenging aerial object detection datasets, including DOTA-v1.0, DOTA-v1.5, and HRSC2016. Furthermore, additional experiments on MS COCO confirm the module's effectiveness in general object detection tasks. Our quantitative and qualitative analyses reveal that the improvements are strongly correlated with image quality, aligning with our original motivation to address global image quality issues in real-world scenarios.

Keywords: Visual Tracking, Vision-Based Navigation, Visual Learning
Abstract: Keypoint detection and local feature description are fundamental tasks in robotic perception, critical for applications such as SLAM, robot localization, feature matching, pose estimation, and 3D mapping. While existing methods predominantly operate on RGB images, we propose a novel network that directly processes raw images, bypassing the need for the Image Signal Processor (ISP). This approach significantly reduces hardware requirements and memory consumption, which is crucial for robotic vision systems. Our method introduces two custom-designed convolutional kernels capable of performing convolutions directly on raw images, preserving inter-channel information without converting to RGB. Experimental results show that our network outperforms existing algorithms on raw images, achieving higher accuracy and stability under large rotations and scale variations. This work represents the first attempt to develop a keypoint detection and feature description network specifically for raw images, offering a more efficient solution for resource-constrained environments.

Keywords: Motion and Path Planning, Planning under Uncertainty, Autonomous Agents
Abstract: Reliable automated driving technology is challenged by various sources of uncertainties, in particular, behavioral uncertainties of traffic agents. It is not uncommon for traffic agents to contain multiple intentions followed by distinguishable maneuvers, and the automated driving car must reflect the uncertainty. This paper formalizes a behavior planning scheme in the presence of multiple possible futures with corresponding probabilities. In essence, we present a maximum entropy formulation and show how, under certain assumptions, this allows delayed decision-making to improve safety. The general formulation is then turned into a model predictive control formulation, which is solved as a quadratic program or a set of quadratic programs. We discuss implementation details for improving computation and present validation results in simulation and on a mobile robot.

Keywords: Constrained Motion Planning, Motion and Path Planning, Task and Motion Planning
Abstract: Trajectory prediction facilitates effective planning and decision-making, while constrained trajectory prediction integrates regulation into prediction. Recent advances in constrained trajectory prediction focus on structured constraints by constructing optimization objectives. However, handling unstructured constraints is challenging due to the lack of differentiable formal definitions. To address this, we propose a novel method for constrained trajectory prediction using a conditional generative paradigm, named Controllable Trajectory Diffusion (CTD). The key idea is that any trajectory corresponds to a degree of conformity to a constraint. By quantifying this degree and treating it as a condition, a model can implicitly learn to predict trajectories under unstructured constraints. CTD employs a pre-trained scoring model to predict the degree of conformity (i.e., a score), and uses this score as a condition for a conditional diffusion model to generate trajectories. Experimental results demonstrate that CTD achieves high accuracy on the ETH/UCY and SDD benchmarks. Qualitative analysis confirms that CTD ensures adherence to unstructured constraints and can predict trajectories that satisfy combinatorial constraints.

Keywords: Collision Avoidance, Robot Safety, Mapping
Abstract: SAFER-Splat (Simultaneous Action Filtering and Environment Reconstruction) is a real-time, scalable, and minimally invasive action filter, based on control barrier functions, for safe robotic navigation in a detailed map constructed at runtime using Gaussian Splatting (GSplat). We propose a novel Control Barrier Function (CBF) that not only induces safety with respect to all Gaussian primitives in the scene, but when synthesized into a controller, is capable of processing hundreds of thousands of Gaussians while maintaining a minimal memory footprint and operating at 15 Hz during online Splat training. Of the total compute time, a small fraction of it consumes GPU resources, enabling uninterrupted training. The safety layer is minimally invasive, correcting robot actions only when they are unsafe. To showcase the safety filter, we also introduce SplatBridge, an open-source software package built with ROS for real-time GSplat mapping for robots. We demonstrate the safety and robustness of our pipeline first in simulation, where our method is 20-50x faster, safer, and less conservative than competing methods based on neural radiance fields. Further, we demonstrate simultaneous GSplat mapping and safety filtering on a drone hardware platform using only on-board perception. We verify that under teleoperation a human pilot cannot invoke a collision. Our videos and codebase can be found at https://chengine.github.io/safer-splat.

Keywords: Motion and Path Planning, Reactive and Sensor-Based Planning
Abstract: The capability of autonomous exploration in complex, unknown environments is important in many robotic applications. While recent research on autonomous exploration have achieved much progress, there are still limitations, e.g., existing methods relying on greedy heuristics or optimal path planning are often hindered by repetitive paths and high computational demands.To address such limitations, we propose a novel exploration framework that utilizes the global topology information of observed environment to improve exploration efficiency while reducing computational overhead.Specifically, global information is utilized based on a skeletal topological graph representation of the environment geometry. We first propose an incremental skeleton extraction method based on wavefront propagation, based on which we then design an approach to generate a lightweight topological graph that can effectively capture the environment's structural characteristics. Building upon this, we introduce a finite state machine that leverages the topological structure to efficiently plan coverage paths, which can substantially mitigate the back-and-forth maneuvers (BFMs) problem. Experimental results demonstrate the superiority of our method in comparison with state-of-the-art methods. The source code will be made publicly available at: url{https://github.com/Haochen-Niu/STGPlanner}.

Keywords: Motion and Path Planning, Collision Avoidance, Aerial Systems: Mechanics and Control
Abstract: Autonomous high-speed flight in unknown, cluttered environments is essential for a variety of quadrotor applications, such as inspection, search, and rescue. In this study, we propose a novel trajectory planner designed to achieve efficient, high-speed, collision-free flights in such environments. The proposed approach begins by generating a safe flight corridor based on the path found by Lazy Theta*, representing the safe regions with polytopic sets. These sets are then used to define discrete-time control barrier function (DCBF), ensuring the quadrotor stays within safe bounds during flight. By selecting one single waypoint ahead of the quadrotor on the path as the next waypoint, the trajectory is optimized by considering both the total flight time and safety constraints. Extensive simulations and real-world experiments have confirmed our method's feasibility, demonstrating its capability for high-speed performance and reliable obstacle avoidance.

Keywords: Coverage Path Planning, Motion and Path Planning, Reactive and Sensor-Based Planning, Service Robots
Abstract: In this paper, we propose a method to replan coverage paths for a robot operating in an environment with initially unknown static obstacles. Existing coverage approaches reduce coverage time by covering along the minimum number of coverage lines (straight-line paths). However, recomputing such paths online can be computationally expensive resulting in robot stoppages that increase coverage time. A naive alternative is greedy detour replanning, i.e., replanning with minimum deviation from the initial path, which is efficient to compute but may result in unnecessary detours. In this work, we propose an anytime coverage replanning approach named OARP-Replan that performs near-optimal replans to an interrupted coverage path within a given time budget. We do this by solving linear relaxations of integer linear programs (ILPs) to identify sections of the interrupted path that can be optimally replanned within the time budget. We validate OARP-Replan in simulation and perform comparisons against a greedy detour replanner and other state-of-the-art coverage planners. We also demonstrate OARP-Replan in experiments using an industrial-level autonomous robot.

Keywords: Force and Tactile Sensing, Grasping, Perception for Grasping and Manipulation, Sensor-based Control
Abstract: Grasping is a crucial task in robotics, necessitating tactile feedback and reactive grasping adjustments for robust grasping of objects under various conditions and with differing physical properties. In this article, we introduce LeTac-MPC, a learning-based model predictive control (MPC) for tactile-reactive grasping. Our approach enables the gripper to grasp objects with different physical properties on dynamic and force-interactive tasks. We utilize a vision-based tactile sensor, GelSight (Yuan et al. 2017), which is capable of perceiving high-resolution tactile feedback that contains information on the physical properties and states of the grasped object. LeTac-MPC incorporates a differentiable MPC layer designed to model the embeddings extracted by a neural network from tactile feedback. This design facilitates convergent and robust grasping control at a frequency of 25 Hz. We propose a fully automated data collection pipeline and collect a dataset only using standardized blocks with different physical properties. However, our trained controller can generalize to daily objects with different sizes, shapes, materials, and textures. The experimental results demonstrate the effectiveness and robustness of the proposed approach. We compare LeTac-MPC with two purely model-based tactile-reactive controllers (MPC and PD) and open-loop grasping. Our results show that LeTac-MPC has optimal performance in dynamic and force-interactive tasks and optimal generalizability.

Keywords: Reinforcement Learning, Force and Tactile Sensing, Deep Learning in Grasping and Manipulation
Abstract: Stable and robust robotic grasping is essential for current and future robot applications. In recent works, the use of large datasets and supervised learning has enhanced speed and precision in antipodal grasping. However, these methods struggle with perception and calibration errors due to large planning horizons. To obtain more robust and reactive grasping motions, leveraging reinforcement learning combined with tactile sensing is a promising direction. Yet, there is no systematic evaluation of how the complexity of force-based tactile sensing affects the learning behavior for grasping tasks. This paper compares various tactile and environmental setups using two model-free reinforcement learning approaches for antipodal grasping. Our findings suggest that under imperfect visual perception, various tactile features improve learning outcomes, while complex tactile inputs complicate training.

Keywords: Haptics and Haptic Interfaces, Embodied Cognitive Science, Behavior-Based Systems
Abstract: In order to obtain a good tactile sensing, traditional dexterous hands always enable all the sensing units installed on them all the time, even if just a few sensor units are actually used, which make the tactile sensing system resource-wasting and energy consuming. In order to reduce their complexities by placing the tactile sensing units only at critical locations, this work proposes an embodied tactile dexterous hand (ET-Hand) and a novel multimodal sensor placement framework that learns multiple tasks to generate optimal placement proposal. Furthermore, our ET-Hand can dynamically adjust the perceived tactile sensor positions, types and numbers during robotic manipulation, providing novel tools and methods for investigating the tactile channels and placement scale required for robot exploration. In the object recognition and slip detection tasks, the results show that our proposed method performs close to or even better than traditional sensing way with large-scale placement.

Keywords: Dexterous Manipulation, Assembly, Learning from Demonstration
Abstract: Assembly is a crucial skill for robots in both modern manufacturing and service robotics. However, mastering transferable insertion skills that can handle a variety of high-precision assembly tasks remains a significant challenge. This paper presents a novel framework that utilizes diffusion models to generate 6D wrench for high-precision tactile robotic insertion tasks. It learns from demonstrations performed on a single task and achieves a zero-shot transfer success rate of 95.7% across various novel high-precision tasks. Our method effectively inherits the self-adaptability demonstrated by our previous work. In this framework, we address the frequency misalignment between the diffusion policy and the real-time control loop with a dynamic system-based filter, significantly improving the task success rate by 9.15%. Furthermore, we provide a practical guideline regarding the trade-off between diffusion models' inference ability and speed.

Keywords: Dexterous Manipulation, Representation Learning, Reinforcement Learning
Abstract: Visual and tactile pretraining have been extensively studied in dexterous robot manipulation tasks. However, existing methods typically require the simultaneous acquisition of visual and tactile data, making it difficult to utilize low-cost, unpaired visual-tactile datasets. Moreover, these methods often rely on tactile sensors to provide input data for reinforcement learning (RL) during the physical deployment of robotic dexterous hands, which highly increases deployment costs. To address these challenges, we propose UpViTaL, an unpaired visual- tactile self-supervised representation learning method for RL- based robot dexterous manipulation. Specifically, we collect low-cost unpaired visual and tactile datasets for manipulation skill learning using a camera and tactile gloves on three robot manipulation tasks. The temporal tactile self-supervised representation learning module of UpViTaL is used to explore efficient tactile representations from time-series tactile data. In parallel, the visual pretraining module of UpViTaL helps to extract efficient visual representations from visual data. In addition, we fuse unpaired visual-tactile representations through an RL reward mechanism, which does not require robotic dexterous hands tactile sensors for practical deployment. We validate our approach on three dexterous robot manipulation tasks. Experimental results demonstrate that UpViTaL can efficiently learn robot manipulation skills. Compared to existing approaches for visual pretraining, our method significantly improves the success rate by more than 30%.

Keywords: Acceptability and Trust, Human-Robot Teaming, Probability and Statistical Methods
Abstract: Shared autonomy functions as a flexible framework that empowers robots to operate across a spectrum of autonomy levels, allowing for efficient task execution with minimal human oversight. However, humans might be intimidated by the autonomous decision-making capabilities of robots due to perceived risks and a lack of trust. This paper proposed a trust-preserved shared autonomy strategy that allows robots to seamlessly adjust their autonomy level, striving to optimize team performance and enhance their acceptance among human collaborators. By enhancing the relational event modeling framework with Bayesian learning techniques, this paper enables dynamic inference of human trust based solely on time-stamped relational events communicated within human-robot teams. Adopting a longitudinal perspective on trust development and calibration in human-robot teams, the proposed trust-preserved shared autonomy strategy warrants robots to actively establish, maintain, and repair human trust, rather than merely passively adapting to it. We validate the effectiveness of the proposed approach through a user study on a human-robot collaborative search and rescue scenario. The objective and subjective evaluations demonstrate its merits on both task execution and user acceptability over the baseline approach that does not consider the preservation of trust.

Keywords: Human Factors and Human-in-the-Loop, Acceptability and Trust, Human-Robot Collaboration
Abstract: In the Industry 5.0 era, the focus shifts from basic automation to fostering collaboration between humans and robots. Trust is crucial in this new paradigm, enabling smooth interaction, especially for users with limited robotics knowledge. This study presents a novel framework that uses human hand gestures and voice commands to control robot movements, aiming to enhance trust, reduce cognitive workload, and minimize task execution time—key for efficient manufacturing. In automated systems, swift completion of micromanagement tasks is essential to prevent process disruption. To evaluate this framework, we devised a testbed scenario within an automated carbon fiber transportation and draping process, focusing on a maintenance task as the micromanagement challenge. Participants inspected the gripper, guided the robot along a defined path, and performed maintenance, such as attaching cables. Two conditions were tested: gestures and voice commands versus a smartPAD. The results showed that gestures and voice commands increased trust, lowered cognitive load, and shortened execution times, improving overall manufacturing efficiency.

Keywords: Acceptability and Trust, Design and Human Factors, Human-Robot Collaboration
Abstract: As robots are prone to make errors that undermine trust, effective trust repair strategies are essential in effective human-robot collaboration. Our lab study evaluates three trust repair strategies --apology, denial, and compensation-- following two types of trust violations: competence-based and integrity-based. Consistent with prior research, integrity-based violations reduced moral trust more, while competence-based violations impacted performance trust. Denial caused greater discomfort than apology or compensation across both violation types. Dispositional trust influenced repair strategies effectiveness, particularly in willingness to engage and re-engage. Notably, individuals with high dispositional trust were more receptive to apologies. These findings underscore the need to consider individual trust differences, suggesting robots should assess human trust disposition to effectively foster continued collaboration.

Keywords: Acceptability and Trust
Abstract: With robots becoming increasingly prevalent in various domains, it has become crucial to equip them with tools to achieve greater fluency in interactions with humans. One of the promising areas for further exploration lies in human trust. A real-time, objective model of human trust could be used to maximize productivity, preserve safety, and mitigate failure. In this work, we attempt to use physiological measures, gaze, and facial expressions to model human trust in a robot partner. We are the first to design an in-person, human-robot supervisory interaction study to create a dedicated trust dataset. Using this dataset, we train machine learning algorithms to identify the objective measures that are most indicative of trust in a robot partner, advancing trust prediction in human-robot interactions. Our findings indicate that a combination of sensor modalities (blood volume pulse, electrodermal activity, skin temperature, and gaze) can enhance the accuracy of detecting human trust in a robot partner. Furthermore, the Extra Trees, Random Forest, and Decision Trees classifiers exhibit consistently better performance in measuring the person's trust in the robot partner. These results lay the groundwork for constructing a real-time trust model for human-robot interaction, which could foster more efficient interactions between humans and robots.

Keywords: Acceptability and Trust, Human-Robot Teaming
Abstract: When humans collaborate, they form positive or negative experiences with each other. These experiences depend on various factors such as the individual's skills, abilities, and agency. In this paper, we consider human-robot collaborations and present a novel model of an autonomous robot's trust in humans based on the probability of the robot having a positive experience with the human. The model defines a dynamic trust-building process that translates into a computationally-accessible implementation. We hypothesize predictors of a positive experience with human teammates and derive trust in individual humans. As the interactions continue, team members develop an affinity toward each other. The robot's affinity towards humans can be viewed as kinship, and we also investigate how kinship affects trust and distrust. We present an algorithm for how the robot may use kinship-mediated trust in its decision-making, and demonstrate its use in simulated missions truly requiring human-robot collaboration.

Keywords: Acceptability and Trust, Human-Robot Collaboration, Design and Human Factors
Abstract: With increasing efficiency and reliability, autonomous systems are becoming valuable assistants to humans in various tasks. In the context of robot-assisted delivery, we investigate how robot performance and trust repair strategies impact human trust. In this task, humans can choose to either send the robot to deliver autonomously or manually control it while handling a secondary task. The trust repair strategies examined include short and long explanations, apology and promise, and denial. Using data from human participants, we model human behavior using an Input-Output Hidden Markov Model (IOHMM) to capture the dynamics of trust and human action probabilities. Our findings indicate that humans are more likely to deploy the robot autonomously when their trust is high. Furthermore, state transition estimates show that long explanations are the most effective at repairing trust following a failure, while denial is most effective at preventing trust loss. We also demonstrate that the trust estimates generated by our model are isomorphic to self-reported trust values, making them interpretable. This model lays the groundwork for developing optimal policies that facilitate real-time adjustment of human trust in autonomous systems.

Keywords: Manipulation Planning, Mobile Manipulation
Abstract: Finding an high-quality solution for the tabletop object rearrangement planning is a challenging problem. Compared to determining a goal arrangement, rearrangement planning is challenging due to the dependencies between objects and the buffer capacity available to hold objects. Although ORLA* has proposed an A* based searching strategy with lazy evaluation for the optimal solution, it is not scalable, with the success rate decreasing as the number of objects increases. Additionally, for noisy state representations, ORLA* provides only suboptimal solutions. To overcome these limitations, we propose an enhanced A*-based algorithm that improves state representation and employs incremental goal attempts with lazy evaluation at each iteration. This approach aims to enhance scalability while maintaining solution quality. Our evaluation demonstrates that our algorithm can provide superior solutions compared to ORLA*, in a shorter time, for both stationary and mobile robots.

Keywords: Dual Arm Manipulation, Compliance and Impedance Control
Abstract: Dual-arm manipulation is an area of growing interest in the robotics community. Enabling robots to perform tasks that require the coordinated use of two arms, is essential for complex manipulation tasks such as handling complex large objects, assembling components, and performing human-like interactions. However, achieving effective dual-arm manipulation is challenging due to the need for precise coordination, dynamic adaptability, and the ability to manage interaction forces between the arms and the objects being manipulated. We propose a novel pipeline that combines advantages of policy learning based on environment feedback and gradient based optimization to learn controller gains as well as the control outputs. This allows the robotic system to dynamically modulate its impedance in response to task demands, ensuring stability and dexterity in dual-arm operations. We evaluate our pipeline on a trajectory-tracking task involving a variety of large, complex objects with different masses and geometries. The performance is then compared to three other established methods for controlling dual-arm robots, demonstrating superior results.

Keywords: Dexterous Manipulation, Manipulation Planning, Model Learning for Control
Abstract: Robotic pushing is one of the intuitive non-prehensile manipulation skills that can handle ungraspable objects without any complex task-specific tools. In this paper, we proposed a goal-driven accurate robotic pushing framework to achieve the robotic pushing tasks in practice that can operate under uncertain object properties. We employed a model predictive path integral (MPPI) as a goal-driven pushing controller building upon our prior work to operate pushing tasks under uncertain object properties. Unlike our prior work, the proposed framework can push the object toward the goal pose without predefined trajectories. The results of the numerical experiments demonstrated that the proposed framework can accomplish the pushing task with a significantly shorter total length, smaller total step, and a higher success rate even though the model parameters are unknown. Moreover, we demonstrated the proposed framework also works well in the real world through real-robot demonstrations.

Keywords: Grasping, Manipulation Planning
Abstract: In complex manipulation tasks, e.g., manipulation by pivoting, the motion of the object being manipulated has to satisfy path constraints that can change during the motion. Therefore, a single grasp may not be sufficient for the entire path, and the object may need to be regrasped. Additionally, geometric data for objects from a sensor are usually available in the form of point clouds. The problem of computing grasps and regrasps from point-cloud representation of objects for complex manipulation tasks is a key problem in endowing robots with manipulation capabilities beyond pick-and-place. In this paper, we formalize the problem of grasping/regrasping for complex manipulation tasks with objects represented by (partial) point clouds and present an algorithm to solve it. We represent a complex manipulation task as a sequence of constant screw motions. Using a manipulation plan skeleton as a sequence of constant screw motions, we use a grasp metric to find graspable regions on the object for every constant screw segment. The overlap of the graspable regions for contiguous screws are then used to determine when and how many times the object needs to be regrasped. We present experimental results on point cloud data collected from RGB-D sensors to illustrate our approach.

Keywords: RGB-D Perception, Probability and Statistical Methods, Learning from Demonstration
Abstract: Visual uncertainties such as occlusions, lack of texture, and noise present significant challenges in obtaining accurate kinematic models for safe robotic manipulation. We introduce a probabilistic real-time approach that leverages the human hand as a prior to mitigate these uncertainties. By tracking the constrained motion of the human hand during manipulation and explicitly modeling uncertainties in visual observations, our method reliably estimates an object’s kinematic model online. We validate our approach on a novel dataset featuring challenging objects that are occluded during manipulation and offer limited articulations for perception. The results demonstrate that by incorporating an appropriate prior and explicitly accounting for uncertainties, our method produces accurate estimates, outperforming two recent baselines by 195% and 140%, respectively. Furthermore, we demonstrate that our approach's estimates are precise enough to allow a robot to manipulate even small objects safely.

Keywords: Dexterous Manipulation, Multi-Contact Whole-Body Motion Planning and Control, Optimization and Optimal Control
Abstract: Designing planners and controllers for contact-rich manipulation is extremely challenging as contact violates the smoothness conditions that many gradient-based controller synthesis tools assume. Contact smoothing approximates a non-smooth system with a smooth one, allowing one to use these synthesis tools more effectively. However, applying classical control synthesis methods to smoothed contact dynamics remains relatively under-explored. This paper analyzes the efficacy of linear controller synthesis using differential simulators based on contact smoothing. We introduce natural baselines for leveraging contact smoothing to compute (a) open-loop plans robust to uncertain conditions and/or dynamics, and (b) feedback gains to stabilize around open-loop plans. Using robotic bimanual whole-body manipulation as a testbed, we perform extensive empirical experiments on over 300 trajectories and analyze why LQR seems insufficient for stabilizing contact-rich plans.

Keywords: Semantic Scene Understanding, Cognitive Control Architectures, Embodied Cognitive Science
Abstract: Robots powered by large language model (LLM) demonstrate significant research and application potential by effectively interpreting scene information to respond to human commands. However, when robots rely on static scene information during task execution, they face difficulties in adapting to changes in the environment, posing a major challenge for dynamic scene perception. To address the above issues, we propose an innovative interaction-driven approach to enhance robots' ability to perceive dynamic scene information. This approach consists of two contributions, the observation point selection module and the dynamic scene maintenance module. Specifically, first, the robot uses the 3D scene graph (3DSG) containing assets and objects to perceive static scene information through the LLM planner. Next, the best observation point for each asset is obtained through the observation point selection module. Then, with the help of the best observation point, the dynamic scene maintenance module interacts with the asset-related objects to dynamically update all the object node information related to the asset node. This approach enables robots to maintain dynamic scene information, enhancing their adaptability in unpredictable environments and improving task reliability.We evaluated our method using the iTHOR and RoboTHOR datasets within the AI2-THOR simulator and in real-world scenarios. Experimental results demonstrate that our method effectively and accurately maintains robots' perception of dynamic scene information.

Keywords: Motion and Path Planning, AI-Based Methods, Integrated Planning and Learning
Abstract: Planning a kinodynamically feasible path for a tractor-trailer vehicle is challenging for both search-based and learning-based methods due to the vehicle’s unique kinematics and complex obstacles. These factors increase the likelihood of infeasible paths and exacerbate long-horizon issues. We introduce ME-PATS: a framework that mutually enhances the search-based planner and the learning-based agent for tractor-trailer systems. The search-based planner provides successful trajectories to help the learning-based agent update its policy, while the agent improves the planner’s efficiency through direct path simulation. Additionally, we propose two approaches to apply our framework to more challenging tasks: designing obstacle-aware networks to enhance the learning-based agent’s capabilities, and combining the planner’s paths with the trained agent’s simulated paths through multi-segment integration. Full details and results are available on our project website at href{https://github.com/FrankSinatral/TTsystems}{https://g ithub.com/FrankSinatral/TTsystems}.

Keywords: AI-Enabled Robotics, Robot Safety, Machine Learning for Robot Control
Abstract: The recent introduction of large language models (LLMs) has revolutionized the field of robotics by enabling contextual reasoning and intuitive human-robot interaction in domains as varied as manipulation, locomotion, and self-driving vehicles. When viewed as a stand-alone technology, LLMs are known to be vulnerable to jailbreaking attacks, wherein malicious prompters elicit harmful text by bypassing LLM safety guardrails. To assess the risks of deploying LLMs in robotics, in this paper, we introduce RoboPAIR, the first algorithm designed to jailbreak LLM-controlled robots. Unlike existing, textual attacks on LLM chatbots, RoboPAIR elicits harmful physical actions from LLM-controlled robots, a phenomenon we experimentally demonstrate in three scenarios: (i) a white-box setting, wherein the attacker has full access to the NVIDIA Dolphins self-driving LLM, (ii) a gray-box setting, wherein the attacker has partial access to a Clearpath Robotics Jackal UGV robot equipped with a GPT-4o planner, and (iii) a black-box setting, wherein the attacker has only query access to the GPT-3.5-integrated Unitree Robotics Go2 robot dog. In each scenario and across three new datasets of harmful robotic actions, we demonstrate that RoboPAIR, as well as several static baselines, finds jailbreaks quickly and effectively, often achieving 100% attack success rates. Our results reveal, for the first time, that the risks of jailbroken LLMs extend far beyond text generation, given the distinct possibility that jailbroken robots could cause physical damage in the real world. Indeed, our results on the Unitree Go2 represent the first successful jailbreak of a deployed commercial robotic system. Addressing this emerging vulnerability is critical for ensuring the safe deployment of LLMs in robotics. Additional media is available at: https://robopair.org.

Keywords: AI-Enabled Robotics, Task and Motion Planning
Abstract: Large Language Models (LLMs) have demonstrated remarkable ability in long-horizon Task and Motion Planning (TAMP) by translating clear and straightforward natural language problems into formal specifications such as the Planning Domain Definition Language (PDDL). However, real-world problems are often ambiguous and involve many complex constraints. In this paper, we introduce Constraints as Specifications through LLMs (CaStL), a framework that identifies constraints such as goal conditions, action ordering, and action blocking from natural language in multiple stages. CaStL translates these constraints into PDDL and Python scripts, which are solved using an custom PDDL solver. Tested across three PDDL domains, CaStL significantly improves constraint handling and planning success rates from natural language specification in complex scenarios.

Keywords: AI-Enabled Robotics, Learning from Demonstration, Big Data in Robotics and Automation
Abstract: This study explores the utility of various internet data sources to select among a set of template robot behaviors to perform skills. Learning contact-rich skills involving tool use from internet data sources has typically been challenging due to the lack of physical information such as contact existence, location, areas, and force in this data. Prior works have generally used internet data and foundation models trained on this data to generate low-level robot behavior. We hypothesize that these data and models may be better suited to selecting among a set of basic robot behaviors to perform these contact-rich skills. We explore three methods of template selection: querying large language models, comparing video of robot execution to retrieved human video using features from a pretrained video encoder common in prior work, and performing the same comparison using features from an optic flow encoder trained on internet data. Our results show that LLMs are surprisingly capable template selectors despite their lack of visual information, optical flow encoding significantly outperforms video encoders trained with an order of magnitude more data, and important synergies exist between various forms of internet data for template selection. By exploiting these synergies, we create a template selector using multiple forms of internet data that achieves a 79% success rate on a set of 16 different cooking skills involving tool-use.

Keywords: AI-Enabled Robotics, Dexterous Manipulation, Compliant Assembly
Abstract: Large Language Models (LLMs) have gained popularity in task planning for long-horizon manipulation tasks. To enhance the validity of LLM-generated plans, visual demonstrations and online videos have been widely employed to guide the planning process. However, for manipulation tasks involving subtle movements but rich contact interactions, visual perception alone may be insufficient for the LLM to fully interpret the demonstration. Additionally, visual data provides limited information on force-related parameters and conditions, which are crucial for effective execution on real robots.

In this paper, we introduce an in-context learning framework that incorporates tactile and force-torque information from human demonstrations to enhance the LLM's ability to generate plans for new task scenarios. We propose a bootstrapped reasoning pipeline that sequentially integrates each modality into a comprehensive task plan. This task plan is then used as a reference for planning in new task configurations. Real-world experiments on two different sequential manipulation tasks demonstrate the effectiveness of our framework in improving LLMs' understanding of multi-modal demonstrations and enhancing the overall planning performance.

Keywords: Imitation Learning, Machine Learning for Robot Control, Safety in HRI
Abstract: Operating effectively in complex environments while complying with specified constraints is crucial for the safe and successful deployment of robots that interact with and operate around people. In this work, we focus on generating long-horizon trajectories that adhere to novel static and temporally-extended constraints/instructions at test time. We propose a data-driven diffusion-based framework, LTLDoG, that modifies the inference steps of the reverse process given an instruction specified using finite linear temporal logic (LTLf). LTLDoG leverages a satisfaction value function on LTLf and guides the sampling steps using its gradient field. This value function can also be trained to generalize to new instructions not observed during training, enabling flexible test-time adaptability. Experiments in robot navigation and manipulation illustrate that the method is able to generate trajectories that satisfy formulae that specify obstacle avoidance and visitation sequences.

Keywords: View Planning for SLAM, Deep Learning Methods, Motion and Path Planning
Abstract: Autonomous robot exploration requires a robot to efficiently explore and map unknown environments. Compared to conventional methods that can only optimize paths based on the current robot belief, learning-based methods show the potential to achieve improved performance by drawing on past experiences to reason about unknown areas. In this paper, we propose DARE, a novel generative approach that leverages diffusion models trained on expert demonstrations, which can explicitly generate an exploration path through one-time inference. We build DARE upon an attention-based encoder and a diffusion model, and introduce ground truth optimal demonstrations for training to learn better patterns for exploration. The trained planner can reason about the partial belief to recognize the potential structure in unknown areas and consider these areas during path planning. Our experiments demonstrate that DARE achieves on-par performance with both conventional and learning-based state-of-the-art exploration planners, as well as good generalizability in both simulations and real-life scenarios.

Keywords: Vision-Based Navigation, Integrated Planning and Learning, Imitation Learning
Abstract: Visual navigation, a fundamental challenge in mobile robotics, demands versatile policies to handle diverse environments. Classical methods leverage geometric solutions to minimize specific costs, offering adaptability to new scenarios but are prone to system errors due to their multi-modular design and reliance on hand-crafted rules. Learning-based methods, while achieving high planning success rates, face difficulties in generalizing to unseen environments beyond the training data and often require extensive training. To address these limitations, we propose a hybrid approach that combines the strengths of learning-based methods and classical approaches for RGB-only visual navigation. Our method first trains a conditional diffusion model on diverse path-RGB observation pairs. During inference, it integrates the gradients of differentiable scene-specific and task-level costs, guiding the diffusion model to generate valid paths that meet the constraints. This approach alleviates the need for retraining, offering a plug-and-play solution. Extensive experiments in both indoor and outdoor settings, across simulated and real-world scenarios, demonstrate zero-shot transfer capability of our approach, achieving higher success rates and fewer collisions compared to baseline methods. Code will be released at url{https://github.com/SYSU-RoboticsLab/NaviD}.

Keywords: Vision-Based Navigation, Deep Learning for Visual Perception, Visual Learning
Abstract: Navigating unfamiliar environments presents significant challenges for household robots, requiring the ability to recognize and reason about novel decoration and layout. Existing reinforcement learning methods cannot be directly transferred to new environments, as they typically rely on extensive mapping and exploration, leading to time-consuming and inefficient. To address these challenges, we try to transfer the logical knowledge and the generalization ability of pre-trained foundation models to zero-shot navigation. By integrating a large vision-language model with a diffusion network, our approach named NavigateDiff constructs a visual predictor that continuously predicts the agent's potential observations in the next step which can assist robots generate robust actions. Furthermore, to adapt the temporal property of navigation, we introduce temporal historical information to ensure that the predicted image is aligned with the navigation scene. We then carefully designed an information fusion framework that embeds the predicted future frames as guidance into goal-reaching policy to solve downstream image navigation tasks. This approach enhances navigation control and generalization across both simulated and real-world environments. Through extensive experimentation, we demonstrate the robustness and versatility of our method, showcasing its potential to improve the efficiency and effectiveness of robotic navigation in diverse settings. Project Page: https://21styouth.github.io/NavigateDiff/.

Keywords: Imitation Learning, Reinforcement Learning, Sensorimotor Learning
Abstract: Imitation learning from human demonstrations enables robots to perform complex manipulation tasks and has recently witnessed huge success. However, these techniques often struggle to adapt behavior to new preferences or changes in the environment. To address these limitations, we propose Fine-tuning Diffusion Policy with Human Preference (FDPP). FDPP learns a reward function through preference-based learning. This reward is then used to fine-tune the pre-trained policy with reinforcement learning (RL), resulting in alignment of pre-trained policy with new human preferences while still solving the original task. Our experiments across various robotic tasks and preferences demonstrate that FDPP effectively customizes policy behavior without compromising performance. Additionally, we show that incorporating Kullback–Leibler (KL) regularization during fine-tuning prevents over-fitting and helps maintain the competencies of the initial policy.

Keywords: Imitation Learning, Learning from Demonstration, Constrained Motion Planning
Abstract: Advances in robot learning have enabled robots to generate skills for a variety of tasks. Yet, robot learning is typically sample inefficient, struggles to learn from data sources exhibiting varied behaviors, and does not naturally incorporate constraints. These properties are critical for fast, agile tasks such as playing table tennis. Modern techniques for learning from demonstration improve sample efficiency and scale to diverse data, but are rarely evaluated on agile tasks. In the case of reinforcement learning, achieving good performance requires training on high-fidelity simulators. To overcome these limitations, we develop a novel diffusion modeling approach that is offline, constraint-guided, and expressive of diverse agile behaviors. The key to our approach is a kinematic constraint gradient guidance (KCGG) technique that computes gradients through both the forward kinematics of the robot arm and the diffusion model to direct the sampling process. KCGG minimizes the cost of violating constraints while simultaneously keeping the sampled trajectory in-distribution of the training data. We demonstrate the effectiveness of our approach for time-critical robotic tasks by evaluating KCGG in two challenging domains: simulated air hockey and real table tennis. In simulated air hockey, we achieved a 25.4% increase in block rate, while in table tennis, we achieved a 17.3% increase in success rate compared to imitation learning baselines.

Keywords: Humanoid and Bipedal Locomotion, Planning under Uncertainty, Optimization and Optimal Control
Abstract: This work presents a high-level navigation framework for bipedal robot that accounts for terrain uncertainty in footstep planning and motion control. Given a terrain map estimated via Gaussian Process (GP) model with nonstationary kernel, we leverage Conformal Prediction (CP) to construct tighter coverage guarantee intervals containing the true terrain elevations. We also formulate a soft CP constraint to ensure safe foot height changes between successive footfalls with probabilistic guarantee. Additionally, we model the CP coverage intervals as a bounded disturbance model which are incorporated into linear inverted pendulum plus flywheel (LIP) model. Given the LIP model with bounded disturbances arising from uncertain terrain, we formulate a flywheel torque control law using Control Contraction Metrics (CCMs). We then design a Robust Control Invariant (RCI) Tube around the desired Center of Mass (CoM) phase-space trajectory, defining a region in which the system states can be stabilized by the flywheel torque control law. This ensures the maintenance of a constant CoM height (or constant surface slope) despite terrain uncertainty. The overall framework results in an uncertainty-informed Model Predictive Controller (MPC) provides probabilistic safety guarantees, enabling robust locomotion across uncertain environments. This approach enhances the robot’s ability to navigate complex, unstructured terrains while maintaining stability in real-world deployment.

Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles, Surgical Robotics: Planning
Abstract: To address the steep learning curve associated with ESD, we reengineered the tissue clip by introducing a magnetic mechanism to improve submucosal exposure and facilitate precise dissection. Furthermore, we proposed a robotic manipulation method that automates the motion of an external magnet mounted on a robotic arm to enhance operational efficiency. Based on simulation trials using ROS Gazebo (ICRA 2025 accepted, manuscript # 4388), which demonstrated promising results, we further evaluated the proposed method in a 3D-printed stomach and ex vivo setup using a clinical gastroscope (Olympus EVIS EXERA III GIF-1TH190). By labeling and training over 21,000 images captured from the gastroscope using a deep learning-based YOLO (You Only Look Once) method, we successfully deployed the magnetic robotic tissue manipulation system, integrating the robotic arm, Olympus gastroscope, mock-up and ex vivo tissue. In 42 trials using a 3D-printed stomach, the magnetic robotic manipulation system was able to orient the internal magnetic clip—clamped onto the tissue—to the desired angle (±35 degrees) in 55.36 ± 7.64 seconds (mean ± standard error, n = 42). Additionally, ex vivo trials using porcine stomach tissue recorded a performance time of 37.19 ± 3.78 seconds over 36 trials. These results lay the groundwork for future validation in in vivo scenarios.

Keywords: Human-Robot Teaming, Human-Robot Collaboration
Abstract: As robots increasingly integrate into human environments, there is growing optimism about expanding and diversifying human-robot team (HRT) missions. However, existing approaches, which primarily rely on teleoperation and contingency handling, encounter limitations such as cognitive overload and scalability challenges. A user study was conducted to investigate how Shared Mental Models (SMMs) influence HRTs' problem-solving abilities when faced with novel challenges. The results revealed that a General SMM (GSMM) significantly improved task performance compared to a Specific SMM (SSMM), although the SMM type did not significantly impact overall team adaptability. Additionally, the ability to handle uncertain situations was crucial for performance.

Keywords: Imitation Learning, Machine Learning for Robot Control, Deep Learning in Grasping and Manipulation
Abstract: Recent advances in diffusion/flow policies have enabled imitation learning of complex, multi-modal action trajectories for robotic control. However, they are slow because they sample a trajectory of trajectories—a diffusion/flow trajectory of robot trajectories. They discard intermediate robot trajectories, and must wait for the sampling process to complete before any actions can be executed on the robot. In this work, we propose a novel framework that simplifies diffusion/flow policies by treating robot trajectories as flow trajectories. Instead of starting from pure noise, our algorithm starts from the current robot configuration, and incrementally integrates a velocity field learned via flow matching to produce a sequence of robot configurations that constitute a single trajectory. This enables computed actions to be streamed to the robot on-the-fly during the flow sampling process, for significantly faster and reactive policies. It is well-suited for receding horizon control where we can adaptively generate only as many actions as are executed on the robot, and no more. Despite streaming, our method retains the ability to model multi-modal behavior. We show that training flows that stabilize around demonstration trajectories reduces distribution shift and improves imitation learning performance. Streaming flow policy outperforms prior diffusion/flow policies on imitation learning benchmarks while enabling faster policy execution and tighter sensorimotor control loops.

Keywords: Machine Learning for Robot Control, Visual Tracking, Micro/Nano Robots
Abstract: The goal of this project is to enable the use of efficient on-device deep learning models for battery-free mobility-based sensing robots. We target the deployment of these models on MilliMobile, a one square centimeter microrobotic platform with only 512 KB of RAM and 1 MB of flash memory (millimobile.cs.washington.edu), making it challenging to run traditional models onboard. We address this challenge by integrating intermittent computing and motion with image-based sensing, ensuring that the system only performs high-power actions–like locomotion, inference, and communication–when specific conditions are satisfied. Event-based vision and intermittent computing approaches will also be leveraged to optimize the MCUs time in ultra-low-power modes. We train on insect images from Insect Detect - insect classification dataset v2 and achieve almost 70% accuracy after 500 epochs with milliWatts of power consumed when running the system on the nRF52840 microcontroller.

Keywords: Social HRI, Human-Aware Motion Planning, Service Robotics
Abstract: Robots operating in public spaces must navigate alongside humans in ways that are not only safe but also intuitive and socially acceptable. In particular, how a robot communicates its navigation intent can significantly impact pedestrian comfort and trust. In this work, we explore how different robot behaviors - ranging from no cue to implicit (speed and trajectory adjustments) and explicit (verbal) signals - affect pedestrian perception during hallway encounters. We conducted a pilot user study with 12 participants, where a robot approached the participant from the front or side using one of five cue strategies. Subjective ratings (comfort, trust, clarity, predictability, and proxemics) and objective measures (hesitation, passing time) were collected. Results indicate that trajectory cues led to the highest perceived comfort and clarity, while sudden stops and no cues caused more confusion and hesitation. These findings highlight the potential of motion-based implicit cues to improve the legibility of robot navigation in shared human environments.

Keywords: Probabilistic Inference, Probability and Statistical Methods, AI-Based Methods
Abstract: We present a novel framework integrating physics and machine learning to estimate frequency-domain acoustic wave propagation coefficients. Using acoustic waveforms captured at speaker-receiver pairs, we estimate attenuation and wavenumber via: (1) Bayesian inference for uncertainty-aware learning from small and noisy data, (2) a neural-physical model trained with forward-backward physical losses, and (3) non-linear least squares as baseline. With inferred propagation coefficients, room impulse responses (RIRs) are derived, enabling robot relocalisation with uncertainty quantification.

Keywords: Search and Rescue Robots, Design and Human Factors, Human-Robot Teaming
Abstract: The use of autonomous systems in medical evacuation (MEDEVAC) scenarios is promising, but existing implementations overlook key insights from human-robot interaction (HRI) research. Studies on human-machine teams demonstrate that human perceptions of a machine teammate are critical in governing the machine's performance. Consequently, it is essential to identify the factors that contribute to positive human perceptions in human-machine teams. Here, we present a mixed factorial design to assess human perceptions of a MEDEVAC robot in a simulated evacuation scenario. Participants were assigned to the role of casualty (CAS) or bystander (BYS) and subjected to three within-subjects conditions based on the MEDEVAC robot's operating mode: autonomous-slow (AS), autonomous-fast (AF), and teleoperation (TO). During each trial, a MEDEVAC robot navigated an 11-meter path, acquiring a casualty and transporting them to an ambulance exchange point while avoiding an idle bystander. Following each trial, subjects completed a questionnaire measuring their emotional states, perceived safety, and social compatibility with the robot. Results indicate a consistent main effect of operating mode on reported emotional states and perceived safety. Pairwise analyses suggest that the employment of the AF operating mode negatively impacted perceptions along these dimensions. There were no persistent differences between CAS and BYS responses.

Keywords: Safety in HRI, Human-Robot Collaboration, Human-Robot Teaming
Abstract: Human safety is critical in applications involving close human-robot interactions (HRI) and is a key aspect of physical compatibility between humans and robots. While measures of human safety in HRI exist, these mainly target industrial settings involving robotic manipulators. Less attention has been paid to settings where mobile robots and humans share the space. This paper introduces a new robot-centered directional framework of human safety. It is particularly useful for evaluating mobile robots as they operate in environments populated by multiple humans. The framework integrates several key metrics, such as each human's relative distance, speed, and orientation. The core novelty lies in the framework's flexibility to accommodate different application requirements while allowing for both the robot-centered and external observer points of view. We instantiate the framework by using RGB-D based vision integrated with a deep learning-based human detection pipeline to yield a proxemics-guided generalized safety index (GSI) that instantaneously assesses human safety. We extensively validate GSI's capability of producing appropriate and fine-grained safety measures in real-world experimental scenarios and demonstrate its superior efficacy against extant safety models.

Keywords: Soft Robot Materials and Design, Soft Sensors and Actuators, Compliant Joints and Mechanisms
Abstract: Tensegrity robots excel in tasks requiring extreme levels of deformability and robustness. However, there are challenges in state estimation and payload versatility due to their high number of degrees of freedom and unconventional shape. This work introduces a modular three-bar tensegrity robot featuring a customizable payload design. The modular exoskeleton supports scalable configurations and efficient packaging of sensing and computation Our tensegrity robot employs a novel Quasi-Direct Drive (QDD) cable actuator with low-stretch polymer cables to achieve accurate proprioception without needing external force or torque sensors. The design allows for on-the-fly stiffness tuning for better environment and payload adaptability. Experimental data demonstrates the high accuracy cable length estimation (<1% error relative to bar length) and variable stiffness control of the cable actuator up to 7 times the minimum stiffness for self support. To augment proprioception with environmental feedback, we develop a scalable, open-source visuotactile sensor embedded within each endcap of the tensegrity structure. The presented tensegrity robot is a platform for future advancements in autonomous operation and open-source module design.

Keywords: Marine Robotics, Simulation and Animation, Field Robots
Abstract: Underwater simulators offer support for building robust underwater perception solutions. Significant work has recently been done to develop new simulators and to advance the performance of existing underwater simulators. Still, there remains room for improvement on physics-based underwater sensor modeling and rendering efficiency. In this paper, we propose OceanSim, a high-fidelity GPU-accelerated underwater simulator to address this research gap. We propose advanced physics-based rendering techniques to reduce the sim-to-real gap for underwater image simulation. We develop OceanSim to fully leverage the computing advantages of GPUs and achieve real-time imaging sonar rendering and fast synthetic data generation. We evaluate the capabilities and realism of OceanSim using real-world data to provide qualitative and quantitative results.

Keywords: Intention Recognition, Physical Human-Robot Interaction, Physically Assistive Devices
Abstract: Many robotic assistive devices require users to perform explicit commanding actions, such as pressing buttons or tilting their bodies, to initiate assistance. However, it would be more natural if the device could infer intent implicitly. Prior work has explored intent prediction using different sensor modalities, such as electromyography (EMG), electroencephalography (EEG), and inertial measurement units (IMUs). Some vision-based approaches rely on external cameras to analyze human action from a third person perspective. While these methods can provide useful insights, they either focus only on body signals without considering environmental context or require external cameras that are impractical for wearable robotics. In contrast, first-person vision offers a direct perspective of how users interact with their surroundings, yet it remains underexplored in intent inference research. We propose a hybrid approach that utilizes a Bayesian network to probabilistically fuse first-person visual context with body motion data to understand human intent. Unlike some prior work that uses computer vision only for perception, our approach reasons about the contextual affordances of objects in environment and integrates it with human motion to improve both prediction accuracy and explainability. Our results show the feasibility of this approach in controlled experiments, and future research could expand on this work to improve robustness and generalizability in real-world environments.

Keywords: Rehabilitation Robotics, Modeling, Control, and Learning for Soft Robots, Soft Robot Applications
Abstract: Soft robotic systems offer remarkable adaptability and safety in dynamic environments, but their compliant structures introduce complex, nonlinear behaviors that challenge traditional modeling and control techniques. This paper proposes a unified modeling and control framework for pneumatic artificial muscles (PAMs), with emphasis on McKibben actuators in both rectilinear and curvilinear configurations. Recognizing the limitations of linear modeling for soft robotics, the framework is extended to include nonlinearities of soft body dynamics by incorporating nonlinear stiffness and

damping terms, enabling accurate representation of hyper-elastic and viscoelastic behavior.

Keywords: Medical Robots and Systems, Hardware-Software Integration in Robotics
Abstract: This study presents a robotic ultrasound imaging system designed to enhance diagnostic image quality through arc-based scanning trajectories and 3D reconstruction. Building on prior work in robotic linear ultrasound for musculoskeletal imaging, we introduce an arc scanning approach that aligns more perpendicularly to curved anatomical surfaces—such as finger joints—thereby improving bone and soft tissue visualization. Our integrated system combines a GE Vivid E95 ultrasound scanner with a Universal Robots UR3 robotic arm to perform automated B-mode, Color Flow Mapping (CFM), and photoacoustic scans. The scanning procedure consists of initial alignment, trajectory execution, motion compensation, and volume reconstruction, with interpolation based on proximity-weighted pixels from the nearest planes. We conducted phantom, volunteer, and patient studies to evaluate performance. Quantitative results from phantom scans demonstrated that the arc scan offers sharper boundary contrast. In both volunteer and patient cases, arc scans consistently produced more continuous and symmetric bone surfaces with uniform soft tissue appearance in B-mode imaging. However, CFM scans revealed fewer motion artifacts and clearer vascular signals in linear trajectories, suggesting arc-induced mechanical wave artifacts. These findings highlight the complementary strengths of each trajectory, motivating future work on hybrid scanning that integrates both linear and arc paths for optimized diagnostics.

Keywords: Formal Methods in Robotics and Automation, Motion and Path Planning, Planning under Uncertainty
Abstract: In this research, we explore the integration of topological data analysis techniques, specifically simplicial complexes and filtrations, with Discrete Morse Theory to enhance robotic motion planning and environmental mapping. While Vietoris-Rips complexes offer computational efficiency and Cˇech complexes capture topological features more accurately, combining these tools within a Discrete Morse Theory framework allows for simplification and identification of critical points in the configuration space without changing the topology of the structure. A central motivation for this work is the search for a simplicial complex that balances the topological accuracy of Cech complexes with the lower computational demands of Vietoris-Rips complexes. While creating such a complex is a challenging task, this trade-off drives the development of more efficient and accurate representations of space for robotics. Additionally, this research addresses the difficulty of identifying local minima and maxima in higher-dimensional settings, extending the utility of Discrete Morse Theory. Building on these ideas, we explore hybrid approaches involving Delaunay-Cˇech filtrations and Discrete Gradient Vector Fields to create more adaptive topological representations.

Keywords: Modeling, Control, and Learning for Soft Robots, Localization, Sensor Fusion
Abstract: This paper presents a novel proprioceptive state estimator for tensegrity robots, addressing the challenges posed by their unique structural configurations and dynamic behaviors. The tensegrity robot, constructed from a synergistic assembly of rigid rods and elastic cables, is designed for high-impact tolerance and shape morphing. To accurately capture both the global pose and internal shape of the robot in real time, our approach fuses high-frequency inertial measurements from onboard IMUs with precise cable length data from motor encoders. An optimization-based shape reconstruction algorithm—enforced by rigorous geometric and chirality constraints—is employed to estimate the endcap positions in the body frame, overcoming the complexities inherent in its continuously deforming configuration.

To further enhance the estimator's robustness, the forward kinematics from shape reconstruction is integrated into a contact-aided Invariant Extended Kalman Filter (InEKF) framework. This filter leverages detected endcap-ground contact events to correct state estimates via forward kinematics, ensuring improved accuracy even in scenarios with camera occlusions or degraded environmental conditions. Extensive simulation and real-world experiments validate the proposed framework, achieving an average drift of approximately 4.2% while maintaining computational efficiency suitable for onboard autonomous operations. Overall, this work not only advances state estimation methodologies for tense

Keywords: Representation Learning, Task and Motion Planning, Deep Learning Methods
Abstract: We envision a future where robots are equipped "out of the box" with portable skills. However, to effectively deploy and compose these skills for robotic tasks, users must know about the conditions under which they can be successfully executed and the consequences of the execution; in task planning, these are known as an action's preconditions and effects. We present SkillWrapper: an approach that automatically learns human-interpretable abstractions of skills, while guaranteeing complete and sound representations for planning, enabling long-horizon skill composition and zero-shot generalization. Our approach exploits foundation models to propose tasks from which it then learns a semantically meaningful grounded representations of preconditions and effects given images of what the robot perceives before and after executing a skill.

Keywords: Model Learning for Control, Wheeled Robots, Dynamics
Abstract: Autonomous off-road vehicles generate rich, high-frequency sensor data that capture complex and nonlinear interactions with unstructured terrain. While terrain dynamics are often modeled using physics-based approaches like Bekker models, such methods are computationally intensive and unsuitable for real-time use. Moreover, streaming data may contain rare or regime-specific events critical for effective modeling and control.

This work addresses the challenge of updating Koopman operator-based models in real time without overwhelming memory and computational resources. Rather than storing all incoming data or discarding potentially useful samples, we propose a batch update method that selectively incorporates only informative datasets. Novelty in dynamics is detected using the Grassmannian distance between subspaces, enabling efficient identification of meaningful regime shifts.

Our approach significantly reduces data volume and computational load while preserving model accuracy. Additionally, it includes basis function learning to optimize the Koopman representation. The framework is validated on simulated systems, demonstrating effective online learning, reduced model complexity, and improved prediction and control performance in off-road environments.

Keywords: Medical Robots and Systems, Machine Learning for Robot Control, Computer Vision for Medical Robotics
Abstract: Biological shape deformation resulting from robot palpation is a challenging problem in surgical robotics due to the complex contact modeling of tool-tissue interaction. The existing methods mainly focused on the development of model-based frameworks (e.g. Finite element analysis) and the use of force sensing to build a shape prediction model. However, these models are either computationally expensive or susceptible to the changes of tool configurations and tissue properties. To overcome this problem, we present an entirely data-driven method for soft-tissue surface prediction during robot palpation. We implement a multi-layer perceptron model to learn the complex relationship between the tool’s configuration (position and orientation) and the resulting surface deformation. Given a robot configuration and a local surface geometry around the contact center, the model can predict 3D displacement vectors of the pre-deform surfaces. We conducted simulation experiments to evaluate the model performance based on the various palpation angles between -40◦ and +40◦ and different tool penetration depths from 0.3 mm to 1.9 mm. The results show an average root mean square error of approximately 0.01 mm and an average maximum error less than 0.03 mm. This can demonstrate the feasibility of using a generic multi-layer perceptron model to learn the tool-tissue physics and shows potential applications of fast surface inference and deformation prediction during robot palpation.

Keywords: Computer Vision for Automation, Agricultural Automation, Robotics and Automation in Agriculture and Forestry
Abstract: The bloodletting in a duck slaughter process requires repetitive and dangerous labors. In order to automate the bloodletting process for duck slaughtering, we developed a machine vision-based bloodletting-slaughtering robot system. This system includes AI-based target recognition technology to detect the bloodletting area automatically and robot slaughtering system. A Mask-RCNN is utilized for object-background segmentation. Following the background segmentation, an integrated pipeline was implemented to find the target bloodletting point based on the estimated cervical vertebrae from the neck area of each duck. A training dataset of 1,789 RGB images was collected from duck slaughter process and the deep learning model for a duck neck-background segmentation was trained. In addition, classifying duck fainting and non-fainting is required for running a duck bloodletting process without system interruption. A lightweight CNN architecture using ONNX library was implemented for an application to edge devices in the workplace. The test results of automatically detecting the target bloodletting point showed high accuracy. A slaughtering robot system consists of a three-axis orthogonal robot, and the target bloodletting point is controlled in real time using position information transmitted from AI vision camera. The slaughtering task time using the robot system was 2.7 sec per duck. For application in slaughterhouses, five robots equipped with AI vision cameras will work in a set.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Multi-Robot Systems, Motion and Path Planning
Abstract: Multi-robot systems enhance efficiency and productivity across various applications, from manufacturing to surveillance. While single-robot motion planning has improved by using databases of prior solutions, extending this approach to multi-robot motion planning (MRMP) presents challenges due to the increased complexity and diversity of tasks and configurations. Recent discrete methods have attempted to address this by focusing on relevant lower-dimensional subproblems, but they are inadequate for complex scenarios like those involving manipulator robots. To overcome this, we propose a novel approach that constructs and utilizes databases of solutions for smaller sub-problems. By focusing on interactions between fewer robots, our method reduces the required database size compared to one that captures the full MRMP problem, enabling efficient handling of more complex scenarios. We validate our approach with experiments on mobile and manipulator robots, showing significant improvements in scalability and efficiency. Our method improves over its previous version with up to 40% faster planning and over existing experience-based methods with a 342× faster database construction and 2× faster planning. Our contributions include a rapidly constructed database for low-dimensional MRMP problems, a framework for applying these solutions to larger problems, and experimental validation with up to 32 mobile and 16 manipulator robots.

Keywords: Omnidirectional Vision, Visual Learning, Vision-Based Navigation
Abstract: Mobility assistance is one of the major supports for elderly and differently abled communities. Since wheelchairs are primarily used in human-centered areas, there is a significant risk of accidents caused by the wheelchair itself and by other pedestrians in the vicinity. To address this serious issue, this work proposes utilizing spherical camera data for environmental perception. However, incorporating spherical data may introduce additional challenges to the topic. The use of omnidirectional cameras is currently very limited for navigational purposes and for scene understanding. However, the equirectangular transformation of spherical data can help understand the surrounding environment. Moreover, these images offer better insight into the direction of a pedestrian's motion and the distance. The study proposes an architecture to extract spatial connections between the wheelchair and nearby pedestrians by identifying visual features and tracking the subjects' movements relative to the wheelchair. This method can easily detect when a pedestrian is approaching a wheelchair or passing nearby without entering a danger zone, and the proposed architecture was able to achieve over 95% accuracy at an average processing frame rate of 15 FPS. Thus, this demonstrates the potential of spherical image data, which can be further enhanced to make intelligent decisions by electric wheelchairs and reduce cognitive load on the user to make their life more comfortable.

Keywords: Multi-Modal Perception for HRI, Physically Assistive Devices, Object Detection, Segmentation and Categorization
Abstract: A robotic feeding system for finger foods for individuals with spinal cord injuries. Our system integrates zero-shot object detection model, it uses Vision Language Model to recognize different food items and leverage large language models to personalize the feeding preferences. The platform is equipped with multimodal capabilities to process visual, auditory and proprioceptive inputs in realtime and respond with natural human like response and execute appropriate low-level robot actions.

The system intelligently categorizes food items as sigle-bite or multi-bite to determine the most optimal grasp points. The robot also offers drink assistance using straw-delivery approach. For safe and comfortable bite transfer, our solution implements dynamic positioning that adapts to the user's height and facial orientation, using visual servoing with lip detection to ensure food is delivered only when the user's mouth is open. Additionally, we also provide a calibration method as an alternative to visual servoing, where the robot is set to admittance mode and manually moved to user's desired bite transfer pose. User studies with seven participants, including one with spinal cord injury, demonstrate excellent usability with SUS scores of 70 and NASA-TLX scores of 17, significantly outperforming the baseline average of 37±11.

Keywords: Medical Robots and Systems
Abstract: We present the design and implementation of a user interface (UI) for a robotic optical coherence tomography (OCT) system, developed to enhance usability and accessibility in ophthalmic imaging. The system integrates a robotic arm that positions an actively tracking OCT scan head over an extended workspace, minimizing the need for mechanical head stabilization and manual alignment. The UI architecture supports both operator-assisted and autonomous imaging modes, offering real-time feedback, intuitive interaction, and motion-adaptive visualization to facilitate effective human-robot collaboration. Future work includes a formal user study to evaluate the UI’s performance in terms of usability, task efficiency, and adaptability.

Keywords: Localization, Computational Geometry, AI-Based Methods
Abstract: Inertial odometry is essential for accurate localization in autonomous systems, particularly in environments where visual data is unreliable. However, noisy IMU measurements hinder accurate gravity compensation, which is critical for precise odometry. This paper presents a fully SO(3)-equivariant neural network framework for IMU-only odometry, designed to improve generalization across varying IMU mount orientations and mitigate challenges in gravity compensation. By representing IMU data in the Lie algebra and leveraging the symmetry properties of the rotation Lie group, the proposed method enforces 3D rotational symmetry within the feature space, leading to accurate state estimation in general unseen trajectories. Experimental results on diverse human motion datasets further demonstrate that our approach is more robust to gravitational perturbation on the IMU measurement, effectively reducing drift accumulation over the sequences. It remains robust under arbitrary IMU installation orientations, outperforming existing methods that rely on data augmentation or frame canonicalization. These improvements are achieved with a more compact network architecture, reducing computational complexity while maintaining accuracy. The results of this work enable reliable inertial position and orientation tracking in mobile robotics applications that have dynamic yaw direction motion with limited sensing capabilities.

Keywords: Force and Tactile Sensing, Perception for Grasping and Manipulation, Grasping
Abstract: Parallel-jaw grippers can perform a wide range of complex, everyday tasks. However, they often rely on open-loop control without direct grasp feedback, leading to failures in delicate manipulation and fragile object handling. Tactile sensing can address this, but existing methods typically involve high integration complexity or compromise gripper compliance. There is a clear need for developing sensorized compliant grippers without sacrificing simplicity. We address this by embedding 3D-printed air channels within Fin-Ray fingers. These channels act as force and slip sensors, providing precise, real-time feedback while remaining simple to fabricate and integrate. Our system estimates grip force with a RMSE under 0.2 N and incorporates an analytical slip detector using second-order spectral features of tactile vibrations. We evaluate the system on an autonomous grasp-and-lift task using 31 objects spanning categories of fragile, slippery, and ordinary. Across 310 trials, our slip detector achieved 0.93 accuracy and 1.0 precision. Our system achieved 91.9% overall success and 98.6% on fragile objects. In contrast, a on-off strategy caused breakage of 84.6% of fragile objects, while a naive initial 0.25 N grip force yielded only 1.25% success on ordinary objects. These results demonstrate that our design combines accurate, real-time tactile feedback with the inherent compliance of the Fin-Ray structure, enabling reliable manipulation of delicate and diverse objects.

Keywords: Manufacturing, Maintenance and Supply Chains, Mobile Manipulation, Object Detection, Segmentation and Categorization
Abstract: Quadruped robots are increasingly being deployed in manufacturing environments for autonomous inspection and manipulation tasks due to their unique advantage in mobility, such as navigating in tight spaces and stairs, which makes them ideal candidates for manufacturing plants. This paper investigates the potential of quadruped robots equipped with a 6-degree-of-freedom arm to perform complex tasks, including pick-and-place operations and identifying target objects in a cluttered environment. We conducted experiments focused on quadruped robots' ability to autonomously pick up, carry, and place a bucket in a structured workflow. We also investigated the use of computer vision and AprilTags for object detection and tracking. The quadruped demonstrated robust performance in both tasks, although challenges arise when handling heavier loads or finding target objects in cluttered environments. These findings make clear the capabilities and limitations of autonomous quadruped robots with manipulators in manufacturing settings.

Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles, Telerobotics and Teleoperation
Abstract: Microsurgical procedures demand exceptional precision and dexterity while posing significant challenges related to depth perception, fatigue, and hand tremor. Al- though intraoperative perception systems in surgical robotics typically provide real-time image feedback, they frequently lack volumetric or depth information. Moreover, teleoperated surgical robotic systems often require extensive training because their control mechanisms may not align with a surgeon’s intuitive practices, and standardizing these controls is further complicated by varying user preferences. To address these issues, we propose a dual-arm teleoperated robotic system that provides high-fidelity intraoperative volu- metric imaging for micro-scale tissue manipulation. The system integrates an optical coherence tomography (OCT) sensor for real-time 3D visualization and is operated through haptic input devices for precise manipulation. Surgeons can choose between multiple teleoperation frameworks to match their individual preferences and intuitions. We evaluate system performance through a precision positioning task and a vessel-following task in a retinal model, demonstrating average positioning errors of approximately 230 μm and 85 μm, respectively. Finally, we demonstrate the fully integrated system in an eggshell mem- brane peeling task that simulates retinal membrane peeling.

Keywords: Swarm Robotics, Multi-Robot Systems, Bioinspired Robot Learning
Abstract: We apply the NeuroEvolution of Augmented Topologies (NEAT) to train adaptive and efficient swarm robotic foraging behavior in unknown environments with random obstacles. By rewarding effective actions and penalizing inefficient ones, the system promotes coordination and obstacle avoidance, minimizing redundant exploration and outperforming traditional stochastic foraging algorithms. The optimization focuses on cumulative reward fitness, evaluated through simulations with three types of distributed resources, where swarm performance is analyzed in terms of foraging time efficiency and resource retrieval rates. Comparative experiments conducted across two swarm sizes demonstrate that the elaborated penalty-reward strategy enhances resource retrieval rates while reducing energy expenditure and obstacle avoidance, showcasing a significant improvement in foraging success and scalability over traditional foraging algorithms. In future work, we will leverage Federated Learning (FL) to build an efficient, distributed, scalable, and secure robot swarm tailored for foraging tasks.

Keywords: Localization
Abstract: The merits of electromagnetic tracking that line of sight is not required, such as utilizing inside a body or behind an obstruction, makes electromagnetic tracking systems attractive for minimally invasive procedures. One critical issue is that electromagnetic tracking does not provide uniform accuracy throughout the tracking volume. This study proposes integrating an electromagnetic tracking system consisting of a magnetic field generator of five excitation coils, a sensing coil, and a robotic arm. The magnetic field generator is installed on the robotic arm’s end effector. The pose of the electromagnetic tracking device placed on the robotic arm can be obtained from the calibration of the electromagnetic tracking system on the robotic arm. After calibration, the posture of the sensing coil in the robotic arm coordinates can be obtained and regarded as a closed-loop signal to control the robotic arm and place the excitation coil in a range suitable for sensing. The proposed method, which integrates an electromagnetic tracking system with a robotic arm, can increase the range of electromagnetic tracking and perform tracking in the accurate range.

Keywords: Probability and Statistical Methods, Optimization and Optimal Control, Motion and Path Planning, Risk-Sensitive Control
Abstract: We provide finite-sample performance guarantees for control policies executed on stochastic robotic systems. Given an open- or closed-loop policy and a finite set of trajectory rollouts under the policy, we bound the expected value, value-at-risk, and conditional-value-at-risk of the trajectory cost, and the probability of failure in a sparse cost setting. The bounds hold, with user-specified probability, for any policy synthesis technique and can be seen as a post-design safety certification. Generating the bounds only requires sampling simulation rollouts, without assumptions on the distribution or complexity of the underlying stochastic system. We adapt these bounds to also give a constraint satisfaction test to verify safety of the robot system. We provide a thorough analysis of the bound sensitivity to sim-to-real distribution shifts and provide results for constructing robust bounds that can tolerate some specified amount of distribution shift. Furthermore, we extend our method to apply when selecting the best policy from a set of candidates, requiring a multi-hypothesis correction. We show the statistical validity of our bounds in the Ant, Half-cheetah, and Swimmer MuJoCo environments and demonstrate our constraint satisfaction test with the Ant. Finally, using the 20 degree-of-freedom MuJoCo Shadow Hand, we show the necessity of the multi-hypothesis correction.

Keywords: Motion and Path Planning, Climbing Robots
Abstract: Autonomous in-pipe inspection robots can automatically navigate through complex pipeline networks and detect potential risks from corrosion and defects, demonstrating great potential for replacing costly manual inspections. However, there is no publicly available simulation environment where researchers can validate their in-pipe navigation algorithms as far as we know, and the navigation algorithms on constrained 3D pipe surface which is the critical software component are less discussed. Firstly, this paper proposes an open-source In-Pipe Navigation Development Environment. It contains various pipeline models, a magnetic wheel climbing robot model realized by the adhesion plugin, and baseline algorithms for navigation tasks. Secondly, a novel effective path planning method is introduced. Instead of planning based on surface structures, the proposed method plans based on pipeline axis and maps it into local path using the Frenet-Serret formula, thereby generating smooth, feasible, and efficient paths. Finally, we conduct both qualitative and quantitative experiments in the proposed simulation and real-world environments. The results show the usability of the development environment, also robustness and efficiency of the proposed planning method.

Keywords: Simulation and Animation, Calibration and Identification
Abstract: Accurate physical simulation is crucial for the development and validation of control algorithms in robotic systems. Recent works in Reinforcement Learning (RL) take notably advantage of extensive simulations to produce efficient robot control. State-of-the-art servo actuator models generally fail at capturing the complex friction dynamics of these systems. This limits the transferability of simulated behaviors to real-world applications. In this work, we present extended friction models that allow to more accurately simulate servo actuator dynamics. We propose a comprehensive analysis of various friction models, present a method for identifying model parameters using recorded trajectories from a pendulum test bench, and demonstrate how these models can be integrated into physics engines. The proposed friction models are validated on four distinct servo actuators and tested on 2R manipulators, showing significant improvements in accuracy over the standard Coulomb-Viscous model. Our results highlight the importance of considering advanced friction effects in the simulation of servo actuators to enhance the realism and reliability of robotic simulations.

Keywords: Motion and Path Planning, Industrial Robots
Abstract: Many robotic applications, such as sanding, polishing, wiping and sensor scanning, require a manipulator to dexterously cover a surface using its end-effector. In this paper, we provide an efficient and effective coverage path planning approach that leverages a manipulator's redundancy and task tolerances to minimize costs in joint space. We formulate the problem as a Generalized Traveling Salesman Problem and hierarchically streamline the graph size. Our strategy is to identify guide paths that roughly cover the surface and accelerate the computation by solving a sequence of smaller problems. We demonstrate the effectiveness of our method through a simulation experiment and an illustrative demonstration using a physical robot.

Keywords: Integrated Planning and Control, Multi-Robot Systems
Abstract: To deploy safe and agile robots in cluttered environments, there is a need to develop fully decentralized controllers that guarantee safety, respect actuation limits, prevent deadlocks, and scale to thousands of agents. Current approaches fall short of meeting all these goals: optimization-based methods ensure safety but lack scalability, while learning-based methods scale but do not guarantee safety. We propose a novel algorithm to achieve safe and scalable control for multiple agents under limited actuation. Specifically, our approach includes: (i) learning a decentralized neural Integral Control Barrier function (neural ICBF) for scalable, input-constrained control, (ii) embedding a lightweight decentralized Model Predictive Control-based Integral Control Barrier Function (MPC-ICBF) into the neural network policy to ensure safety while maintaining scalability, and (iii) introducing a novel method to minimize deadlocks based on gradient-based optimization techniques from machine learning to address local minima in deadlocks. Our numerical simulations show that this approach outperforms state-of-the-art multi-agent control algorithms in terms of safety, input constraint satisfaction, and minimizing deadlocks. Additionally, we demonstrate strong generalization across scenarios with varying agent counts, scaling up to 1000 agents.

Keywords: Motion and Path Planning, Parallel Robots, Robotics and Automation in Construction
Abstract: We present an algorithmic framework for a multi-robot modular assembly system. Motivated by the prospects of in-space assembly, we focus on the NASA Automated Reconfigurable Mission Adaptive Digital Assembly Systems (ARMADAS) framework, in which multiple types of robots work together in a team to build large structures. Unlike with other multi-robot construction systems, the geometry of structures that ARMADAS robots can build is not limited to the class of histogram shapes. To address the intractability of path planning for a robot system with the exponentially growing number of dimensions, we present a decoupled planning approach, where the assembly and path planning is performed iteratively by one robot team at a time. We present a number of data structures which help us avoid collisions and deadlocks in the resulting robot schedule.

Keywords: Visual Servoing, Reactive and Sensor-Based Planning, Motion and Path Planning
Abstract: This letter presents a versatile trajectory planning pipeline for aerial tracking. The proposed tracker is capable of handling various chasing settings such as complex unstructured environments, crowded dynamic obstacles and multiple-target following. Among the entire pipeline, we focus on developing a predictor for future target motion and a chasing trajectory planner. For rapid computation, we employ the sample-check-select strategy: modules sample a set of candidate movements, check multiple constraints, and then select the best trajectory. Also, we leverage the properties of Bernstein polynomials for quick calculations. The prediction module predicts the trajectories of the targets, which do not overlap with static and dynamic obstacles. Then the trajectory planner outputs a trajectory, ensuring various conditions such as occlusion and collision avoidance, the visibility of all targets within a camera image and dynamical limits. We fully test the proposed tracker in simulations and hardware experiments under challenging scenarios, including dual-target following, environments with dozens of dynamic obstacles and complex indoor and outdoor spaces.

Keywords: SLAM, Mapping, Localization, Deep Learning
Abstract: Accurate and robust localization and mapping are essential components for most autonomous robots. In this paper, we propose a SLAM system for building globally consistent maps, called PIN-SLAM, that is based on an elastic and compact point-based implicit neural map representation. Taking range measurements as input, our approach alternates between incremental learning of the local implicit signed distance field and the pose estimation given the current local map using a correspondence-free, point-to-implicit model registration. Our implicit map is based on sparse optimizable neural points, which are inherently elastic and deformable with the global pose adjustment when closing a loop. Loops are also detected using the neural point features. Extensive experiments validate that PIN-SLAM is robust to various environments and versatile to different range sensors such as LiDAR and RGB-D cameras. PIN-SLAM achieves pose estimation accuracy better or on par with the state-of-the-art LiDAR odometry or SLAM systems and outperforms the recent neural implicit SLAM approaches while maintaining a more consistent, and highly compact implicit map that can be reconstructed as accurate and complete meshes. Finally, thanks to the voxel hashing for efficient neural points indexing and the fast implicit map-based registration without closest point association, PIN-SLAM can run at the sensor frame rate on a moderate GPU.